rfnoc: Add a new builder script for FPGA images based on eRFNoC.

The builder has two major jobs:
 * generate an image core file which reflects the FPGA image
   configuration given by the user
 * invoke Xilinx toolchain to actually build the FPGA image

For this purpose it needs to know where to find the FPGA source tree.
This tree can be give by the -F option.

The code that represents the user configurable part of the image is
written to a file called <device>_rfnoc_sandbox.v. To generate the file
these configuration files are needed:
 * io_signatures.yml: A file describing the known IO signatures. This file
                      is global for all devices and contains the superset
                      of all signatures (not all signatures are used by all
                      devices). It resides in usrp3/top/ of the tree given
                      by -F.
* bsp.yml: A file describing interfaces of a specific device such as AXIS
           transport interfaces or IO ports as well as device specific
           settings. It resides in usrp3/top/<device> of the tree given by -F.
* <image>.yml: a file provided by the user with freely chosen name.
               It describes which elements the image should contain
               (RFNoC blocks, streaming endpoints, IO ports) and how
               to connect them. The file also contains image setting
               such as the CHDR width to be used.

The script uses mako templates to generate the sandbox file. Before the
template engine is invoked sanity checks are executed to ensure the
configuration is synthactic correct. The script also build up structures
to ease Verilog code generation in the template engine. The engine should
not invoke more Python than echoing variables or iterating of lists or
dictionaries. This eases debugging as errors in the template engine are
hard to track and difficult to read for the user.

All Python code is placed in a package called rfnoc. The templates used
by the builder are also part of this package. image_builder.py contains
a method called build_image which is the main entry point for the builder.
It can also be utilized by other Python programs. To align with the
existing uhd_image_builder there is also a wrapper in bin called
rfnoc_image_builder which expects similar commands as the uhd_image_builder.

For debugging purpuse the script can be invoked from host/utils using
$ PYTHONPATH=. python bin/rfnoc_image_builder <options>

When installed using cmake/make/make install the builder installs to
${CMAKE_INSTALL_PREFIX}bin and can be invoked without specifying a
PYTHONPATH.

One can also install the package using pip from host/utils
$ pip install .

Image config generation can also be done from  GNU Radio Companion
files. The required GRC files are merged into gr-ettus.

Example usage:

$ rfnoc_image_builder -F ~/src/fpgadev -d x310 \
    -r path/to/x310_rfnoc_image_core.grc \
    -b path/to/gr-ettus/grc

Co-Authored-By: Alex Williams <alex.williams@ni.com>
Co-Authored-By: Sugandha Gupta <sugandha.gupta@ettus.com>
Co-Authored-By: Martin Braun <martin.braun@ettus.com>

1 files changed, 107 insertions, 0 deletions

diff --git a/host/utils/rfnoc/yaml_utils.py b/host/utils/rfnoc/yaml_utils.py
new file mode 100644
index 000000000..e64556334
--- /dev/null
+++ b/host/utils/rfnoc/yaml_utils.py
@@ -0,0 +1,107 @@
+"""
+Copyright 2019 Ettus Research, A National Instrument Brand
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+import json
+import logging
+import os
+import sys
+from ruamel import yaml
+
+# Allow jsonschema import to fail. If not available no schema validation will
+# be done (but warning will be printed for each skipped validation).
+try:
+    import jsonschema
+except ImportError:
+    logging.warning("Module jsonschema is not installed. Configuration files "
+                    "will not be validated against their schema.")
+
+
+def find_schema(schema_name, config_path):
+    """
+    Recursive search for schema file. Only looks for a file with appropriate
+    name without checking for content or accessibility.
+    This check will be performed later when trying to load the schema.
+    :param schema_name: name of schema file to search for
+    :param config_path: root path to start search in
+    :return: full path to schema file if found, None else
+    """
+    for root, _, _ in os.walk(config_path):
+        filename = os.path.join(root, schema_name)
+        if os.path.isfile(filename):
+            return filename
+    return None
+
+
+def validate_config(config, config_path):
+    """
+    Try to validate config.
+
+    config contains a configuration loaded from a yaml file. config is assumed
+    to be a dictionary which contains a key 'schema' which determines
+    which schema to validate against. The schema (json formatted) needs to be
+    located in config_path or any sub folder of it.
+    If "jsonschema" module cannot be loaded validation is skipped and config
+    is assumed to be valid. This way a configuration can also be loaded if
+    "jsonschema" is not available.
+    The method raises ValueError if no schema is defined in config or the
+    schema defined in config cannot be found. The validation itself may throw
+    a jsonschema.exceptions.ValidationError if config does not confirm to its
+    schema.
+    :param config: a dictionary to validate (loaded from yaml file).
+    :param config_path: a path holding schema definitions
+    """
+    if "jsonschema" not in sys.modules:
+        logging.warning("Skip schema validation (missing module jsonschema).")
+        return
+
+    if not "schema" in config:
+        raise ValueError("Missing schema in configuration.")
+
+    schema_name = config["schema"]
+    logging.debug("Validate against schema %s.", schema_name)
+
+    schema_file = find_schema('%s.json' % schema_name, config_path)
+    if not schema_file:
+        raise ValueError("Unknown schema %s." % schema_name)
+
+    logging.debug("Using schema file %s.", schema_file)
+
+    with open(schema_file) as stream:
+        jsonschema.validate(instance=config, schema=json.load(stream))
+        logging.debug("Configuration successful validated.")
+
+
+def load_config(config_file, config_path):
+    """
+    Wrapper method to unify loading of configuration files.
+    Beside loading the configuration (yaml file format) itself from config_file
+    this method also validates the configuration against a schema. The root
+    element of the configuration must load to a dictionary which contains a
+    "schema" key. The value of schema points to a file named {value}.json which
+    must be located in config_path or one of its sub folders.
+    .. seealso:: validate_config.
+    :param config_file: configuration to load
+    :param config_path: root path of schema definition files
+    :return:
+    """
+    logging.debug("Load configuration %s.", config_file)
+    with open(config_file) as stream:
+        rt_yaml = yaml.YAML(typ='rt')
+        config = rt_yaml.load(stream)
+        logging.debug("Configuration successful loaded.")
+        validate_config(config, config_path)
+        return config