mpm: mg: Adding fast-reinit mode

When a device is re-initialized without any changes (e.g.,
master_clock_rate, ref_clock_freq) then we can skip the initialization
sequence and move on. This shaves a significant amount of time from the
init sequence.

Fast re-init can be overridden by providing the `force_reinit=1` device
arg.

3 files changed, 123 insertions, 19 deletions

diff --git a/host/docs/usrp_n3xx.dox b/host/docs/usrp_n3xx.dox
index 47984aeb1..c9efa642d 100644
--- a/host/docs/usrp_n3xx.dox
+++ b/host/docs/usrp_n3xx.dox
@@ -378,6 +378,7 @@ For a list of which arguments can be passed into make(), see Section
  second_addr           | IPv4 address of secondary SFP+ port to connect to.                           | All N3xx          | second_addr=192.168.40.2
  mgmt_addr             | IPv4 address or hostname which to connect the RPC client. Defaults to `addr'.| All N3xx          | mgmt_addr=ni-sulfur-311FE00 (can also go to RJ45)
  find_all              | When using broadcast, find all devices, even if unreachable via CHDR.        | All N3xx          | find_all=1
+ force_reinit          | Force full reinitialization of all subsystems. Will increase init time.      | N310              | force_reinit=1
  master_clock_rate     | Master Clock Rate in Hz                                                      | N310              | master_clock_rate=125e6
  identify              | Causes front-panel LEDs to blink. The duration is variable.                  | N310              | identify=5 (will blink for about 5 seconds)
  serialize_init        | Force serial initialization of daughterboards.                               | All N3xx          | serialize_init=1
@@ -758,6 +759,29 @@ AD9371 RFIC).
 
 \image html N310fp.png N310 Front Panel
 
+\subsection n3xx_mg_initialization Device Initialization (Fast and Slow)
+
+When a UHD session is created, an initialization sequence is started. As part of
+the initialization sequence, the following steps are performed:
+
+- All clocking is initialized
+- The JESD links are trained and brought up (between the FPGA and the AD9371)
+- The AD9371 is reset, its firmware is uploaded, and calibrations are
+  initialized (See also \section n3xx_mg_calibrations)
+- N310 only: The multi-chip synchronization is performed to align all the RFICs
+  to the common time and clock reference
+
+This sequence can take a while, depending on the master clock rate and the
+calibration sequence. To speed things up, the device will retain a state
+between sessions, but only if no relevant settings were touched. In particular,
+changing the master clock rate, the clock source, or the calibration masks will
+force a full re-initialization which is very slow compared to the fast
+re-initialization. By setting the log level to DEBUG you will be able to observe
+the exact settings that cause fast vs. slow re-initialization.
+If you require a full re-initialization every time a UHD session is spawned,
+specify the `force_reinit` flag as a device arg. Specifying it will always do
+the full, slow initialization, but will guarantee a full reset of the RFIC.
+
 \subsection n3xx_mg_calibrations RF Calibrations
 
 The onboard RFIC (AD9371) has built-in calibrations which can be enabled from
diff --git a/mpm/python/usrp_mpm/dboard_manager/magnesium.py b/mpm/python/usrp_mpm/dboard_manager/magnesium.py
index b19ebf789..35d3bb22f 100644
--- a/mpm/python/usrp_mpm/dboard_manager/magnesium.py
+++ b/mpm/python/usrp_mpm/dboard_manager/magnesium.py
@@ -283,27 +283,47 @@ class Magnesium(DboardManagerBase):
             error_msg = "Cannot run init(), peripherals are not initialized!"
             self.log.error(error_msg)
             raise RuntimeError(error_msg)
-        fast_reinit = True
+        # Check if ref clock freq changed (would require a full init)
+        ref_clk_freq_changed = False
         if 'ref_clk_freq' in args:
             new_ref_clock_freq = float(args['ref_clk_freq'])
             assert new_ref_clock_freq in (10e6, 20e6, 25e6)
             if new_ref_clock_freq != self.ref_clock_freq:
-                fast_reinit = False
                 self.ref_clock_freq = float(args['ref_clk_freq'])
+                ref_clk_freq_changed = True
         assert self.ref_clock_freq is not None
+        # Check if master clock freq changed (would require a full init)
         master_clock_rate = \
             float(args.get('master_clock_rate',
                            self.default_master_clock_rate))
         assert master_clock_rate in (122.88e6, 125e6, 153.6e6), \
                 "Invalid master clock rate: {:.02f} MHz".format(
                     master_clock_rate / 1e6)
-        master_clock_rate_changed = master_clock_rate != self.master_clock_rate
+        master_clock_rate_changed = \
+            master_clock_rate != self.master_clock_rate
         if master_clock_rate_changed:
-            fast_reinit = False
             self.master_clock_rate = master_clock_rate
-            self.log.debug("Updating master clock rate to {:.02f} MHz!".format(
-                self.master_clock_rate / 1e6
-            ))
+            self.log.debug(
+                "Updating master clock rate to {:.02f} MHz!"
+                .format(self.master_clock_rate / 1e6)
+            )
+        # Track if we're able to do a "fast reinit", which means there were no
+        # major changes and can skip all slow initialization steps.
+        fast_reinit = \
+            not bool(args.get("force_reinit", False)) \
+            and not master_clock_rate_changed \
+            and not ref_clk_freq_changed
+        if fast_reinit:
+            self.log.debug(
+                "Attempting fast re-init with the following settings: "
+                "master_clock_rate={} MHz ref_clk_freq={}"
+                .format(
+                    self.master_clock_rate / 1e6,
+                    self.ref_clock_freq,
+                )
+            )
+        # Note: MagnesiumInitManager.init() can still override fast_reinit.
+        # Consider it a hint.
         result = MagnesiumInitManager(self, self._spi_ifaces).init(
             args, self._init_args, fast_reinit)
         if result:
diff --git a/mpm/python/usrp_mpm/dboard_manager/mg_init.py b/mpm/python/usrp_mpm/dboard_manager/mg_init.py
index fccdc2e27..f9e3de32e 100644
--- a/mpm/python/usrp_mpm/dboard_manager/mg_init.py
+++ b/mpm/python/usrp_mpm/dboard_manager/mg_init.py
@@ -445,6 +445,14 @@ class MagnesiumInitManager(object):
         self.log.trace("Pulsing Mykonos Hard Reset...")
         self.mg_class.cpld.reset_mykonos()
         self.log.trace("Initializing Mykonos...")
+        # TODO: If we can set the LO source after initialization, that would
+        # enable us to switch LO sources without doing the entire JESD and
+        # clocking bringup. For now, we'll keep it here, and every time the LO
+        # source needs to be changed, we need to re-initialize (this is because
+        # MYKONOS_initialize() takes in the entire device config, which includes
+        # the LO source), but we can revisit this if we want to either
+        # - speed up init when the only change is the LO source, or
+        # - we want to make the LO source runtime-configurable.
         self.init_lo_source(args)
         self.mykonos.begin_initialization()
         # Multi-chip Sync requires two SYSREF pulses at least 17us apart.
@@ -543,24 +551,76 @@ class MagnesiumInitManager(object):
             self.init_jesd(jesdcore, master_clock_rate, args)
             jesdcore = None # Help with garbage collection
             # That's all that requires access to the dboard regs!
-        self.init_rf_cal(args)
+        return True
+
+
+    def init(self, args, old_args, fast_reinit):
+        """
+        Runs the actual initialization.
+
+        Arguments:
+        args -- Dictionary with user-specified args
+        old_args -- Dictionary with user-specified args from the previous
+                    initialization run.
+        fast_reinit -- A hint to do a fast reinit. If nothing changes, then
+                       we don't have to re-init everything and their dogs, we
+                       can skip a whole bunch of things.
+        """
+        # If any of these changes, we need a full re-init:
+        # TODO: This is not very DRY (because we're repeating default values),
+        # and is generally smelly design. However, we're being super
+        # conservative for now, because the only reliable reset sequence we
+        # have for AD9371 is the full Monty. As we learn more about the chip,
+        # we might be able to get away with a partial (fast) reinit even when
+        # some of these values change.
+        args_that_must_not_change = [
+            ('rx_lo_source', 'internal'),
+            ('tx_lo_source', 'internal'),
+            ('init_cals', 'DEFAULT'),
+            ('tracking_cals', 'DEFAULT'),
+            ('init_cals_timeout', str(self.mykonos.DEFAULT_INIT_CALS_TIMEOUT)),
+        ]
+        if fast_reinit:
+            for arg_key, arg_default in args_that_must_not_change:
+                old_value = old_args.get(arg_key, arg_default)
+                new_value = args.get(arg_key, arg_default)
+                if old_value != new_value:
+                    self.log.debug(
+                        "The following init arg changed and caused "
+                        "a full re-init sequence: {}".format(arg_key))
+                    fast_reinit = False
+            # TODO: Maybe we can switch to digital loopback without running the
+            # initialization. For now, force init when rfic_digital_loopback is
+            # set because we're being conservative.
+            if bool(args.get('rfic_digital_loopback')):
+                self.log.debug("Using rfic_digital_loopback causes a full "
+                               "re-init sequence.")
+                fast_reinit = False
+        # If we can't do fast re-init, start from scratch:
+        if not fast_reinit:
+            if not self._full_init(
+                    self.mg_class.slot_idx,
+                    self.mg_class.master_clock_rate,
+                    self.mg_class.ref_clock_freq,
+                    args
+                ):
+                return False
+        else:
+            self.log.debug("Running fast re-init with the following settings:")
+            for arg_key, arg_default in args_that_must_not_change:
+                self.log.debug(
+                    "{}={}".format(arg_key, args.get(arg_key, arg_default)))
+            return True
         if bool(args.get('rfic_digital_loopback')):
             self.log.warning(
                 "RF Functionality Disabled: JESD204b digital loopback "
                 "enabled inside Mykonos!")
             self.mykonos.enable_jesd_loopback(1)
         else:
+            # Now initialize calibrations:
+            # TODO: This also takes a long time. It might be faster to somehow
+            # just reset the calibrations, but one thing at a time.
+            self.init_rf_cal(args)
             self.mykonos.start_radio()
         return True
 
-    def init(self, args, fast_reinit):
-        """
-        Runs the actual initialization.
-        """
-        if not fast_reinit or True:
-            return self._full_init(
-                self.mg_class.slot_idx,
-                self.mg_class.master_clock_rate,
-                self.mg_class.ref_clock_freq,
-                args
-            )