\section{System Environment} In this section, we are going to discuss system requirements for a continuous operation of the tools. This environment differs in some regards to the one used for experiments and laboratory tests, given that monitoring, self-recovery in case of errors and resistance to failure are much more important in a 24/7 operation. We will be using the term \emph{production environment} to refer to such a usage. \subsection{Launching the tools} Services running in a production environment are usually administered remotely, and have to be able to run without requiring a user to be connected all the time. Traditionally, such services are implemented as daemons in UNIX terminology, and are started and stopped using the init system of the distribution. The ODR-mmbTools cannot daemonise themselves, and require another approach. \paragraph{Screen multiplexer} An easy approach is to use a screen multiplexer, like \emph{GNU Screen} or \emph{tmux} that can be used to launch a session from which the user can detach and reattach. See the relevant manpages for more information. A screen multiplexer alone does permit the tools to run without a user connected, but does not automatically restart failed processes, and does not send any warning emails in case of a fault. The dab-scripts\footnote{The dab-scripts are available on the Opendigitalradio github.}, already mentioned in \ref{usingexistingwebstreams}, can be a helpful addition in that case. They can monitor that the processes are still running, and if necessary restart them. \paragraph{supervisord} The execution of the tools can also be supervised by a dedicated tool instead of using the scripts. \texttt{supervisor}\footnote{\url{http://supervisord.org}} is (its name will not surprise you) exactly such a tool. Once installed, it reads the configuration in \texttt{/etc/supervisor.conf} and launches the processes it has to monitor. Each process to monitor is described by a file. The following example assumes the tools run as user \texttt{odr}, and the multiplex configuration is in \texttt{/home/odr/config.mux}, and launches ODR-DabMux. The logs of ODR-DabMux is written to the log specified files. \begin{lstlisting} [program:ODR-DabMux] command=odr-dabmux config.mux directory=/home/odr user=odr autostart=true autorestart=true stdout_logfile=/home/odr/logs/mux.out.log stderr_logfile=/home/odr/logs/mux.err.log \end{lstlisting} Once this configuration has been added to the supervisor configuration, the settings have to be reread using: \begin{lstlisting} supervisorctl reread \end{lstlisting} In order for supervisor to start managing and running this process, it has to be added: \begin{lstlisting} supervisorctl add ODR-DabMux \end{lstlisting} Setting up more processes can be simply achieved by customising this configuration template for the other tools. Examples are available in the \texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder, and have to be customised to the paths used in your setup. supervisor also includes a small web-server that can display the state of the managed processes. It is enabled with the \verb+[inet_http_server]+ setting in the configuration file. \subsection{Logging} Essential for a production environment is traceability of events, which is achieved using logging of important messages. ODR-DabMux and ODR-DabMod both support logging to standard error, to a file and to the system logger \texttt{syslog}. Logging to syslog is the most flexible solution, because the log information can be forwarded over the network to a centralised logging server, and can be filtered according to the priority of each message. Both tools log to the LOCAL0 facility, which can be redirected to a file, in order to group ODR-mmbTools-related messages together. %\sidenote{Describe rsyslog configuration} In order to avoid that the log files grow over time to unreasonable sizes, \texttt{logrotate} should be setup to rotate the files automatically. %\sidenote{Describe logrotate configuration} \subsection{Timing} The ODR-mmbTools require a correct system time to function properly. This is especially important when running a SFN, but cannot be neglected for a proper production operation even with a single transmitter. The system needs to run a NTP client that synchronises the system time over the network. Correct synchronisation can be checked using the \texttt{lpeers} command of the \texttt{ntpq} tool. The magnitude of the offset, indicated in milliseconds, should be below 10. The performance of the NTP synchronisation should also be monitored permanently during operation. \subsection{Monitoring using munin} The Munin\footnote{\url{http://munin-monitoring.org/}} monitoring tool can create graphs for essential system health parameters, and can also send emails in case a specific value exceeds its allowed bounds. This helps the operator in assessing the system status and the well-functioning of the services. In addition to basic system measurements like CPU, RAM and disk usage, NTP synchronisation, disk and network performance and many other information, there are also custom data sources for ODR-DabMux. These data sources include ZMQ input buffer monitoring (buffer level, underruns and overruns) and input audio level (peak for both channels). This data source can be installed by copying \verb+doc/stats_dabmux_multi.py+ to \texttt{/etc/munin/plugins.d}. They require that the ODR-DabMux management server is enabled in the configuration, and will automatically generate the graphs for the subchannels used in the configuration. \subsection{Real-time Scheduling} As a general principle, it is advisable to run tools that do not need superuser privileges under another user than \texttt{root}. The same principle also applies to the ODR-mmbTools, where care has to be taken that the tools can request real-time scheduling when needed. This is achieved by adding the following to \texttt{/etc/security/limits.conf}, assuming the tools are run under the user \texttt{odr}. \begin{lstlisting} odr - rtprio 65 odr - nice -10 \end{lstlisting} If you have installed JACK with real-time privileges, you might already find the same setting, but for the audio group, written as \texttt{@audio}, which is sufficient. \subsection{Accessing the USRP as Non-root} Superuser privileges are not required to access USB-connected USRP devices, but sometimes the system lacks the configuration to enable normal users to communicate to the device. In that case, it is necessary to add a rule file for \texttt{udev}. This file is included in the UHD sources, but might not have been automatically installed. The file is called \texttt{uhd-usrp.rules}, should be placed into \texttt{/etc/udev/rules.d/} and should contain \begin{lstlisting} #USRP1 SUBSYSTEMS=="usb", ATTRS{idVendor}=="fffe", ATTRS{idProduct}=="0002", MODE:="0666" #B100 SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0002", MODE:="0666" #B200 SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0020", MODE:="0666" \end{lstlising} % vim: spl=en spell tw=80 et