\section{System Environment}

In this section, we are going to discuss system requirements for a continuous
operation of the tools. This environment differs in some regards to the one used
for experiments and laboratory tests, given that monitoring, self-recovery in
case of errors and resistance to failure are much more important in a 24/7
operation. We will be using the term \emph{production environment} to refer to
such a usage.

\subsection{Launching the tools}

Services running in a production environment are usually administered remotely,
and have to be able to run without requiring a user to be connected all the
time. Traditionally, such services are implemented as daemons in UNIX
terminology, and are started and stopped using the init system of the
distribution.
The ODR-mmbTools cannot daemonise themselves, and require another approach.

\paragraph{Screen multiplexer}
An easy approach is to use a screen multiplexer, like \emph{GNU Screen} or
\emph{tmux} that can be used to launch a session from which the user can detach
and reattach. See the relevant manpages for more information.

A screen multiplexer alone does permit the tools to run without a user
connected, but does not automatically restart failed processes, and does not
send any warning emails in case of a fault. The
dab-scripts\footnote{The dab-scripts are available on the Opendigitalradio
github.}, already mentioned in \ref{usingexistingwebstreams}, can be a helpful
addition in that case. They can monitor that the processes are still running,
and if necessary restart them.

\paragraph{supervisord}
The execution of the tools can also be supervised by a dedicated tool instead of
using the scripts. \texttt{supervisor}\footnote{\url{http://supervisord.org}} is
(its name will not surprise you) exactly such a tool.

Once installed, it reads the configuration in \texttt{/etc/supervisor.conf} and
launches the processes it has to monitor. Each process to monitor is described
by a file. The following example assumes the tools run as user \texttt{odr}, and
the multiplex configuration is in
\texttt{/home/odr/config.mux}, and launches ODR-DabMux.
The logs of ODR-DabMux is written to the log specified files.

\begin{lstlisting}
[program:ODR-DabMux]
command=odr-dabmux config.mux
directory=/home/odr
user=odr
autostart=true
autorestart=true
stdout_logfile=/home/odr/logs/mux.out.log
stderr_logfile=/home/odr/logs/mux.err.log
\end{lstlisting}

Once this configuration has been added to the supervisor configuration, the
settings have to be reread using:
\begin{lstlisting}
supervisorctl reread
\end{lstlisting}

In order for supervisor to start managing and running this process, it has to be added:
\begin{lstlisting}
supervisorctl add ODR-DabMux
\end{lstlisting}

Setting up more processes can be simply achieved by customising this
configuration template for the other tools. Examples are available in the
\texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder, and have
to be customised to the paths used in your setup.

supervisor also includes a small web-server that can display the state of the
managed processes. It is enabled with the \verb+[inet_http_server]+ setting in
the configuration file.

\subsection{Logging}
Essential for a production environment is traceability of events, which is
achieved using logging of important messages.

ODR-DabMux and ODR-DabMod both support logging to standard error, to a file and
to the system logger \texttt{syslog}. Logging to syslog is the most flexible
solution, because the log information can be forwarded over the network to a
centralised logging server, and can be filtered according to the priority of
each message. Both tools log to the LOCAL0 facility, which can be redirected to
a file, in order to group ODR-mmbTools-related messages together.

%\sidenote{Describe rsyslog configuration}

In order to avoid that the log files grow over time to unreasonable sizes,
\texttt{logrotate} should be setup to rotate the files automatically.
%\sidenote{Describe logrotate configuration}


\subsection{Timing}
The ODR-mmbTools require a correct system time to function properly. This is
especially important when running a SFN, but cannot be neglected for a proper
production operation even with a single transmitter.

The system needs to run a NTP client that synchronises the system time over the
network. Correct synchronisation can be checked using the \texttt{lpeers}
command of the \texttt{ntpq} tool. The magnitude of the offset, indicated in
milliseconds, should be below 10.

The performance of the NTP synchronisation should also be monitored permanently
during operation.


\subsection{Monitoring using munin}

The Munin\footnote{\url{http://munin-monitoring.org/}} monitoring tool can
create graphs for essential system health parameters, and can also send emails
in case a specific value exceeds its allowed bounds. This helps the operator in
assessing the system status and the well-functioning of the services.

In addition to basic system measurements like CPU, RAM and disk usage, NTP
synchronisation, disk and network performance and many other information, there
are also custom data sources for ODR-DabMux.

These data sources include ZMQ input buffer monitoring (buffer level, underruns
and overruns) and input audio level (peak for both channels). This data source
can be installed by copying \verb+doc/stats_dabmux_multi.py+ to
\texttt{/etc/munin/plugins.d}. They require that the ODR-DabMux management
server is enabled in the configuration, and will automatically generate the
graphs for the subchannels used in the configuration.


\subsection{Real-time Scheduling}

As a general principle, it is advisable to run tools that do not need superuser
privileges under another user than \texttt{root}. The same principle also
applies to the ODR-mmbTools, where care has to be taken that the tools can
request real-time scheduling when needed.

This is achieved by adding the following to \texttt{/etc/security/limits.conf},
assuming the tools are run under the user \texttt{odr}.

\begin{lstlisting}
odr          -       rtprio          65
odr          -       nice           -10
\end{lstlisting}

If you have installed JACK with real-time privileges, you might already find the
same setting, but for the audio group, written as \texttt{@audio}, which is
sufficient.

\subsection{Accessing the USRP as Non-root}

Superuser privileges are not required to access USB-connected USRP devices, but
sometimes the system lacks the configuration to enable normal users to
communicate to the device.

In that case, it is necessary to add a rule file for \texttt{udev}. This file is
included in the UHD sources, but might not have been automatically installed.

The file is called \texttt{uhd-usrp.rules}, should be placed into
\texttt{/etc/udev/rules.d/} and should contain
\begin{lstlisting}
#USRP1
SUBSYSTEMS=="usb", ATTRS{idVendor}=="fffe", ATTRS{idProduct}=="0002", MODE:="0666"
#B100
SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0002", MODE:="0666"
#B200
SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0020", MODE:="0666"
\end{lstlising}

% vim: spl=en spell tw=80 et