% LICENSE: see LICENCE
\section{System Environment}
\label{systemenvironment}

In this section, we describe the system configuration requirements for the
continuous operation of the tools. The production environment differs in some
respects to those used for experimentation and in laboratory testing. Monitoring,
automatic recovery (in case of errors) and resilience are crucial in 24/7
operations. The term \emph{production environment} will be used here to refer to
such use.

\subsection{Processing requirements}

The tools have differing requirements regarding CPU performance and amount of
memory, and while the performance of most desktop PCs is sufficient to run the
tools, it is important to take the requirements in consideration when setting up
a system.
Memory requirements are easily met with 1GB of RAM, so we'll look at CPU more in
depth.

The most resource-consuming part is the modulator ODR-DabMod. The
following impact its CPU usage: number of sub-channels; enabling of the
resampler; enabling crest factor reduction; enabling the predistorter.
Compilation options to optimise ODR-DabMod for your system are described in the
README. While you should have no trouble running it even on an older desktop PC,
the computing power of embedded ARM boards (like the Raspberry Pi) could be
insufficient, especially if the resampler is needed.\footnote{See section
\ref{otherhardware} for an example.}
When using a USB SDR device, the USB controller can have a large impact on the
robustness of the transmission, even if CPU usage is low. Such issues are visible as
underruns during operation: with a good controller, less than one underrun per
day is easily achievable on a machine dedicated to only this task. When using
a graphical interface at the same time, interaction with the user interface can
also trigger underruns. For a production system, it is better if no graphical
user interface is running.
In any case, it is required to evaluate a given system over several days if
reliable operation is to be proven.

The multiplexer ODR-DabMux mostly rearranges data internally, and doesn't do
much processing. Its resource requirements are low and it runs well on small
systems. The same goes for ODR-PadEnc, ODR-EDI2EDI, ODR-zmq2edi and ODR-zmq2farsync.

Audio encoding using ODR-AudioEnc is in-between ODR-DabMux and ODR-DabMod in
terms of resource usage, and running one encoder is not a problem even on small
embedded ARM boards.
However, you might want to run a dozen encoders on a single machine, where you
will have to plan for more headroom.

In general, for a robust 24/7 system, you should strive for a CPU usage below
50\%, regardless of which tools you are using. This gives you headroom for
monitoring, remote administration and background jobs run by cron.
Once your system is in operation, monitoring performance and observing logs is
essential to assess the health of your transmission.


\subsection{Launching the tools}

Services running in a production environment are usually administered remotely,
and must be able to run without user intervention, or connection. Traditionally,
such services are implemented (in UNIX terminology) as `daemons'. These are
started and stopped using the init system contained within the distribution.
As the ODR-mmbTools cannot daemonise themselves, a process supervisor is used.

\paragraph{supervisord}
One possibility is to use
\texttt{supervisord}\footnote{\url{http://supervisord.org}}
which can launch the tools and monitor their proper execution. It will
restart the processes and optionally inform the operator by email.

Once installed, supervisord reads its configuration file in
\texttt{/etc/supervisor.conf}
and launches the processes that are to be monitored. Each process is described
by a file. The following example assumes the tools are run as user \texttt{odr},
and that the multiplex configuration is in \texttt{/home/odr/config.mux}, and
that ODR-DabMux is to be launched.
The standard output and standard error streams of ODR-DabMux are written to the
specified log files.

\begin{lstlisting}
[program:ODR-DabMux]
command=odr-dabmux config.mux
directory=/home/odr
user=odr
autostart=true
autorestart=true
stdout_logfile=/home/odr/logs/mux.out.log
stderr_logfile=/home/odr/logs/mux.err.log
\end{lstlisting}

Once this configuration has been added to the supervisord configuration, the
settings have to be re-read using:
\begin{lstlisting}
supervisorctl reread
\end{lstlisting}

In order for supervisord to start managing and running this process, it needs to
be added:

\begin{lstlisting}
supervisorctl add ODR-DabMux
\end{lstlisting}

Setting up more processes (such as any of the other tools) can be easily
achieved by customising the configuration template above. Examples are provided
in the \texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder -
these need to be changed to reflect the paths that are in use on your system.

supervisord also includes a small web-server that can display the state of the
managed processes. It is enabled with the \verb+[inet_http_server]+ setting in
the configuration file.

\paragraph{systemd}
Most recent GNU/Linux distributions use \texttt{systemd} as init system, which
also can handle the supervision of processes. To achieve this, systemd unit
files have to be written for the tools. For more information, see the systemd
documentation.\sidenote{Give an example unit file}


\subsection{Logging}
Collecting information about events is essential within a production environment.
This information is essential forensic analysis, and tracing sources of trouble.
This is achieved through the logging of important messages that can be sent by
the tools.

ODR-DabMux and ODR-DabMod both support logging to standard error, to a file and
to the system logger \texttt{syslog}. Logging to syslog is the most flexible
approach; log information can be forwarded over the network to a
centralised logging server - where logs can then be filtered according to the
priority of each message. Both tools log to the LOCAL0 facility which in turn
can be redirected into an ODR-mmbtools specific log file.

\sidenote{Describe rsyslog configuration}

In order to avoid the log files from becoming undesirably large, \texttt{logrotate}
should be set to rotate the files automatically.

\sidenote{Describe logrotate configuration}


\subsection{Timing}
The ODR-mmbTools require the system time to be accurate in order for them to
function correctly - this is especially important when running a SFN, but is
also important for standalone transmitters when in a production environment. It
is also important to remember that most receivers have a clock that is
synchronised to the clock time which is being transmitted by the multiplex to
which it has been tuned.

The system needs to run a NTP client that synchronises the system time over the
network. Correct synchronisation can be checked using \texttt{chronyc tracking}
or the the \texttt{lpeers} command of the \texttt{ntpq} tool, depending on if
you use chrony or openntp.
The magnitude of the offset should be below $10$\ms.

The performance of the NTP synchronisation should also be monitored permanently
during operation.

ODR-DabMux can run a command at startup to verify if the NTP client is properly
working, using the \texttt{startupcheck} setting. This can be used to call
\texttt{ntp-wait} or \texttt{chronyc waitsync} to wait for proper NTP sync.

\subsection{Monitoring through SNMP}
There is ongoing work to make the monitoring of the tools possible using SNMP.
Please see \url{https://wiki.opendigitalradio.org/SNMP} for information about
this effort.

\subsection{Monitoring using munin}
\label{monitmunin}
The Munin\footnote{\url{http://munin-monitoring.org/}} monitoring tool can
create graphs for essential system health parameters. It can also send emails
if values transgress the defined bounds - this assists the operator in the
assessment of system status, as well as the health of the services.

In addition to basic system measurements like CPU, RAM and disk usage, NTP
synchronisation, disk and network performance, there are custom data sources
available for ODR-DabMux and ODR-DabMod.

The ODR-DabMux data sources include EDI and ZMQ input buffer monitoring (buffer
level, underruns and overruns) and the peak audio input levels (mono, or
stereo). The plugin for ODR-DabMod can monitor SDR device statistics among
others.

The plugins are written in python and are located in the \verb+doc+ folder in
the repositories. Copy them to \texttt{/etc/munin/plugins.d} and restart
munin-node. They require that the ODR-DabMux management
server, and the ZeroMQ remote control enabled on the ports give in the example
configurations.


\subsection{Real-time Scheduling}
As a general principle, it is prudent not to run tools (that do not need superuser
privileges) as the \texttt{root} user. The same principle also applies to the
ODR-mmbTools, but care has to be taken that the tools can still request real-time
scheduling when it is needed.

This is achieved by adding the following to \texttt{/etc/security/limits.conf},
assuming the tools are run under the user \texttt{odr}.

\begin{lstlisting}
odr          -       rtprio          65
odr          -       nice           -10
\end{lstlisting}

If you have installed JACK with real-time privileges, you may find this has
already been configured for the `audio' group, written as \texttt{@audio}, which
should suffice providing your desired user is a member of the `audio' group.

\subsection{Accessing the USRP as Non-root}

Superuser privileges are not required to access USB-connected USRP devices, but
sometimes the system lacks the configuration to enable normal users to
communicate with the device.
In that case, it is necessary to add a rule file for \texttt{udev}. This file is
included in the UHD sources, but might not have been automatically installed.
The file is called \texttt{10-uhd-usrp.rules}, should be placed into
\texttt{/etc/udev/rules.d/} and should contain

\subsection{Authentication Support}
In order to be able to use the Internet as contribution network, some form of
protection has to be put in place to make sure the audio data cannot be altered
by third parties. Usually, some form of VPN is set up for this case.

{ \footnotesize
\begin{lstlisting}
#USRP1
SUBSYSTEMS=="usb", ATTRS{idVendor}=="fffe", ATTRS{idProduct}=="0002", MODE:="0666"
#B100
SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0002", MODE:="0666"
#B200
SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0020", MODE:="0666"
\end{lstlisting}
}

% vim: spl=en spell tw=80 et