1 files changed, 165 insertions, 0 deletions
diff --git a/environments.tex b/environments.tex
new file mode 100644
index 0000000..594b7d6
--- /dev/null
+++ b/environments.tex
@@ -0,0 +1,165 @@
+\section{System Environment}
+
+In this section, we are going to discuss system requirements for a continuous
+operation of the tools. This environment differs in some regards to the one used
+for experiments and laboratory tests, given that monitoring, self-recovery in
+case of errors and resistance to failure are much more important in a 24/7
+operation. We will be using the term \emph{production environment} to refer to
+such a usage.
+
+\subsection{Launching the tools}
+
+Services running in a production environment are usually administered remotely,
+and have to be able to run without requiring a user to be connected all the
+time. Traditionally, such services are implemented as daemons in UNIX
+terminology, and are started and stopped using the init system of the
+distribution.
+The ODR-mmbTools cannot daemonise themselves, and require another approach.
+
+\paragraph{Screen multiplexer}
+An easy approach is to use a screen multiplexer, like \emph{GNU Screen} or
+\emph{tmux} that can be used to launch a session from which the user can detach
+and reattach. See the relevant manpages for more information.
+
+A screen multiplexer alone does permit the tools to run without a user
+connected, but does not automatically restart failed processes, and does not
+send any warning emails in case of a fault. The
+dab-scripts\footnote{The dab-scripts are available on the Opendigitalradio
+github.}, already mentioned in \ref{usingexistingwebstreams}, can be a helpful
+addition in that case. They can monitor that the processes are still running,
+and if necessary restart them.
+
+\paragraph{supervisord}
+The execution of the tools can also be supervised by a dedicated tool instead of
+using the scripts. \texttt{supervisor}\footnote{\url{http://supervisord.org}} is
+(its name will not surprise you) exactly such a tool.
+
+Once installed, it reads the configuration in \texttt{/etc/supervisor.conf} and
+launches the processes it has to monitor. Each process to monitor is described
+by a file. The following example assumes the tools run as user \texttt{odr}, and
+the multiplex configuration is in
+\texttt{/home/odr/config.mux}, and launches ODR-DabMux.
+The logs of ODR-DabMux is written to the log specified files.
+
+\begin{lstlisting}
+[program:ODR-DabMux]
+command=odr-dabmux config.mux
+directory=/home/odr
+user=odr
+autostart=true
+autorestart=true
+stdout_logfile=/home/odr/logs/mux.out.log
+stderr_logfile=/home/odr/logs/mux.err.log
+\end{lstlisting}
+
+Once this configuration has been added to the supervisor configuration, the
+settings have to be reread using:
+\begin{lstlisting}
+supervisorctl reread
+\end{lstlisting}
+
+In order for supervisor to start managing and running this process, it has to be added:
+\begin{lstlisting}
+supervisorctl add ODR-DabMux
+\end{lstlisting}
+
+Setting up more processes can be simply achieved by customising this
+configuration template for the other tools. Examples are available in the
+\texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder, and have
+to be customised to the paths used in your setup.
+
+supervisor also includes a small web-server that can display the state of the
+managed processes. It is enabled with the \verb+[inet_http_server]+ setting in
+the configuration file.
+
+\subsection{Logging}
+Essential for a production environment is traceability of events, which is
+achieved using logging of important messages.
+
+ODR-DabMux and ODR-DabMod both support logging to standard error, to a file and
+to the system logger \texttt{syslog}. Logging to syslog is the most flexible
+solution, because the log information can be forwarded over the network to a
+centralised logging server, and can be filtered according to the priority of
+each message. Both tools log to the LOCAL0 facility, which can be redirected to
+a file, in order to group ODR-mmbTools-related messages together.
+
+%\sidenote{Describe rsyslog configuration}
+
+In order to avoid that the log files grow over time to unreasonable sizes,
+\texttt{logrotate} should be setup to rotate the files automatically.
+%\sidenote{Describe logrotate configuration}
+
+
+\subsection{Timing}
+The ODR-mmbTools require a correct system time to function properly. This is
+especially important when running a SFN, but cannot be neglected for a proper
+production operation even with a single transmitter.
+
+The system needs to run a NTP client that synchronises the system time over the
+network. Correct synchronisation can be checked using the \texttt{lpeers}
+command of the \texttt{ntpq} tool. The magnitude of the offset, indicated in
+milliseconds, should be below 10.
+
+The performance of the NTP synchronisation should also be monitored permanently
+during operation.
+
+
+\subsection{Monitoring using munin}
+
+The Munin\footnote{\url{http://munin-monitoring.org/}} monitoring tool can
+create graphs for essential system health parameters, and can also send emails
+in case a specific value exceeds its allowed bounds. This helps the operator in
+assessing the system status and the well-functioning of the services.
+
+In addition to basic system measurements like CPU, RAM and disk usage, NTP
+synchronisation, disk and network performance and many other information, there
+are also custom data sources for ODR-DabMux.
+
+These data sources include ZMQ input buffer monitoring (buffer level, underruns
+and overruns) and input audio level (peak for both channels). This data source
+can be installed by copying \verb+doc/stats_dabmux_multi.py+ to
+\texttt{/etc/munin/plugins.d}. They require that the ODR-DabMux management
+server is enabled in the configuration, and will automatically generate the
+graphs for the subchannels used in the configuration.
+
+
+\subsection{Real-time Scheduling}
+
+As a general principle, it is advisable to run tools that do not need superuser
+privileges under another user than \texttt{root}. The same principle also
+applies to the ODR-mmbTools, where care has to be taken that the tools can
+request real-time scheduling when needed.
+
+This is achieved by adding the following to \texttt{/etc/security/limits.conf},
+assuming the tools are run under the user \texttt{odr}.
+
+\begin{lstlisting}
+odr          -       rtprio          65
+odr          -       nice           -10
+\end{lstlisting}
+
+If you have installed JACK with real-time privileges, you might already find the
+same setting, but for the audio group, written as \texttt{@audio}, which is
+sufficient.
+
+\subsection{Accessing the USRP as Non-root}
+
+Superuser privileges are not required to access USB-connected USRP devices, but
+sometimes the system lacks the configuration to enable normal users to
+communicate to the device.
+
+In that case, it is necessary to add a rule file for \texttt{udev}. This file is
+included in the UHD sources, but might not have been automatically installed.
+
+The file is called \texttt{uhd-usrp.rules}, should be placed into
+\texttt{/etc/udev/rules.d/} and should contain
+\begin{lstlisting}
+#USRP1
+SUBSYSTEMS=="usb", ATTRS{idVendor}=="fffe", ATTRS{idProduct}=="0002", MODE:="0666"
+#B100
+SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0002", MODE:="0666"
+#B200
+SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0020", MODE:="0666"
+\end{lstlising}
+
+% vim: spl=en spell tw=80 et