diff options
| -rw-r--r-- | environments.tex | 165 | ||||
| -rw-r--r-- | systemenvironments.tex | 4 |
2 files changed, 2 insertions, 167 deletions
diff --git a/environments.tex b/environments.tex deleted file mode 100644 index 594b7d6..0000000 --- a/environments.tex +++ /dev/null @@ -1,165 +0,0 @@ -\section{System Environment} - -In this section, we are going to discuss system requirements for a continuous -operation of the tools. This environment differs in some regards to the one used -for experiments and laboratory tests, given that monitoring, self-recovery in -case of errors and resistance to failure are much more important in a 24/7 -operation. We will be using the term \emph{production environment} to refer to -such a usage. - -\subsection{Launching the tools} - -Services running in a production environment are usually administered remotely, -and have to be able to run without requiring a user to be connected all the -time. Traditionally, such services are implemented as daemons in UNIX -terminology, and are started and stopped using the init system of the -distribution. -The ODR-mmbTools cannot daemonise themselves, and require another approach. - -\paragraph{Screen multiplexer} -An easy approach is to use a screen multiplexer, like \emph{GNU Screen} or -\emph{tmux} that can be used to launch a session from which the user can detach -and reattach. See the relevant manpages for more information. - -A screen multiplexer alone does permit the tools to run without a user -connected, but does not automatically restart failed processes, and does not -send any warning emails in case of a fault. The -dab-scripts\footnote{The dab-scripts are available on the Opendigitalradio -github.}, already mentioned in \ref{usingexistingwebstreams}, can be a helpful -addition in that case. They can monitor that the processes are still running, -and if necessary restart them. - -\paragraph{supervisord} -The execution of the tools can also be supervised by a dedicated tool instead of -using the scripts. \texttt{supervisor}\footnote{\url{http://supervisord.org}} is -(its name will not surprise you) exactly such a tool. - -Once installed, it reads the configuration in \texttt{/etc/supervisor.conf} and -launches the processes it has to monitor. Each process to monitor is described -by a file. The following example assumes the tools run as user \texttt{odr}, and -the multiplex configuration is in -\texttt{/home/odr/config.mux}, and launches ODR-DabMux. -The logs of ODR-DabMux is written to the log specified files. - -\begin{lstlisting} -[program:ODR-DabMux] -command=odr-dabmux config.mux -directory=/home/odr -user=odr -autostart=true -autorestart=true -stdout_logfile=/home/odr/logs/mux.out.log -stderr_logfile=/home/odr/logs/mux.err.log -\end{lstlisting} - -Once this configuration has been added to the supervisor configuration, the -settings have to be reread using: -\begin{lstlisting} -supervisorctl reread -\end{lstlisting} - -In order for supervisor to start managing and running this process, it has to be added: -\begin{lstlisting} -supervisorctl add ODR-DabMux -\end{lstlisting} - -Setting up more processes can be simply achieved by customising this -configuration template for the other tools. Examples are available in the -\texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder, and have -to be customised to the paths used in your setup. - -supervisor also includes a small web-server that can display the state of the -managed processes. It is enabled with the \verb+[inet_http_server]+ setting in -the configuration file. - -\subsection{Logging} -Essential for a production environment is traceability of events, which is -achieved using logging of important messages. - -ODR-DabMux and ODR-DabMod both support logging to standard error, to a file and -to the system logger \texttt{syslog}. Logging to syslog is the most flexible -solution, because the log information can be forwarded over the network to a -centralised logging server, and can be filtered according to the priority of -each message. Both tools log to the LOCAL0 facility, which can be redirected to -a file, in order to group ODR-mmbTools-related messages together. - -%\sidenote{Describe rsyslog configuration} - -In order to avoid that the log files grow over time to unreasonable sizes, -\texttt{logrotate} should be setup to rotate the files automatically. -%\sidenote{Describe logrotate configuration} - - -\subsection{Timing} -The ODR-mmbTools require a correct system time to function properly. This is -especially important when running a SFN, but cannot be neglected for a proper -production operation even with a single transmitter. - -The system needs to run a NTP client that synchronises the system time over the -network. Correct synchronisation can be checked using the \texttt{lpeers} -command of the \texttt{ntpq} tool. The magnitude of the offset, indicated in -milliseconds, should be below 10. - -The performance of the NTP synchronisation should also be monitored permanently -during operation. - - -\subsection{Monitoring using munin} - -The Munin\footnote{\url{http://munin-monitoring.org/}} monitoring tool can -create graphs for essential system health parameters, and can also send emails -in case a specific value exceeds its allowed bounds. This helps the operator in -assessing the system status and the well-functioning of the services. - -In addition to basic system measurements like CPU, RAM and disk usage, NTP -synchronisation, disk and network performance and many other information, there -are also custom data sources for ODR-DabMux. - -These data sources include ZMQ input buffer monitoring (buffer level, underruns -and overruns) and input audio level (peak for both channels). This data source -can be installed by copying \verb+doc/stats_dabmux_multi.py+ to -\texttt{/etc/munin/plugins.d}. They require that the ODR-DabMux management -server is enabled in the configuration, and will automatically generate the -graphs for the subchannels used in the configuration. - - -\subsection{Real-time Scheduling} - -As a general principle, it is advisable to run tools that do not need superuser -privileges under another user than \texttt{root}. The same principle also -applies to the ODR-mmbTools, where care has to be taken that the tools can -request real-time scheduling when needed. - -This is achieved by adding the following to \texttt{/etc/security/limits.conf}, -assuming the tools are run under the user \texttt{odr}. - -\begin{lstlisting} -odr - rtprio 65 -odr - nice -10 -\end{lstlisting} - -If you have installed JACK with real-time privileges, you might already find the -same setting, but for the audio group, written as \texttt{@audio}, which is -sufficient. - -\subsection{Accessing the USRP as Non-root} - -Superuser privileges are not required to access USB-connected USRP devices, but -sometimes the system lacks the configuration to enable normal users to -communicate to the device. - -In that case, it is necessary to add a rule file for \texttt{udev}. This file is -included in the UHD sources, but might not have been automatically installed. - -The file is called \texttt{uhd-usrp.rules}, should be placed into -\texttt{/etc/udev/rules.d/} and should contain -\begin{lstlisting} -#USRP1 -SUBSYSTEMS=="usb", ATTRS{idVendor}=="fffe", ATTRS{idProduct}=="0002", MODE:="0666" -#B100 -SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0002", MODE:="0666" -#B200 -SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0020", MODE:="0666" -\end{lstlising} - -% vim: spl=en spell tw=80 et diff --git a/systemenvironments.tex b/systemenvironments.tex index d9cf5ff..b47c85c 100644 --- a/systemenvironments.tex +++ b/systemenvironments.tex @@ -83,11 +83,11 @@ centralised logging server, and can be filtered according to the priority of each message. Both tools log to the LOCAL0 facility, which can be redirected to a file, in order to group ODR-mmbTools-related messages together. -%\sidenote{Describe rsyslog configuration} +\sidenote{Describe rsyslog configuration} In order to avoid that the log files grow over time to unreasonable sizes, \texttt{logrotate} should be setup to rotate the files automatically. -%\sidenote{Describe logrotate configuration} +\sidenote{Describe logrotate configuration} \subsection{Timing} |