diff options
| -rw-r--r-- | Makefile | 3 | ||||
| -rw-r--r-- | environments.tex | 165 | ||||
| -rw-r--r-- | mmbtools.tex | 1 | ||||
| -rw-r--r-- | scenarios.tex | 6 | ||||
| -rw-r--r-- | systemenvironments.tex | 167 |
5 files changed, 338 insertions, 4 deletions
@@ -2,7 +2,8 @@ latexfile = mmbtools includes = introduction.tex \ interfaces.tex \ scenarios.tex \ - appendix.tex + appendix.tex \ + systemenvironments.tex bib = dab.bib diff --git a/environments.tex b/environments.tex new file mode 100644 index 0000000..594b7d6 --- /dev/null +++ b/environments.tex @@ -0,0 +1,165 @@ +\section{System Environment} + +In this section, we are going to discuss system requirements for a continuous +operation of the tools. This environment differs in some regards to the one used +for experiments and laboratory tests, given that monitoring, self-recovery in +case of errors and resistance to failure are much more important in a 24/7 +operation. We will be using the term \emph{production environment} to refer to +such a usage. + +\subsection{Launching the tools} + +Services running in a production environment are usually administered remotely, +and have to be able to run without requiring a user to be connected all the +time. Traditionally, such services are implemented as daemons in UNIX +terminology, and are started and stopped using the init system of the +distribution. +The ODR-mmbTools cannot daemonise themselves, and require another approach. + +\paragraph{Screen multiplexer} +An easy approach is to use a screen multiplexer, like \emph{GNU Screen} or +\emph{tmux} that can be used to launch a session from which the user can detach +and reattach. See the relevant manpages for more information. + +A screen multiplexer alone does permit the tools to run without a user +connected, but does not automatically restart failed processes, and does not +send any warning emails in case of a fault. The +dab-scripts\footnote{The dab-scripts are available on the Opendigitalradio +github.}, already mentioned in \ref{usingexistingwebstreams}, can be a helpful +addition in that case. They can monitor that the processes are still running, +and if necessary restart them. + +\paragraph{supervisord} +The execution of the tools can also be supervised by a dedicated tool instead of +using the scripts. \texttt{supervisor}\footnote{\url{http://supervisord.org}} is +(its name will not surprise you) exactly such a tool. + +Once installed, it reads the configuration in \texttt{/etc/supervisor.conf} and +launches the processes it has to monitor. Each process to monitor is described +by a file. The following example assumes the tools run as user \texttt{odr}, and +the multiplex configuration is in +\texttt{/home/odr/config.mux}, and launches ODR-DabMux. +The logs of ODR-DabMux is written to the log specified files. + +\begin{lstlisting} +[program:ODR-DabMux] +command=odr-dabmux config.mux +directory=/home/odr +user=odr +autostart=true +autorestart=true +stdout_logfile=/home/odr/logs/mux.out.log +stderr_logfile=/home/odr/logs/mux.err.log +\end{lstlisting} + +Once this configuration has been added to the supervisor configuration, the +settings have to be reread using: +\begin{lstlisting} +supervisorctl reread +\end{lstlisting} + +In order for supervisor to start managing and running this process, it has to be added: +\begin{lstlisting} +supervisorctl add ODR-DabMux +\end{lstlisting} + +Setting up more processes can be simply achieved by customising this +configuration template for the other tools. Examples are available in the +\texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder, and have +to be customised to the paths used in your setup. + +supervisor also includes a small web-server that can display the state of the +managed processes. It is enabled with the \verb+[inet_http_server]+ setting in +the configuration file. + +\subsection{Logging} +Essential for a production environment is traceability of events, which is +achieved using logging of important messages. + +ODR-DabMux and ODR-DabMod both support logging to standard error, to a file and +to the system logger \texttt{syslog}. Logging to syslog is the most flexible +solution, because the log information can be forwarded over the network to a +centralised logging server, and can be filtered according to the priority of +each message. Both tools log to the LOCAL0 facility, which can be redirected to +a file, in order to group ODR-mmbTools-related messages together. + +%\sidenote{Describe rsyslog configuration} + +In order to avoid that the log files grow over time to unreasonable sizes, +\texttt{logrotate} should be setup to rotate the files automatically. +%\sidenote{Describe logrotate configuration} + + +\subsection{Timing} +The ODR-mmbTools require a correct system time to function properly. This is +especially important when running a SFN, but cannot be neglected for a proper +production operation even with a single transmitter. + +The system needs to run a NTP client that synchronises the system time over the +network. Correct synchronisation can be checked using the \texttt{lpeers} +command of the \texttt{ntpq} tool. The magnitude of the offset, indicated in +milliseconds, should be below 10. + +The performance of the NTP synchronisation should also be monitored permanently +during operation. + + +\subsection{Monitoring using munin} + +The Munin\footnote{\url{http://munin-monitoring.org/}} monitoring tool can +create graphs for essential system health parameters, and can also send emails +in case a specific value exceeds its allowed bounds. This helps the operator in +assessing the system status and the well-functioning of the services. + +In addition to basic system measurements like CPU, RAM and disk usage, NTP +synchronisation, disk and network performance and many other information, there +are also custom data sources for ODR-DabMux. + +These data sources include ZMQ input buffer monitoring (buffer level, underruns +and overruns) and input audio level (peak for both channels). This data source +can be installed by copying \verb+doc/stats_dabmux_multi.py+ to +\texttt{/etc/munin/plugins.d}. They require that the ODR-DabMux management +server is enabled in the configuration, and will automatically generate the +graphs for the subchannels used in the configuration. + + +\subsection{Real-time Scheduling} + +As a general principle, it is advisable to run tools that do not need superuser +privileges under another user than \texttt{root}. The same principle also +applies to the ODR-mmbTools, where care has to be taken that the tools can +request real-time scheduling when needed. + +This is achieved by adding the following to \texttt{/etc/security/limits.conf}, +assuming the tools are run under the user \texttt{odr}. + +\begin{lstlisting} +odr - rtprio 65 +odr - nice -10 +\end{lstlisting} + +If you have installed JACK with real-time privileges, you might already find the +same setting, but for the audio group, written as \texttt{@audio}, which is +sufficient. + +\subsection{Accessing the USRP as Non-root} + +Superuser privileges are not required to access USB-connected USRP devices, but +sometimes the system lacks the configuration to enable normal users to +communicate to the device. + +In that case, it is necessary to add a rule file for \texttt{udev}. This file is +included in the UHD sources, but might not have been automatically installed. + +The file is called \texttt{uhd-usrp.rules}, should be placed into +\texttt{/etc/udev/rules.d/} and should contain +\begin{lstlisting} +#USRP1 +SUBSYSTEMS=="usb", ATTRS{idVendor}=="fffe", ATTRS{idProduct}=="0002", MODE:="0666" +#B100 +SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0002", MODE:="0666" +#B200 +SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0020", MODE:="0666" +\end{lstlising} + +% vim: spl=en spell tw=80 et diff --git a/mmbtools.tex b/mmbtools.tex index 6632e67..68343fc 100644 --- a/mmbtools.tex +++ b/mmbtools.tex @@ -199,6 +199,7 @@ clock_config_t}, \include{introduction} \include{interfaces} \include{scenarios} +\include{systemenvironments} \appendix \include{appendix} diff --git a/scenarios.tex b/scenarios.tex index 3de9b23..cd5f4f3 100644 --- a/scenarios.tex +++ b/scenarios.tex @@ -108,9 +108,8 @@ transmit synchronisation necessary for SFN and is robust enough for 24/7 operation. \begin{wrapfigure}{r}{13em} - \begin{center} - \includegraphics[width=12em]{figures/USRP-B200.jpg} - \end{center} + \centering + \includegraphics[width=12em]{figures/USRP-B200.jpg} \caption{Ettus USRP B200} \label{fig:usrp-b200} \end{wrapfigure} @@ -282,6 +281,7 @@ FDK-AAC-DABplus supports JACK and ALSA input, but Toolame-dab supports only the JACK input. \subsubsection{Using Existing Web-Streams} +\label{usingexistingwebstreams} One common scenario is to transmit radio stations that already are available as web-radio streams. For simplicity, it makes sense to get these web streams, which are most often encoded in mp3 and available through HTTP, decode them, and diff --git a/systemenvironments.tex b/systemenvironments.tex new file mode 100644 index 0000000..d9cf5ff --- /dev/null +++ b/systemenvironments.tex @@ -0,0 +1,167 @@ +\section{System Environment} + +In this section, we are going to discuss system requirements for a continuous +operation of the tools. This environment differs in some regards to the one used +for experiments and laboratory tests, given that monitoring, self-recovery in +case of errors and resistance to failure are much more important in a 24/7 +operation. We will be using the term \emph{production environment} to refer to +such a usage. + +\subsection{Launching the tools} + +Services running in a production environment are usually administered remotely, +and have to be able to run without requiring a user to be connected all the +time. Traditionally, such services are implemented as daemons in UNIX +terminology, and are started and stopped using the init system of the +distribution. +The ODR-mmbTools cannot daemonise themselves, and require another approach. + +\paragraph{Screen multiplexer} +An easy approach is to use a screen multiplexer, like \emph{GNU Screen} or +\emph{tmux} that can be used to launch a session from which the user can detach +and reattach. See the relevant manpages for more information. + +A screen multiplexer alone does permit the tools to run without a user +connected, but does not automatically restart failed processes, and does not +send any warning emails in case of a fault. The +dab-scripts\footnote{The dab-scripts are available on the Opendigitalradio +github.}, already mentioned in \ref{usingexistingwebstreams}, can be a helpful +addition in that case. They can monitor that the processes are still running, +and if necessary restart them. + +\paragraph{supervisord} +The execution of the tools can also be supervised by a dedicated tool instead of +using the scripts. \texttt{supervisor}\footnote{\url{http://supervisord.org}} is +(its name will not surprise you) exactly such a tool. + +Once installed, it reads the configuration in \texttt{/etc/supervisor.conf} and +launches the processes it has to monitor. Each process to monitor is described +by a file. The following example assumes the tools run as user \texttt{odr}, and +the multiplex configuration is in +\texttt{/home/odr/config.mux}, and launches ODR-DabMux. +The logs of ODR-DabMux is written to the log specified files. + +\begin{lstlisting} +[program:ODR-DabMux] +command=odr-dabmux config.mux +directory=/home/odr +user=odr +autostart=true +autorestart=true +stdout_logfile=/home/odr/logs/mux.out.log +stderr_logfile=/home/odr/logs/mux.err.log +\end{lstlisting} + +Once this configuration has been added to the supervisor configuration, the +settings have to be reread using: +\begin{lstlisting} +supervisorctl reread +\end{lstlisting} + +In order for supervisor to start managing and running this process, it has to be added: +\begin{lstlisting} +supervisorctl add ODR-DabMux +\end{lstlisting} + +Setting up more processes can be simply achieved by customising this +configuration template for the other tools. Examples are available in the +\texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder, and have +to be customised to the paths used in your setup. + +supervisor also includes a small web-server that can display the state of the +managed processes. It is enabled with the \verb+[inet_http_server]+ setting in +the configuration file. + +\subsection{Logging} +Essential for a production environment is traceability of events, which is +achieved using logging of important messages. + +ODR-DabMux and ODR-DabMod both support logging to standard error, to a file and +to the system logger \texttt{syslog}. Logging to syslog is the most flexible +solution, because the log information can be forwarded over the network to a +centralised logging server, and can be filtered according to the priority of +each message. Both tools log to the LOCAL0 facility, which can be redirected to +a file, in order to group ODR-mmbTools-related messages together. + +%\sidenote{Describe rsyslog configuration} + +In order to avoid that the log files grow over time to unreasonable sizes, +\texttt{logrotate} should be setup to rotate the files automatically. +%\sidenote{Describe logrotate configuration} + + +\subsection{Timing} +The ODR-mmbTools require a correct system time to function properly. This is +especially important when running a SFN, but cannot be neglected for a proper +production operation even with a single transmitter. + +The system needs to run a NTP client that synchronises the system time over the +network. Correct synchronisation can be checked using the \texttt{lpeers} +command of the \texttt{ntpq} tool. The magnitude of the offset, indicated in +milliseconds, should be below 10. + +The performance of the NTP synchronisation should also be monitored permanently +during operation. + + +\subsection{Monitoring using munin} + +The Munin\footnote{\url{http://munin-monitoring.org/}} monitoring tool can +create graphs for essential system health parameters, and can also send emails +in case a specific value exceeds its allowed bounds. This helps the operator in +assessing the system status and the well-functioning of the services. + +In addition to basic system measurements like CPU, RAM and disk usage, NTP +synchronisation, disk and network performance and many other information, there +are also custom data sources for ODR-DabMux. + +These data sources include ZMQ input buffer monitoring (buffer level, underruns +and overruns) and input audio level (peak for both channels). This data source +can be installed by copying \verb+doc/stats_dabmux_multi.py+ to +\texttt{/etc/munin/plugins.d}. They require that the ODR-DabMux management +server is enabled in the configuration, and will automatically generate the +graphs for the subchannels used in the configuration. + + +\subsection{Real-time Scheduling} + +As a general principle, it is advisable to run tools that do not need superuser +privileges under another user than \texttt{root}. The same principle also +applies to the ODR-mmbTools, where care has to be taken that the tools can +request real-time scheduling when needed. + +This is achieved by adding the following to \texttt{/etc/security/limits.conf}, +assuming the tools are run under the user \texttt{odr}. + +\begin{lstlisting} +odr - rtprio 65 +odr - nice -10 +\end{lstlisting} + +If you have installed JACK with real-time privileges, you might already find the +same setting, but for the audio group, written as \texttt{@audio}, which is +sufficient. + +\subsection{Accessing the USRP as Non-root} + +Superuser privileges are not required to access USB-connected USRP devices, but +sometimes the system lacks the configuration to enable normal users to +communicate to the device. + +In that case, it is necessary to add a rule file for \texttt{udev}. This file is +included in the UHD sources, but might not have been automatically installed. + +The file is called \texttt{uhd-usrp.rules}, should be placed into +\texttt{/etc/udev/rules.d/} and should contain +{ \footnotesize +\begin{verbatim} +#USRP1 +SUBSYSTEMS=="usb", ATTRS{idVendor}=="fffe", ATTRS{idProduct}=="0002", MODE:="0666" +#B100 +SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0002", MODE:="0666" +#B200 +SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0020", MODE:="0666" +\end{verbatim} +} + +% vim: spl=en spell tw=80 et |