diff options
| -rw-r--r-- | systemenvironments.tex | 141 |
1 files changed, 75 insertions, 66 deletions
diff --git a/systemenvironments.tex b/systemenvironments.tex index b47c85c..b319c30 100644 --- a/systemenvironments.tex +++ b/systemenvironments.tex @@ -1,45 +1,46 @@ \section{System Environment} -In this section, we are going to discuss system requirements for a continuous -operation of the tools. This environment differs in some regards to the one used -for experiments and laboratory tests, given that monitoring, self-recovery in -case of errors and resistance to failure are much more important in a 24/7 -operation. We will be using the term \emph{production environment} to refer to -such a usage. +In this section, we describe the system configuration requirements for the +continuous operation of the tools. The production environment differs in some +respects to those used for experimentation and in laboratory testing. Monitoring, +automatic recovery (in case of errors) and resiliance are crucial in 24/7 +operations. The term \emph{production environment} will be used here to refer to +such use. \subsection{Launching the tools} Services running in a production environment are usually administered remotely, -and have to be able to run without requiring a user to be connected all the -time. Traditionally, such services are implemented as daemons in UNIX -terminology, and are started and stopped using the init system of the -distribution. -The ODR-mmbTools cannot daemonise themselves, and require another approach. +and must be able to run without user intervention, or connection. Traditionally, +such services are implemented (in UNIX terminology) as 'daemons'. These are +started and stopped using the init system contained within the distribution. +The ODR-mmbTools cannot daemonise themselves, so this requires another approach: \paragraph{Screen multiplexer} -An easy approach is to use a screen multiplexer, like \emph{GNU Screen} or -\emph{tmux} that can be used to launch a session from which the user can detach -and reattach. See the relevant manpages for more information. - -A screen multiplexer alone does permit the tools to run without a user -connected, but does not automatically restart failed processes, and does not -send any warning emails in case of a fault. The -dab-scripts\footnote{The dab-scripts are available on the Opendigitalradio -github.}, already mentioned in \ref{usingexistingwebstreams}, can be a helpful -addition in that case. They can monitor that the processes are still running, -and if necessary restart them. +A simple approach is to use a screen multiplexer such as \emph{GNU Screen} or +\emph{tmux} - either of these can be used to launch a session from which the +user can detach, and later re-attach at will - leaving the tools running within +it. Please see the relevant manpages for more information. + +Although a screen multiplexer alone permits the tools to run without a user +being connected, it alone cannot automatically restart failed processes, and it +is unable to provide warnings in the case of a problem. + +The dab-scripts \footnote{The dab-scripts are available on the Opendigitalradio +github.} ,already mentioned in \ref{usingexistingwebstreams}, can be employed to +monitor the processes and (if necessary) restart them, and send an alert via +email. \paragraph{supervisord} -The execution of the tools can also be supervised by a dedicated tool instead of -using the scripts. \texttt{supervisor}\footnote{\url{http://supervisord.org}} is -(its name will not surprise you) exactly such a tool. +As an alternative to using the scripts, the execution of the tools can also be +carried out by a dedicated tool. \texttt{supervisor}\footnote{\url{http://supervisord.org}} +is (as the name implies) such a tool. -Once installed, it reads the configuration in \texttt{/etc/supervisor.conf} and -launches the processes it has to monitor. Each process to monitor is described -by a file. The following example assumes the tools run as user \texttt{odr}, and -the multiplex configuration is in -\texttt{/home/odr/config.mux}, and launches ODR-DabMux. -The logs of ODR-DabMux is written to the log specified files. +Once installed, supervisor reads its configuration file in \texttt{/etc/supervisor.conf} +and launches the processes that are to be monitored. Each process is described +by a file. The following example assumes the tools are run as user \texttt{odr}, +and that the multiplex configuration is in \texttt{/home/odr/config.mux}, and +that ODR-DabMux is to be launched.The logs of ODR-DabMux is written to the +specified log files. \begin{lstlisting} [program:ODR-DabMux] @@ -53,52 +54,60 @@ stderr_logfile=/home/odr/logs/mux.err.log \end{lstlisting} Once this configuration has been added to the supervisor configuration, the -settings have to be reread using: +settings have to be re-read using: \begin{lstlisting} supervisorctl reread \end{lstlisting} -In order for supervisor to start managing and running this process, it has to be added: +In order for supervisor to start managing and running this process, it needs to +be added: + \begin{lstlisting} supervisorctl add ODR-DabMux \end{lstlisting} -Setting up more processes can be simply achieved by customising this -configuration template for the other tools. Examples are available in the -\texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder, and have -to be customised to the paths used in your setup. +Setting up more processes (such as any of the other tools) can be easily +achieved by customising the configuration template above. Examples are provided +in the \texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder - +these need to be changed to reflect the paths that are in use on your system. supervisor also includes a small web-server that can display the state of the managed processes. It is enabled with the \verb+[inet_http_server]+ setting in the configuration file. \subsection{Logging} -Essential for a production environment is traceability of events, which is -achieved using logging of important messages. +Collecting information about events is essential within a production environment. +This information is essential forensic analysis, and tracing sources of trouble. +This is achieved through the logging of important messages that can be sent by +the tools. ODR-DabMux and ODR-DabMod both support logging to standard error, to a file and to the system logger \texttt{syslog}. Logging to syslog is the most flexible -solution, because the log information can be forwarded over the network to a -centralised logging server, and can be filtered according to the priority of -each message. Both tools log to the LOCAL0 facility, which can be redirected to -a file, in order to group ODR-mmbTools-related messages together. +approach; log information can be forwarded over the network to a +centralised logging server - where logs can then be filtered according to the +priority of each message. Both tools log to the LOCAL0 facility which in turn +can be redirected into an ODR-mmbtools specific log file. \sidenote{Describe rsyslog configuration} -In order to avoid that the log files grow over time to unreasonable sizes, -\texttt{logrotate} should be setup to rotate the files automatically. +In order to avoid the log files from becomming undesirably large, \texttt{logrotate} +should be set to rotate the files automatically. + \sidenote{Describe logrotate configuration} \subsection{Timing} -The ODR-mmbTools require a correct system time to function properly. This is -especially important when running a SFN, but cannot be neglected for a proper -production operation even with a single transmitter. +The ODR-mmbTools require the system time to be accurate in order for them to +function correctly - this is especially important when running a SFN, but is +also important for standalone transmitters when in a production environment. It +is also important to remember that most receivers have a clock that is +synchronised to the clock time which is being transmitted by the multiplex to +which it has been tuned. The system needs to run a NTP client that synchronises the system time over the network. Correct synchronisation can be checked using the \texttt{lpeers} -command of the \texttt{ntpq} tool. The magnitude of the offset, indicated in -milliseconds, should be below 10. +command of the \texttt{ntpq} tool. The magnitude of the offset should be below +10 mS. The performance of the NTP synchronisation should also be monitored permanently during operation. @@ -107,17 +116,17 @@ during operation. \subsection{Monitoring using munin} The Munin\footnote{\url{http://munin-monitoring.org/}} monitoring tool can -create graphs for essential system health parameters, and can also send emails -in case a specific value exceeds its allowed bounds. This helps the operator in -assessing the system status and the well-functioning of the services. +create graphs for essential system health parameters. It can also send emails +if values transgress the defined bounds - this assists the operator in the +assessment of system status, as well as the health of the services. In addition to basic system measurements like CPU, RAM and disk usage, NTP -synchronisation, disk and network performance and many other information, there -are also custom data sources for ODR-DabMux. +synchronisation, disk and network performance (and much more besides), there +are also custom data sources for ODR-DabMux; These data sources include ZMQ input buffer monitoring (buffer level, underruns -and overruns) and input audio level (peak for both channels). This data source -can be installed by copying \verb+doc/stats_dabmux_multi.py+ to +and overruns) and the peak audio input levels (mono, or stereo). This data +source can be installed by copying \verb+doc/stats_dabmux_multi.py+ to \texttt{/etc/munin/plugins.d}. They require that the ODR-DabMux management server is enabled in the configuration, and will automatically generate the graphs for the subchannels used in the configuration. @@ -125,10 +134,10 @@ graphs for the subchannels used in the configuration. \subsection{Real-time Scheduling} -As a general principle, it is advisable to run tools that do not need superuser -privileges under another user than \texttt{root}. The same principle also -applies to the ODR-mmbTools, where care has to be taken that the tools can -request real-time scheduling when needed. +As a general principle, it is prudent not to run tools (that do not need superuser +privileges) as the \texttt{root} user. The same principle also applies to the +ODR-mmbTools, but care has to be taken that the tools can still request real-time +scheduling when it is needed. This is achieved by adding the following to \texttt{/etc/security/limits.conf}, assuming the tools are run under the user \texttt{odr}. @@ -138,20 +147,20 @@ odr - rtprio 65 odr - nice -10 \end{lstlisting} -If you have installed JACK with real-time privileges, you might already find the -same setting, but for the audio group, written as \texttt{@audio}, which is -sufficient. +If you have installed JACK with real-time privileges, you may find this has +already been configured for the 'audio' group, written as \texttt{@audio}, which +should suffice providing your desired user is a member of the @audio group. \subsection{Accessing the USRP as Non-root} Superuser privileges are not required to access USB-connected USRP devices, but sometimes the system lacks the configuration to enable normal users to -communicate to the device. +communicate with the device. In that case, it is necessary to add a rule file for \texttt{udev}. This file is included in the UHD sources, but might not have been automatically installed. -The file is called \texttt{uhd-usrp.rules}, should be placed into +The file is called \texttt{10-uhd-usrp.rules}, should be placed into \texttt{/etc/udev/rules.d/} and should contain { \footnotesize \begin{verbatim} |