1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
|
\section{System Environment}
In this section, we describe the system configuration requirements for the
continuous operation of the tools. The production environment differs in some
respects to those used for experimentation and in laboratory testing. Monitoring,
automatic recovery (in case of errors) and resilience are crucial in 24/7
operations. The term \emph{production environment} will be used here to refer to
such use.
\subsection{Launching the tools}
Services running in a production environment are usually administered remotely,
and must be able to run without user intervention, or connection. Traditionally,
such services are implemented (in UNIX terminology) as `daemons'. These are
started and stopped using the init system contained within the distribution.
The ODR-mmbTools cannot daemonise themselves, so this requires another approach:
\paragraph{Screen multiplexer}
A simple approach is to use a screen multiplexer such as \emph{GNU Screen} or
\emph{tmux} - either of these can be used to launch a session from which the
user can detach, and later re-attach at will - leaving the tools running within
it. Please see the relevant manpages for more information.
Although a screen multiplexer alone permits the tools to run without a user
being connected, it alone cannot automatically restart failed processes, and it
is unable to provide warnings in the case of a problem.
The dab-scripts, already mentioned in \ref{usingexistingwebstreams}, can be
employed to monitor the processes and (if necessary) restart them, and send an
alert via email.
\paragraph{supervisord}
As an alternative to using the scripts, the execution of the tools can also be
carried out by a dedicated tool. \texttt{supervisor}\footnote{\url{http://supervisord.org}}
is (as the name implies) such a tool.
Once installed, supervisor reads its configuration file in \texttt{/etc/supervisor.conf}
and launches the processes that are to be monitored. Each process is described
by a file. The following example assumes the tools are run as user \texttt{odr},
and that the multiplex configuration is in \texttt{/home/odr/config.mux}, and
that ODR-DabMux is to be launched.The logs of ODR-DabMux is written to the
specified log files.
\begin{lstlisting}
[program:ODR-DabMux]
command=odr-dabmux config.mux
directory=/home/odr
user=odr
autostart=true
autorestart=true
stdout_logfile=/home/odr/logs/mux.out.log
stderr_logfile=/home/odr/logs/mux.err.log
\end{lstlisting}
Once this configuration has been added to the supervisor configuration, the
settings have to be re-read using:
\begin{lstlisting}
supervisorctl reread
\end{lstlisting}
In order for supervisor to start managing and running this process, it needs to
be added:
\begin{lstlisting}
supervisorctl add ODR-DabMux
\end{lstlisting}
Setting up more processes (such as any of the other tools) can be easily
achieved by customising the configuration template above. Examples are provided
in the \texttt{mmbtools-aux} repository, under the \texttt{supervisor} folder -
these need to be changed to reflect the paths that are in use on your system.
supervisor also includes a small web-server that can display the state of the
managed processes. It is enabled with the \verb+[inet_http_server]+ setting in
the configuration file.
\subsection{Logging}
Collecting information about events is essential within a production environment.
This information is essential forensic analysis, and tracing sources of trouble.
This is achieved through the logging of important messages that can be sent by
the tools.
ODR-DabMux and ODR-DabMod both support logging to standard error, to a file and
to the system logger \texttt{syslog}. Logging to syslog is the most flexible
approach; log information can be forwarded over the network to a
centralised logging server - where logs can then be filtered according to the
priority of each message. Both tools log to the LOCAL0 facility which in turn
can be redirected into an ODR-mmbtools specific log file.
\sidenote{Describe rsyslog configuration}
In order to avoid the log files from becomming undesirably large, \texttt{logrotate}
should be set to rotate the files automatically.
\sidenote{Describe logrotate configuration}
\subsection{Timing}
The ODR-mmbTools require the system time to be accurate in order for them to
function correctly - this is especially important when running a SFN, but is
also important for standalone transmitters when in a production environment. It
is also important to remember that most receivers have a clock that is
synchronised to the clock time which is being transmitted by the multiplex to
which it has been tuned.
The system needs to run a NTP client that synchronises the system time over the
network. Correct synchronisation can be checked using the \texttt{lpeers}
command of the \texttt{ntpq} tool. The magnitude of the offset should be below
$10$\ms.
The performance of the NTP synchronisation should also be monitored permanently
during operation.
\subsection{Monitoring using munin}
The Munin\footnote{\url{http://munin-monitoring.org/}} monitoring tool can
create graphs for essential system health parameters. It can also send emails
if values transgress the defined bounds - this assists the operator in the
assessment of system status, as well as the health of the services.
In addition to basic system measurements like CPU, RAM and disk usage, NTP
synchronisation, disk and network performance (and much more besides), there
are also custom data sources for ODR-DabMux;
These data sources include ZMQ input buffer monitoring (buffer level, underruns
and overruns) and the peak audio input levels (mono, or stereo). It
can be installed by copying \verb+doc/stats_dabmux_multi.py+ to
\texttt{/etc/munin/plugins.d}. They require that the ODR-DabMux management
server is enabled in the configuration, and will automatically generate the
graphs for the subchannels used in the configuration.
\subsection{Real-time Scheduling}
As a general principle, it is prudent not to run tools (that do not need superuser
privileges) as the \texttt{root} user. The same principle also applies to the
ODR-mmbTools, but care has to be taken that the tools can still request real-time
scheduling when it is needed.
This is achieved by adding the following to \texttt{/etc/security/limits.conf},
assuming the tools are run under the user \texttt{odr}.
\begin{lstlisting}
odr - rtprio 65
odr - nice -10
\end{lstlisting}
If you have installed JACK with real-time privileges, you may find this has
already been configured for the `audio' group, written as \texttt{@audio}, which
should suffice providing your desired user is a member of the `audio' group.
\subsection{Accessing the USRP as Non-root}
Superuser privileges are not required to access USB-connected USRP devices, but
sometimes the system lacks the configuration to enable normal users to
communicate with the device.
In that case, it is necessary to add a rule file for \texttt{udev}. This file is
included in the UHD sources, but might not have been automatically installed.
The file is called \texttt{10-uhd-usrp.rules}, should be placed into
\texttt{/etc/udev/rules.d/} and should contain
{ \footnotesize
\begin{verbatim}
#USRP1
SUBSYSTEMS=="usb", ATTRS{idVendor}=="fffe", ATTRS{idProduct}=="0002", MODE:="0666"
#B100
SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0002", MODE:="0666"
#B200
SUBSYSTEMS=="usb", ATTRS{idVendor}=="2500", ATTRS{idProduct}=="0020", MODE:="0666"
\end{verbatim}
}
% vim: spl=en spell tw=80 et
|
