TinyMON version 0.1.2beta



TinyMON Monitoring

The TinyMON monitoring framework allows you to save data from one or more serial forwarders into a single TinyMON data repository. All data in this repository is dynamically exported on-demand as images or XML text via HTTP requests. TinyMON repositories have a consistent namespace that makes finding the data you need very easy. TinyMON also stores historical data in fixed-size round-robin archives that allow you to easily budget storage requirements.


Available Downloads

The official web site for TinyMON is http://tinymon.cs.berkeley.edu/. Official TinyMON releases are available from http://tinymon.cs.berkeley.edu/downloads/ .

TinyMON will compile and run on Windows (using Cygwin) and Linux (both x86 and ARM). It may compile on other platforms but hasn't been tested.


TinyMON Component Layout

This diagram shows the overall layout of TinyMON with data flowing from top (motes) to bottom (HTTP clients).

      +-----------------+               +-----------------+
      | Group of Motes  |               | Group of Motes  |
      | Communicating   |    *  *  *    | Communicating   | 
      |   Via Radio     |               |   Via Radio     |
      +-----------------+               +-----------------+
      |  TOSBase Mote   |               |  TOSBase Mote   |
      |   Listening     |               |   Listening     |
      |   Via Radio     |               |   Via Radio     |
      +-----------------+               +-----------------+
              |                                 |
        Serial|Link                       Serial|Link
             \|/                               \|/
              *                                 *
      +-----------------+               +-----------------+
      | SerialForwarder |    *  *  *    | SerialForwarder |
      +-----------------+               +-----------------+
              |                                 |           
           TCP/IP                            TCP/IP         
             \|/                               \|/          
              *                                 *
      +-----------------+               +-----------------+
      | TinyMON sf2rrd  |    *  *  *    | TinyMON sf2rrd  | 
      +-----------------+               +-----------------+
                     |                     |
                 File|Write            File|Write
                    \|/                   \|/
                     *                     *
                  +----------------------------+
                  | TinyMON RRD Repository     |
                  | Rooted by default at       |
                  | /var/www/html/TinyMON/data |
                  +----------------------------+
                                |
                            File|Read
                               \|/
                                *
                  +----------------------------+ 
                  | Apache Web Server shares   |
                  | Repository Data at URL     |
                  |      /TinyMON/data         |
                  | with the handler installed |
                  |  at /TinyMON/tinymon.php   |
                  +----------------------------+
                      |      |      |      | 
                    HTTP   HTTP    HTTP   HTTP
                     \|/    \|/    \|/    \|/
                      *      *      *      *
                    wget   lynx  mozilla exploder

TinyMON sf2rrd can collect data from a serial forwarder running locally or remotely. You must have a single sf2rrd instance running for each serial forwarder that you want to collect data from. However, you can run as many sf2rrd instances as you like and point them to the same TinyMON repository (usually started by a cron job as described in the documentation).


TinyMON Graph Examples

If you want to see some example TinyMON images, visit http://tinymon.cs.berkeley.edu/examples .


TinyMON Installation

These instructions assume that you already have Apache running and have a basic understanding of how to configure Apache. Also, this document will not describe how to get a TinyOS serial forwarder up and running (it is assumed that you already have them running for the sensor nets you want to monitor).

Configuring the TinyMON Apache Frontend

  1. Install the TinyMON Apache Handler
  2. There is a tinymon.conf file as part of the TinyMON distribution. This file contains the necessary updates to the Apache configuration file to install the TinyMON handler. You can simple cut and past the text into your apache config (httpd.conf) or most Linux distributions have a configuration directory (/etc/httpd/conf.d for example) and all files in that directory are included in the configuration. If that is the case for you, simply drop tinymon.conf into that directory as such.
      % cp ./tinymon.conf /etc/httpd/conf.d

    Your Apache configuration directory may be in /etc/apache/conf.d or some other directory.

    You will need to restart Apache for the changes to take effect.

    TinyMON requires that mod_actions is loaded in your Apache server in order to process the Action configuration option. If you do not have mod_actions, loaded you will get an error message similar to the following...

      Configuration syntax error detected. Not reloading.
      Syntax error on line 4 of /etc/apache/conf.d/tinymon.conf:
      Invalid command 'Action', perhaps mis-spelled or defined by a module not included in the server configuration

    ... when you restart Apache with the new configuration.

  3. Copy the RRD icon into the Apache icon directory
  4. Copy the rrd.png icon to your apache icon directory. This icon will be displayed when Apache indexes a .rrd file. From the TinyMON distribution root run...
      % cp ./srclib/rrdtool/contrib/rrd-file-icon/rrd.png /var/www/icons

    The icon directory for your apache installation may be different than /var/www/icons.

    If your feeling particularly creative, feel free to create your own icon to customize the look of your repository.

  5. Copy the TinyMON web code to your web server root
  6. All necessary web code for TinyMON is contained in the ./web directory of the distribution. This PHP code implements the TinyMON .rrd file handler which is at the heart of the HTTP repository. By default, you must install the code in your Apache root directory in a sub-directory named ./TinyMON. From the TinyMON distribution root simply...
      % mkdir /var/www/html/TinyMON
      % cd ./web
      % cp * /var/www/html/TinyMON

    Of course you will need to replace /var/www/html/ with your Apache web root directory (which might be in say /var/www/htdocs or somewhere else)

    You may want to take a look at the default values in config.inc and modify them to taste. The defaults should work for most applications.

Configuring the TinyMON backend

The TinyMON backend consists of the sf2rrd commandline utility and a script for starting sf2rrd.

Compile TinyMON
Compiling TinyMON should be as simple as running
  % ./configure 
  % make
  % make install

By default, TinyMON installs all files to /opt/TinyMON. To see the configure options available to you simple run.

  % ./configure --help

If you want to install TinyMON into a directory other than /opt/TinyMON, you can specify that on the commandline by running...

  % ./configure --prefix=/another/tinymon/base/dir

NOTE: If you installed TinyMON into a directory other than /opt/TinyMON you will need to update the variable RRDTOOL in config.inc to point to the new location!.

If you have any problems compiling TinyMON, feel free to submit a bug report to massie at cs dot berkeley dot edu.

sf2rrd
sf2rrd is responsible for connecting to a serial forwarder and writing the data is receives to a TinyMON data store. The command line options for sf2rrd are...
  % sf2rrd -h
   Purpose:
     The TinyMON sf2rrd listens to data coming from a TinyOS serial forwarder
     and saves the data to a round-robin database repository
   
   Usage: sf2rrd [OPTIONS]...
   
     -h, --help                      Print help and exit
     -V, --version                   Print version and exit
     -i, --host=STRING               Hostname/IP of the Serial Forwarder
                                       (default=`localhost')
     -p, --port=INT                  Port of the Serial Forwarder  (default=
                                       `9001')
     -t, --timeout=INT               Timeout for connecting to Serial Forwarder
                                       (default=`30')
     -d, --rrd_root=STRING           Round-Robin data storage directory  (default=
                                       `/var/www/html/TinyMON/data')
     -s, --rrd_step=INT              Seconds between samples (s>=1)  (default=
                                       `1')
     -b, --rrd_heartbeat=INT         Seconds until a NaN is recorded for missing
                                       data  (default=`30')
     -f, --rrd_data_archives=STRING  RRD Round-Robin Archives  (default=
                                       `RRA:MIN:0.5:60:12000
                                       RRA:AVERAGE:0.5:60:12000
                                       RRA:MAX:0.5:60:12000 RRA:MIN:0.5:3600:9000
                                       RRA:AVERAGE:0.5:3600:9000
                                       RRA:MAX:0.5:60:12000')
     -v, --verbose                   Verbose Output  (default=off)

The --host option can take a hostname, IPv6 or IPv4 address.

sf2rrd uses flock to lock a portion of the repository that relates to the particular serial forwarder it is saving data from. sf2rrd will quietly exit is another instance is collecting data from the same serial forwarder.

sf2rrd by default will detach itself from the terminal when run. this prevents debugging information from being output. If you are having problems with sf2rrd, run it with the --verbose flag for a while. This will prevent it from detaching from the terminal and cause it to output information about exactly what it is doing.

If you still don't get output from sf2rrd when running with the --verbose flag, it's likely that another sf2rrd process is already running (and the process you are trying to start is quietly exiting because it can't get the lock). Check your process table (ps -ef | grep sf2rrd) to see if sf2rrd is already running.

Understanding sf2rrd data archive format
I highly recommend that you read the man page for rrdtool create functions that explain the syntax for round-robin archives available at http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/manual/rrdcreate.html . It will help you understand the format of the --rrd_data_archives parameter for sf2rrd if you need to customize the data archives for your application.

The default round-robin archive format should work for you in most cases as it tries to strike a balance between resolution and archive size. The default rrd format for sf2rrd has a size of 2.6 MB per sensor. You could, for example, monitor 128 motes each with 3 sensors each using ~998 MB of fixed space (128*3*2.6).

The fact that you can pack so much data into such a small fixed space means you could write the data to memory (using say ramfs or /dev/shm) for better performance and create a cron job to periodically sync the data to disk. In terms of scalability, disk io will be your biggest obstacle.

The default round-robin archive format saves the minimum, average and maximum value along with total number of messages received and lost with minute resolution over a week and an hour resolution over a year.

Currently, rrdtool does not support subsecond updates of round-robin databases. Given that sensors usually send data at subsecond intervals, sf2rrd caches all the data it receives, calculating the min, max and average sensor values on-the-fly before syncing the data from memory to the round-robin database. This caching means that you are guaranteed that no subsecond values are lost. However, you will only be able to track the event to a single second without knowing when the event occurred within that second.

Installing a cron job to kick sf2rrd
To finish up the installation you will need to install a cronjob to start (or restart) sf2rrd for each serial forwarder you want to save data from. There is a basic script that gets installed into /opt/TinyMON/share/tinymon.sh will can serve as a starting point for this cron script.

As a reminder, the data fields are (from crontab(5))...

              field          allowed values
              -----          --------------
              minute         0-59
              hour           0-23
              day of month   1-31
              month          1-12 (or names, see below)
              day of week    0-7 (0 or 7 is Sun, or use names)

So having a line in your crontab like

  0,15,30,45 * * * * /opt/TinyMON/share/tinymon.sh

would run the tinymon.sh script every 15 minutes. Since sf2rrd must obtain a lock in the repository to run, you do not need to worry about multiple instances of sf2rrd spawning from this cron job.


TinyMON Archive Structure

The data directory consists of directories and files just like any other filesystem. However, there is a strict naming convention employed.

All directories have the format object=instance, e.g. sf=localhost or mote=4.

All historical files have the format format.rrd. Currently, the only format is data. (In the future, TinyMON will likely support building aggregate data sets such as ``average value all all motes in group 5'' which will require a different .rrd format.)

To demonstrate, running find . in the TinyMON data directory might yield

   ./sf=localhost
   ./sf=localhost/port=9001
   ./sf=localhost/port=9001/group=36
   ./sf=localhost/port=9001/group=36/mote=2
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=0
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=0/data.rrd
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=1
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=1/data.rrd
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=2
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=2/data.rrd
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=3
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=3/data.rrd
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=4
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=4/data.rrd
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=5
   ./sf=localhost/port=9001/group=36/mote=2/adc_channel=5/data.rrd
   ./sf=localhost/port=9001/group=36/mote=3
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=0
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=0/data.rrd
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=1
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=1/data.rrd
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=2
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=2/data.rrd
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=3
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=3/data.rrd
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=4
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=4/data.rrd
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=5
   ./sf=localhost/port=9001/group=36/mote=3/adc_channel=5/data.rrd

This example shows that there is a single serial forwarder being monitored, sf=localhost, on port 9001, port=9001, which is collecting data for mote group 36, group=36, that has two motes, mote=2,3 that have six ADC channels, adc_channel=0,1,2,3,4,5. The historical archives that are being saved for each adc_channel is called data.rrd. This means that the format of the archive is simply data.

When TinyMON is installed, it serves as a handler for all .rrd data files on your web server. The TinyMON handler processes the request url to determine the act location of the data in the overall sensor network. For example, when it presents ./sf=localhost/port=9001/group=36/mote=3/adc_channel=5/data.rrd the handler knows the data is coming from localhost:9001 mote group 36, mote 3, adc channel 5.


Building TinyMON Reports

The TinyMON archive structure allows you to find exactly what data you need in an analysis. TinyMON also allows you to customize how that data is presented by appending GET variables to the base TinyMON url string.

Setting the report time range

The start and end variables allow you to specify the time range you are interested in analyzing. You need to specify the timestamps in AT-STYLE TIME. For more details about the time specification see the AT-STYLE TIME section of the rrd fetch man page at http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/manual/rrdfetch.html .

The step variable allows you to tweak the step in seconds between the data points.

It is important that start be a time earlier than end or you will generate an error.

For example, if you want to view data over the last hour.

   ...mote=4/adc_channel=3/data.rrd?start=-1hour

or view the data from yesterday by running...

   ...mote=4/adc_channel=3/data.rrd?start=-2days&end=-1day

The value for start and end are written onto graphs and xport xml in order for you to know exactly what the time domain is.

You can modify the default start and end values in config.inc.

Controlling the report format

Currently there are two actions supported: graph and xport. When no action is explicitly specified, graph is assumed.

For example,


   ...mote=4/adc_channel=3/data.rrd?action=graph
   ...mote=4/adc_cahnnel=3/data.rrd

are equivalent.

action=graph
A graph report will also process the following GET variables: title, height and width.

For example,

   ...mote=4/adc_channel=3/data.rrd?title="Graph of Temperature in Machine Room"
   ...mote=4/adc_channel=4/data.rrd?height=120&width=320

You alter the default values for graph height and width in config.inc.

There is a sample graph at http://tinymon.cs.berkeley.edu/examples/ if you would like to view a graph while you read the following few paragraphs.

The background color of the graph at each point along the x-axis tells you about the quality of the communication with the mote. If the background is red then you know that communcation with the mote was lost (either the mote stopped communicating with the serial forwarder or the serial forwarder stopped communicating with sf2rrd). If the background is not red then you know that at least some data was received.

TinyMON saves the number of messages received and lost for each time slice. The background color of the graph allows you to know the quality of the communication with the mote. The lighter the background the better the quality of the signal. A white background means near perfect communication and darker shades of grey denote poorer communication. The legend directly under the graph shows how to interpret the background color.

The maximum value is expressed with a green line, the average value a black line and the minimum value a blue line. The legend below the graph expresses this along with the maximum, minimum and average value of the sensor over the entire time range.

The graph also lists the serial forwarder, mote identification and time range information.

action=xport
The xport action will cause TinyMON to report TinyMON data in XML. The XML format is explained on the rrdtool xport man page at http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/manual/rrdxport.html .

Using the time domain GET variables allows for a very flexible way to view and collect data from TinyMON. For example, if I find a graph at

   ...mote=4/adc_channel=3/data.rrd

that is interesting to me. I can zero in on a specific time range that I'm interested in (say the last hour).

   ...mote=4/adc_channel=3/data.rrd?start=-1day

and I can then download the raw data points in XML format

   ...mote=4/adc_channel=3/data.rrd?start=-1day&action=xport

The urls are intensionally built to be symmetrical between xport and graph. The xport action will silently ignore special graphing attributes like height and width.


TinyMON Help and Support

The author of TinyMON is Matt Massie (massie at cs dot berkeley dot edu). Please feel free to contact the authors with questions and comments about TinyMON.


TinyMON Limitations and Future Work

Currently, rrdtool only allows for no more than one data point per second. Subsecond monitoring is not supported. TinyMON deals with this limitation by writing the min, average and max values for all samples received over each second.

TinyMON currently only processes Active Messages of type 10 (Oscope) and 32 (Oscope Resets). A plugin API could be added in the future to allow other message formats, if there is a demand. If you would like a plugin API, please send the author an email.

In the future, there will likely be an XMLRPC, http://xmlrpm.com/ , interface added to TinyMON.


TinyMON ChangeLog

 Version 0.1.2beta
   * Removed a "rpc/rpc.h" include in sf2rrd.c that breaks the compile on Cygwin when
     the SunRPC library is not installed.
 Version 0.1.1 (2005-03-17)
   * Fixed a bug in the graph loss calculation that caused inaccurately high message loss values
 Version 0.1.0 (2005-03-16)
   * Initial Release
 TinyMON version 0.1.2beta