Processing remote weather data with a Raspberry Pi

Enter Raspberry Pi

During the software development period, I started to explore the capabilities of the Raspberry Pi Model B. I quickly realized that for this application, a Raspberry Pi (version 1) running "wheezy" with a 16GB SD card was more than capable of replacing a Linux desktop PC. Using a Raspberry Pi (Rasp Pi) meant reduced hardware costs and gains in energy efficiency, not only making the collection system sustainable but reducing its carbon footprint.

Because both Linux Mint and wheezy are Debian distributions, it was simply a matter of copying the Python script over to the Rasp Pi and changing the file paths. The primary difference was that the serial connection was implemented using a USB-to-serial cable connected to one of the USB ports on the Rasp Pi, compared with the hardware serial port on the desktop.

Data Visualization

The initial purpose of this project was to produce a system that allows a user to view the collected data, and because the Rasp Pi was physically located far from the primary users of the dataset, thereby limiting direct access, I decided to make the data available on the network. To achieve this, I opted to go with HTML tables, because the dataset composition was going to remain fixed.

The HTML code to produce the web tables was written into the Python script. Every time the script ran, a text file received the raw HTML code, with updated data values inserted at their relevant positions (Figure 5). The key thing when creating the file was to save it with a .html extension. Once the HTML file was complete, it was stored to the web hosting folder to make the latest data available on the web page.

Figure 5: Tabular data with updated values from the locally stored data.

Initially, lighttpd [10] was the web server on the Rasp Pi, but I subsequently installed Apache [11] to push the envelope. Although Apache requires more resources than lighttpd, I found that the additional resource requirements did not degrade the performance of the software on the Rasp Pi. In fact, the project saw a net benefit from the use of Apache, with better security and wider community support, so I have continued to use it.

To enhance the capability of the software and the user experience, time-dependent graphs of some of the datasets supplemented the tabular data presentation. Gnuplot [12], a command-line graphing utility requiring limited computing resources, facilitated the visualization. With a Gnuplot Python plugin (gnuplot-py) [13], I added code to the main script to save the data in files for plotting, and the plotting code was also added to the main script.

Although the existing text files could be used for plotting, each file only held one hour's worth of data, which would have required reading across several files to plot ranges greater than one hour. Therefore, I created a separate set of daily files just for graphing. These files carried the same format as the hourly text files; however, they were stored in a separate location, and the new data values were appended to their corresponding file. With one file holding the raw data, a temporary file was created to plot the graphs from that file (Listing 1). By issuing tail -n <number_of_lines> > <temp_file>.dat for both the 24-hour and seven-day graphs – for example,

tail -n 5040 > AT_temp.dat

I could create a file that would be used to generate the Gnuplot graphs.

import Gnuplot
g = Gnuplot.Gnuplot(debug=0)
g('cd "/home/pi/Sutron_Linux/Daily/"')
g('set xdata time')
g('set terminal png size 800,400')
g('set timefmt "%d%b%Y   %H:%M:%S"')
g('set output "/home/pi/Sutron_Linux/Daily/Daily_Images/temperature.png"')
g('set format x "%H:%M\\n%d%b"')
g('set title " Daily Temperature and Dew Point Graph"')
g('set key noautotitles')
g('set grid')
g('set xlabel "Time\\nDate"')
g('set yrange [10.0:40.0]')
g('set ylabel " Temperature - C"')
g('set datafile missing "NaN"')
g('plot "AT_daily.dat" using 1:($4) with lines title "Temperature","DP_daily.dat" using 1:($4) with lines title "Dew Point"')
//code used to generate the graph for the AT (Air Temperature) and DP (Dew Point) graph.
os.system('tail -n 5040 /home/pi/Sutron_Linux/Weekly/AT_weekly.dat > /home/pi/Sutron_Linux/Weekly/AT_temp.dat')
os.system('tail -n 5040 /home/pi/Sutron_Linux/Weekly/DP_weekly.dat > /home/pi/Sutron_Linux/Weekly/DP_temp.dat')
g('cd "/home/pi/Sutron_Linux/Weekly/"')
g('set xdata time')
g('set terminal png size 800,400')
g('set timefmt "%d%b%Y   %H:%M:%S"')
g('set output "/home/pi/Sutron_Linux/Weekly/Weekly_Images/AT_weekly2.png"')
g('set format x "%H:%M\\n%d%b"')
g('set title " Weekly Temperature and Dew Point Graph"')
g('set key noautotitles')
g('set key top right')
g('set grid')
g('set xlabel "Time\\nDate"')
g('set yrange [10.0:40.0]')
g('set ylabel " Temperature - C"')
g('set datafile missing "NaN"')
g('plot "AT_temp.dat" using 1:($4) with lines title "Temperature", "DP_temp.dat" using 1:($4) with lines title "Dew Point"')
print "06  of  10 completed"

Initially the HTML table and the graphs were implemented on separate web pages, then the HTML code in the Python 2.7 script was updated with some simple Cascading Style Sheet (CSS) files, and the pages were combined into one. Moreover, the thumbnails were now linked to the full image (Figure 6).

Figure 6: Web page for combined HTML table and Gnuplot graphs.