Processing remote weather data with a Raspberry Pi

Creating a Database

At this point, the data for each variable was still stored in its own folder, so support for data retrieval was tedious, because a separate Python script had to be written and placed in the corresponding folder(s) when more than one variable was requested. A database would be the most logical and efficient solution, and to simplify the implementation, I looked for a standalone system that could be housed on the Rasp Pi.

An SQLite database was the best option, because it is a self-contained serverless system. The idea of going back through all of the archived text files to populate the database was daunting, until I remembered that I had already created 10-minute datasets that included all of the variables with their corresponding timestamps. Using that format as a template for the database structure, I then wrote a Python script to read these data files, some of which went as far back as 2012 for some stations. Once the data was ingested, I experimented with data-retrieval commands using SQL and wondered why I did not do this from the beginning. The script used to ingest the data was then modified to update the database with the latest 10-minute formatted dataset. You can find examples of my scripts on the Linux Magazine FTP site [9].

With the data now in a database, the possibilities were endless. With some HTML knowledge and far less CSS knowledge, I made use of Common Gateway Interface (CGI) scripting and some help from PHP to execute a Python script on the basis of user selections from an HTML form (Figure 7). Because Python is my preferred scripting language, I enabled Python in CGI [14]. This method reads the variables using the POST method from the HTML form to retrieve the requested data.

Figure 7: The database query form for variables over a specified time interval.

From the start_date, end_date, and selected variables, the Python program generates a SQL query and processes it on the database to retrieve the data. If any data is returned, it is then displayed on the screen, where the user can download the data in CSV format. If the user enters a file name other than the generated default name, the download file is downloaded to their system with that name. With this query form now available on the network, any user can request and retrieve data directly to their system quickly and easily.

Another web page, primarily designed for those interested in the data from a climatological perspective, displays maximum and minimum values for the year, month, and day of data access (Figure 8). Once again HTML tables were the choice to structure this data for display – however, with a different approach to its implementation.

Figure 8: Current conditions, minimums, and maximums.

Instead of writing the HTML code within the Python script and updating the data on the fly, I stored an HTML template with keywords. Using the sed command, these keywords were then replaced with the respective values (Listing 2). This approach was taken because the idea was to have several display templates available to the end user from which they could choose. However, to date, other templates have not been designed, although this approach does make the code more manageable.

os.system("cp /home/pi/Basher/Analysis/template.html /home/pi/Basher/Analysis/index.html")
os.system(""" sed -i 's/current_time/""" + current_time+ """/g' /home/pi/Basher/Analysis/index.html""")
os.system(""" sed -i 's/current_AT/""" + current_AT+ """/g' /home/pi/Basher/Analysis/index.html""")
os.system(""" sed -i 's/current_RH/""" + current_RH+ """/g' /home/pi/Basher/Analysis/index.html""")
os.system(""" sed -i 's/current_QFF/""" + current_QFF+ """/g' /home/pi/Basher/Analysis/index.html""")
os.system(""" sed -i 's/current_winds/""" + current_winds+ """/g' /home/pi/Basher/Analysis/index.html""")

Finishing Touches

The Rasp Pi, sprouting cables and connectors, is not exactly office friendly, especially to curious onlookers, so I decided to house it in an easy-to-obtain electrical housing box. For power, I used a 2.1A mobile phone charger with a standard USB-to-micro-USB cable to connect to the Raspberry Pi (Figure 9). By making use of the standard input from an old computer power supply, I was then able to power the charger and a standard 110VAC plug that powered the RF radio. I picked up a power switch with an internal light to turn on the power going to the mobile phone.

Figure 9: The enclosure for the Raspberry Pi and the Ethernet and power connections.

For Ethernet connectivity, I used a surface-mount adapter to connect the Ethernet cable to the Rasp Pi internally. A USB-to-serial cable connected to the makeshift serial adapter was used to accept the serial input from the RF radio.

Configuration Files. As you can imagine, the Python script to execute all of these tasks was beginning to get rather extensive. With the idea to distribute this system to CIMH member countries, I needed to make the system as easily configurable as possible. Using the built-in Python 2.7 ConfigParser module [15], some of the parameters of the script were read from a configuration file, which would allow a user to make minor adjustments to the script without having to go into the actual script itself.

Low Battery Indicator. After the installation of an AWS on St. Vincent island, communication ceased, although from all records, the system appeared to be working as it should. On closer inspection of the data, the battery was last recorded to be just above the operational level allowed by the XLite 9210B datalogger. As a result, a simple check of the battery level and a low-level indicator were implemented. This indicator shows up on the main HTML page and is only displayed if the battery level goes below 10V.

Write to USB. Before the web form to download data was created, an option was available to save data to a USB drive. Any generic USB drive can be inserted into the system, and, once detected, the current file structure is duplicated onto it. Once write access is enabled on the USB device, the data is saved to the USB drive as well. This process was developed to allow non-technical personnel to download data manually.

FTP Server. All that is needed to upload the data to a local FTP server is an IP address, password, and the path of the folder in which the file is to be placed. Through the use of the Python MySQLdb module [16], linking to the FTP server was pretty straightforward.

Future Projects. Like the battery indicator, an alert for wind gusts and average winds greater than 10 knots would be desirable. This addition would place an alert indicator on the main HTML display screen. Because of the implications of this alert, the inclusion of an audio alarm would be useful, as well. Another project is to complete the programming for the rainfall analysis and insert it on the web page.