Our First Map

We are happy to announce that we have our first Climate Visualization Map

http://innova-t.appspot.com/marovi/listPoints/

The Process
We have created a semi-automatic process.  We download the GRIB file to a local linux box. We then process that file so that we extract:

• initial timestamp
• latitude and longitude
• temperature in Kelvin

The excel that is generated is the input to the datastore in GAE.

Styles and Heatmap Layer
Maps are good if they display what we want clearly. That´s why we were inspired by several talks of the Google Maps Developer team to include some of their recommendations on the maps. Of course, we turned on the Heatmap layer -- which is a part of the Google Maps API. We configured the terrain, inverted the lightness to display water masses and countries more clearly. We also specified the opacity in a custom format. The data points were also important and we turned off the dissipation for a better effect of the temperature.

And of course, we used the Style Map Wizard for testing our styles.

Content
The map displays only 1000 data points from a set of 150K temperature measurements taken during the first four months of 2012. We plan to integrate more than 30 years worth of data. So this is a good start. We are able to validate that the coordinates are correct (displays temperature on top of land masses) and that they make sense. The points displayed show that in Australia is is warmer than America due probably to the hour of the day.

Understanding GRID files

Abstract
Notes on the interpretation of GRIB and HDF5 file formats for EOS.

Grid Data
Our files are mostly grids which means they represents data points taken at some point in time and contains a set of projection equations. Although we do not care that much about the different type of projection used (Mercartor, for example) due to libraries that do the hard work for us. However, we are interested in how to extract the actual value taken along with the relevant latitude and longitude. I try to explain what I did with the files in order not to forget in the future.

Basics
There are three important features of a grid file: data fields, dimensions, and the projection. The data fields are rectilinear arrays of two or more dimensions and are related to each other by common geolocation.

Dimensions are used to relate data fields to each other and to the geolocation information. To be interpreted properly, each data field must make use of two predefined dimensions "XDim" and "YDim".

Projections are used to encode and interpret. It provides with a convenient way to encode geolocation information as a set of math equations capable of transforming earth coordinates to x-y coordinates on a sheet of paper.

Most sources of data files are websites like Giovanni and provide with GRIB or HDF-EOS file formats. We use those file formats to extract the values we are interested: average surface temperature.

Extracting Values
Thanks to libraries like pygrib we are able to read the GRIB files and extract:
- lat,lng
- measurement

The files that we are currently processing are taken at 3 hours intervals so we just have to take the first measurement and iterate forward. The date is given by the file name.

You can read the "headers" of the file that will tell you which index to use to extract the measurement of interest.

Data Extraction Plans

This week we are planning to work on extracting the data from GRIB files. After reviewing the spec from the file we are going to use the average temperature from the last 3 hour periods.

We also reviewed the Google Maps API in particular the heatmap layer API.

The library in python that we are going to use is pygrib It requires several libraries like numpy in order to work.

We are starting to develop the business model that will allow us to extract the data in the following dimensions:

• lat/lon
• city (for the United States): we are going to get an approximate
• timestamp when the data point was collected
• month, week, day, hour (averaged value)

Regarding the data source, some important points: 

• each coordinate have the state in the united states (the resolution is every 10 Km)
• the time period is an instant (every 3hrs)
• it is the average surface temperature code 138 from the GLDAS model

We are also reviewing the use of Fusion Tables for displaying the geodata.

Keep you posted.

GRIB and GLDAS files

Finally we were able to find a library in python that allows us to read the data in GRIB format

http://pygrib.googlecode.com/svn/trunk/docs/index.html

We found the library easy to use and for now, we are thinking in extracting the data as part of a manual process.

Defining our first map

It is one of the main objectives of the project is to find the visualizing tools to facilitate the general public to process data from climate data sources. Our first business case is to display surface temperature. As we investigate this week and next, we are coming close to define the User Interface (UI) as the combination of two maps.

A definite example to follow is this map of the boss. as it shows the concerts of Bruce in the US. We plan to use the HeatMap layer as part of the Google Maps API for styled maps.

Another example that complements our effort is vote night. Which shows a combination of user control with charts displays. We want the map to be the UI!

As Giuseppe said this week in finalizing the details of the business case,

This visualization can be used to show the variability in time of T in an area that the user can select. We should think about what is the best variable to show. Temperature varies within each day; thus, we may show the animation of monthly mean (or minimum or maximum) temperature for the all set of years.
And we may create some flash marker that appears when we reach, for example, the maximum (and/or the minimum) temperature ever in that area. After appeared, the marker will remain in the area with the data shown until the animation ends. We may have a few markers with different spatial information on the records of data we have shown.

On another note, Sri is working in setting his workarea with the svn repo and the app engine project ready to go. He will be installing Eclipse with all the support tools for Django and python. 

The Beginning

This is our first post in the blog. We'll provide with the latest update on the project that is a finalist of the Google App Engine Research Award.

After Enrique workout the details and redtape behind accepting the award, we are currently on the requirement definition and functional specification for the project.