The beauty of open data one year after the post that started it all

A new way to look at the Heat vs Mavericks NBA Finals Games 1 – 4

OK sports are great for the fact that they are a statisticians dream, so many numbers, so many angles, who could blame them. I just came across this impressive set of graphs in the Washington Post that depict the NBA finals this year, where the Miami Heat are struggling against the Mavericks despite the fact that they seem to have a dream roster. The large graphs depict the flow of the game through the 4 quarters showing the total points per team, time that the teams led during the game, and final score. Directly under each of the graphs you see 4 bar charts that represent each quarter and display the minutes that each team led, and finally under the bar charts is a pie chart that shows the total minutes that each team led during the game. This is an excellent example of an executive dashboard, and turning mundane statistics into information.

A good example of an executive overview

A practical example – An ExtJS Stock game Using YQL, PHP, and MySQL

I have been designing a practical example that will incorporate as many different features as possible, for teaching all of the ways that ExtJS works and also provide a useful tool for learning a little bit about the stock market.

Some of the obvious major features that we will be taking a look at ‘under the hood’ of this application, include YQL, Yahoo’s Query Language that serves as the stock information look up, obviously some PHP and JSON connections including just how all of that fits together to give you a simple user log in and portfolio storage system, several ExtJS components like tab panels, external content rendered into popup windows, paging, polling and some basic charting.

Don’t worry if all of this seems a bit much after deconstructing this example I am sure that you will not only understand how a simple web application such as this one works, you will also be able to build one.

Before starting your Ext masterpiece think about how you may want to layout your application, for some inspiration you can check out the different layouts available in the extjs framework by looking here ( http://dev.sencha.com/deploy/dev/examples/layout-browser/layout-browser.html ).

The Stock Game Layout

The Stock Game Layout an Ext tab panel, with a managed i frame panel, and a grid.

You may have noticed that there is a lot of information coming back about the selected row in the top grid, and since this is a relatively small site, you might be wondering how this application was able to gain such data on the fly for any stock traded on any market? The answer lies in using YQL to scrape the financial data from the Yahoo Finance website. You will see that the data presented in the lower panel in the screen shot above looks much like the data in this screen shot for the same stock on the same day.

Yahoo Finance GILD Stock Lookup

Yahoo Finance GILD Stock Lookup

I could use the managed iframe extension (http://www.sencha.com/learn/Extension:ManagedIframe) to get the identical site to load into say a pop-up window such as the one that is linked to the stock ticker in the top grid, however, that poses the following problems, you will get a JavaScript error like the one below:

This is what happens when you try to load external content into an ExtJs panel

This is what happens when you try to load external content into an ExtJs panel

To get around this annoying cross domain content problem fire up the YQL console and give the following query a try

SELECT * FROM html WHERE url="http://finance.yahoo.com/q?s=f" AND xpath="//div[@class='yfi_quote_summary']"

Before coding your PHP page that will contain the output that we can see in the console we have to understand how we got here in the first place, right? You will have to determine what content you are after, and only you can make the call on whether that content is copyrighted or not, one you see a table, paragraph, tag, something that you would like to access just fire up Firebug and find out the name, id, etc of the tag that contains the data and the URL. Plug the URL in here:

url="http://finance.yahoo.com/q?s=f"

and the tag and class / id in here

xpath="//div[@class='yfi_quote_summary']"

now press the button and watch your data come back to you in the form of formatted XML.

A simple PHP page that would handle the data return and formatting would be:

<style>
  
  .yfitp table{ font-size:12px; width:100%}
  .yfitp tr{ font-size:12px;}
  .yfitp td{ font-size:12px; border-bottom:1px solid #3F567D }
  .yfi_quote_summary{
    font-family:arial, Helvetica, sans-serif;
    font-size:10pt;
    display:block;
    width:45%;
    float:left;
    margin:5px;
    padding:5px;
  }
  #yfi_headlines{
     font-family:arial, Helvetica, sans-serif;
    font-size:10pt;
    display:block;
    width:45%;
    float:left;
    margin:5px;
    padding:5px;
  }
  .display_links{display:none;}
  .chart{
    display:block;
    float:left;
    width:45%;
    margin:5px;
    padding:5px;
  }
  .chart a{text-decoration:none; border:none;}
  img{border:none;}
  .ft{display:none;}
  diagnostics{display:none;}
</style>
<?php
$ticker = $_REQUEST['ticker'];
  $BASE_URL = "http://query.yahooapis.com/v1/public/yql?q=SELECT%20*%20FROM%20html%20WHERE%20url%3D%22http%3A%2F%2Ffinance.yahoo.com%2Fq%3Fs%3D" . $ticker . "%22%20AND%20xpath%3D%22%2F%2Fdiv%5B%40class%3D'yfi_quote_summary'%5D%22&diagnostics=true";
  
  
    // Make call with cURL
    $session = curl_init($BASE_URL);
    curl_setopt($session, CURLOPT_RETURNTRANSFER,true);
    $data = curl_exec($session);
   
   echo $data;
   
  $BASE_URL = "http://query.yahooapis.com/v1/public/yql?q=SELECT%20*%20FROM%20html%20WHERE%20url%3D%22http%3A%2F%2Ffinance.yahoo.com%2Fq%3Fs%3D" . $ticker . "%22%20AND%20xpath%3D%22%2F%2Fdiv%5B%40class%3D'chart'%5D%22&diagnostics=true";
    $session = curl_init($BASE_URL);
    curl_setopt($session, CURLOPT_RETURNTRANSFER,true);
    $data = curl_exec($session);
  
   echo $data;
  
  
  $BASE_URL = "http://query.yahooapis.com/v1/public/yql?q=SELECT%20*%20FROM%20html%20WHERE%20url%3D%22http%3A%2F%2Ffinance.yahoo.com%2Fq%3Fs%3D" . $ticker . "%22%20AND%20xpath%3D%22%2F%2Fdiv%5B%40id%3D'yfi_headlines'%5D%22&diagnostics=true";
    $session = curl_init($BASE_URL);
    curl_setopt($session, CURLOPT_RETURNTRANSFER,true);
    $data = curl_exec($session);
    $pattern = "/<(a¦img)([^>]+)>/i";
    $replacement = "<\\1 target=\"_blank\"\\2>";
    $new_str = preg_replace($pattern,$replacement,str_replace('target="_blank"','',$data));
  
   echo $new_str;
  
?>

You can see the code working (work in progress) above by clicking on the Stock game tab.

Data Visualization, how much is a billion really ….

The billion dollar chart can be found at http://www.informationisbeautiful.net/visualizations/the-billion-dollar-o-gram-2009/ . The real take away is that you have to realize the difference between absolute numbers and relative numbers, which this presentation explains brilliantly.

Why what we think we know is often farther from the truth than any of us can imagine.

This is an extremely interesting data tool that is open for all to use (and explore at http://www.gapminder.org/). The speech goes to show that views are often formed early in life and stay that way despite the constant change that takes place around us. The term ‘conventional wisdom’ was coined years ago to describe an idea or set of beliefs that was accepted by most and was often predictable, for example, we all thought the world was flat at some time. The problem with conventional wisdom today is that we are inundated with data and finally, we are seeing tools being built that will visually display that data so that it can be understood in the context of the larger picture. This will in turn challenge our long standing beliefs, and shape the way that the next generation views the world.

So how can you turn your internal datasets into valuable insight?

You could view the datasets that exist at Hans Rosling’s Gapminder site http://www.gapminder.org/, you can run the types of comparisons talked about in the video at Health and Wealth of Nations

There are also some tools coming out of Microsoft Labs that will take excel spreadsheet like data sets and display them in a manner that is similar to the display featured in the video above. If you want to know more or try your hand at creating a simple data visualization navigate to http://www.getpivot.com/