Apr 21

Dynamic Graph Visualization in Flex, Ruby and Amazon SQS - Part 2 - Ruby and SQS

Ruby 1 Comment »

Update: An Amazon guy sent me a mail because I used the old SQS API. Please use the gems from RightScale instead of the one presented below, it supports the new SQS WSDL, which is cheaper.

Thanks to the first part of this article, you master Flex and ActionScript 3 like nobody. Don’t you ?

It’s now time to retrieve some data. You could need a realtime updated graph to monitor a bunch of rackmount servers, your database, your mailbox, or the activity of your friends of Facebook, who knows. After taking a look at the Technorati top blogs list, I wondered what was the activity of readers on those blogs, ie. check how many comments are posted on their last articles.

This article will focus on the data provider, coded in Ruby. And remember, the retrieved data will be sent to an Amazon SQS queue.

I know, why would we need to use Ruby and Amazon SQS when we can do all those tasks directly in ActionScript 3 ? Like I wrote before, you could use it to monitor something which is not directly accessible by Flash, as the ActionScript code is executed on the client computer. Data providers could also be CPU/Network expensive and/or spreaded in a computing cloud. Hey, finally, don’t you want to read some Ruby code ?

Let’s parse those RSS feeds. You’ll need one gem, the “Flexible RSS and Atom reader for Ruby”: simple-rss.

require 'open-uri' # To retrieve data from the word wide web
 
require 'rubygems' # Needed to load installed gems
require 'simple-rss'
 
feeds_url = %w{
  http://feeds.feedburner.com/Techcrunch
  http://feeds.feedburner.com/Mashable
  ... a lot more here ...
}
 
feeds_url.each do |feed_url|
 
  # Parse each feed
  rss = SimpleRSS.parse open(feed_url)
  rss.entries.each do |entry|
 
    begin
      entry.guid << "/" unless entry.guid[-1] == 47
      comments_feed_url = "#{entry.guid}feed"
      rss_comments = SimpleRSS.parse open(comments_feed_url)
      rss_comments.entries.each do |comment|
        puts "Found a new comment #{comment.title} on '#{entry.title}'"
      end
    rescue
      puts "There was a problem retrieving comments from #{comments_feed_url}"
    end
 
  end
end

As I want to keep the code simple, I only chose Wordpress blogs, as you can retrieve the comments feed for each entry easily: adding “/feed” to the end of the entry url.

Whis this code, you’ll get something like:

Found a new comment By: Tim-TechFruit on 'LoveFilm merges with Amazon’s DVD rentals'
Found a new comment By: Esi on 'Cuill has Irish roots'
Found a new comment By: Sam B on 'Women more likely to give passwords for chocolate? Yeah, but did they offer the men guilt-free sex?'
Found a new comment By: john b on 'Women more likely to give passwords for chocolate? Yeah, but did they offer the men guilt-free sex?'
Found a new comment By: Carl on 'Women more likely to give passwords for chocolate? Yeah, but did they offer the men guilt-free sex?'

We said earlier that we wanted to monitor the comments, right ? We will put this code in a infinite loop, and keep the found comments in a hash, just to announce only the new ones.

comments = {} # hash of found comments
 
loop do
  feeds_url.each do |feed_url|
 
    # Parse each feed
    rss = SimpleRSS.parse open(feed_url)
    rss.entries.each do |entry|
 
      begin
        entry.guid << "/" unless entry.guid[-1] == 47
        comments_feed_url = "#{entry.guid}feed"
        rss_comments = SimpleRSS.parse open(comments_feed_url)
        rss_comments.entries.each do |comment|
          if not comments[comment.guid]
            comments[comment.guid] = comment.guid
            puts "Found a new comment #{comment.title} on '#{entry.title}'"
          end
        end
      rescue
        puts "There was a problem retrieving comments from #{comments_feed_url}"
      end
 
    end
  end
 
  puts "... waiting 1 minute ..."
  sleep 60
end

Whis this code, you’ll get something like:

Found a new comment by: Buster Cherry on 'Crazy Gets Evicted!'
Found a new comment by: DONT MAKE FUN OF HOMELESS on 'Crazy Gets Evicted!'
Found a new comment by: joe on 'Crazy Gets Evicted!'
Found a new comment by: Oh, Give Me a Home Where the... on 'Crazy Gets Evicted!'
... waiting 1 minute ...
... waiting 1 minute ...
... waiting 1 minute ...
Found a new comment by: beth on 'Ben Savage Is Still Alive'
Found a new comment by: Dain Starr on 'Ben Savage Is Still Alive'
Found a new comment by: I LOVE BEN SAVAGE!! on 'Ben Savage Is Still Alive'
Found a new comment by: Chuy on 'Ben Savage Is Still Alive'

It would be a good idea to check each new feed in a new thread, as 99% of the processing time is devoted to network requests. We’ll use a new gem for this: fastthread.

require 'fastthread'
 
threads = []
comments = {} # hash of found comments
 
feeds_url.each do |feed_url|
  t = Thread.new do
  loop do
      rss = SimpleRSS.parse open(feed_url)
      rss.entries.each do |entry|
 
        # ...
      end
 
      sleep 60
    end
  end
  threads << t
 
end
 
# Wait for all threads to exit (never, in this case)
threads.each { |t| t.join }

With this code, you’ll see the same output as earlier, but, you guess, with every messages mixed.

Finally, we can put our data in an Amazon SQS queue. In the graph, we’ll show one node per feed, add a child node when a new comment comes in. This way, we only need one message format, only containing the feed URL.

It’s time to register to Amazon SQS. You’ll get two keys: AWS access key and AWS secret access key. You’ll also need to install a new gem: SQS.

Just require ’sqs’ as the other gems, set your credentials, and create a queue named “feeds”.

require 'rubygems'
require 'sqs'
 
SQS.access_key_id = "my key here"
SQS.secret_access_key = "my secret key here"
queue = SQS.create_queue "feeds"

Each time a comment is detected, we can put a message in the queue.

rss_comments.entries.each do |comment|
  if not comments[comment.guid]
    comments[comment.guid] = comment.guid
    puts "Found a new comment #{comment.title} on '#{entry.title}'"
    queue.send_message "#{feed_url}"
  end
end

You can download the full code here.

When you’ve finished playing, you can delete the queue with the following piece of code.

SQS.get_queue("feeds").delete!

You’re now ready to animate this with our Flex graph. It will be the subject of the next and last part of this article. (actually, there will be a fourth post to illustrate all of this, think about a racing game…).

Apr 18

Dynamic Graph Visualization in Flex, Ruby and Amazon SQS - Part 1 - Flex

Ruby 10 Comments »

In this first post, we’ll try to create a animated graph in Flex. Data is retrieved periodically from an Amazon SQS queue and the graph is updated in realtime. Why Amazon SQS ? You work in a distributed environment, don’t you ?

As Ruby is my favourite language these days, the data provider will be implemented in Ruby. Is there a better language to show concise but featureful snippets of code ?

Part 1 is dedicated to Flex. I’m not a real fan of Flash for web apps, but sometimes, you can’t avoid it when you have to impress colleagues… I’m sure you could do the same thing with Silverlight, but as far as I know, my Mac Mini does not really like Microsoft IDEs. In the second part, we’ll retrieve data from a famous API, in Ruby, and put them in an Amazon SQS queue. The last and third part will show you how to interact between Flex and Amazon SQS, the graph will be alive, to make you glad like hell.

You’ll need the Flex 3.0 SDK. For Flex and Air fans out there, you can obviously use Flex Builder 3 Professional.

If you discover the Flex 3.0 SDK, take a look at the Adobe Flex resources website, a gold mine. If you fell lazy, for now, simply follow these instructions to install the Flex SDK.

It’s time to begin coding, simply create a .mxml file. It will contain all our Flex code.

<?xml version="1.0"?> 
<mx:Application
    xmlns:mx="http://www.adobe.com/2006/mxml"
    xmlns:adobe="http://www.adobe.com/2006/fc">
</mx:Application>

You can compile and run it with the commands:

mxmlc dynamicgraph.mxml
open dynamicgraph.swf

See this code in action.

To generate graphs, I noticed two Flex components, flexvizgraphlib and SpringGraph. flexvizgraphlib seems promising but misses one mandatory feature for this project: realtime node updates. SpringGraph is a bit old, but it’s well designed and answers our need, so let’s use it. Copy the SpringGraph.swc file to the “/frameworks/libs” folder of your Flex SDK installation, simple.

Let’s create a simple graph:

<?xml version="1.0"?> 
  <mx:Application
      xmlns:mx="http://www.adobe.com/2006/mxml"
      xmlns:adobe="http://www.adobe.com/2006/fc"
      initialize="onLoad()">
 
  <adobe:SpringGraph id="springgraph" width="100%" height="100%" 
  	backgroundColor="#869ca7" repulsionFactor="1">
    <adobe:itemRenderer>
      <mx:Component>
          <mx:Label fontSize="14" text="{data.id}" color="#ffffff"/>					
       </mx:Component>
    </adobe:itemRenderer>
  </adobe:SpringGraph>
 
  <mx:Script>
    <![CDATA[
      import com.adobe.flex.extras.controls.springgraph.Graph;
      import com.adobe.flex.extras.controls.springgraph.Item;
 
      private var graph: Graph = new Graph();
      private var rootItem: Item;
 
      private function onLoad(): void {
        addNode("R");
        addNode("1", "R");
        addNode("1.1", "1");
        addNode("1.2", "1");
        addNode("1.3", "1");
        addNode("2", "R");
        addNode("2.1", "2");
        addNode("2.2", "2");
      }
 
      private function addNode(id:String, linkedTo:String = null): void {
        var item: Item = new Item(id);
        graph.add(item);
        if(linkedTo)
          graph.link(graph.find(linkedTo), item);
        springgraph.dataProvider = graph;
      }
    ]]>
  </mx:Script>
 
</mx:Application>

See this code in action.

It seems to be a lot of code, but it’s dead simple. You’re smart, just read it and you’ll understand it.

So, we’ve just created an animated 8-nodes graph, and we can even drag nodes and pan the whole graph with the mouse. Remember, the objective of this project is to retrieve data from an Amazon SQS queue. We want this to be fluid, so we won’t update the graph more than x times per second. Let’s implement a Timer which will periodically add nodes to our graph.

import com.adobe.flex.extras.controls.springgraph.Graph;
import com.adobe.flex.extras.controls.springgraph.Item;
import flash.utils.Timer;
import flash.events.TimerEvent;
 
private var graph: Graph = new Graph();
private var rootItem: Item;
private var timer: Timer = new Timer(300, 0);
private var itemCount: int = 0;
 
private function onLoad(): void {
  addNode("R")
  timer.addEventListener("timer", timerHandler);
  timer.start();
}
 
public function timerHandler(event:TimerEvent):void {
  addNode(new Number(++itemCount).toString(), "R");
 
  if(itemCount == 30)
    timer.stop();
}
 
private function addNode(id:String, linkedTo:String = null): void {
   (...)
}

See this code in action.

This code creates 30 nodes, every 300 milliseconds. And to reward your hard work, you can still play with the nodes while the graph is updated. We can now put the finishing touches, special effects. Let’s blur new nodes and play a funny sound as they appear.

Add this XML code after the <adobe:SpringGraph/> tag definition:

  <mx:Parallel id="newItemEffect">
    <mx:SoundEffect source="@Embed(source='assets/bloop.mp3')"/>
    <mx:Blur duration="200" 
      blurXFrom="10.0" blurXTo="0.0" 
      blurYFrom="10.0" blurYTo="0.0"/>
  </mx:Parallel>

You can then simply associate this effect to “new-node” events:

private function onLoad(): void {
  addNode("R")
  timer.addEventListener("timer", timerHandler);
  timer.start();
  springgraph.addItemEffect = newItemEffect;
}

See this code in action.

Of course, all credits go to Mark Shepherd, for providing this wonderful library and its source code.

If you want to know more about Flex, especially making a Rails app powered by a Flex UI, go buy Flexible Rails: Flex 3 on Rails 2 ! French readers, you can get it here. I bought this book 2 years ago, when it still was a beta PDF. Peter Armstrong then found a paperback editor, and you’ll easily understand why if you read it.

Check back soon for the next part.

WP Theme & Icons by N.Design Studio
Entries RSS Log in