Let me walk you through my exploration of these new technologies as I get acquainted with them. The objective will be to display geo-tagged tweets around the world in real-time on a mapview, with the profile pic and tweet info on the annotation (see screenshot below). This can be accomplished in just a few lines of code, less than 100 with the help of some cool new frameworks and libraries.
There’s a lot of buzz surrounding the real-time web nowadays, which involves pushing data and events from servers straight to clients just as it happens. There’s plenty of ways of achieving this, both on the browsers (websockets, long-polling, etc), and on mobile devices (sockets, AMQP clients, etc). The catch-all term is Comet.
Here’s a stackoverflow discussion on some of the options on the iPhone. I tried using the STOMP client they mention and setting up an Apache ActiveMQ server (a AMQP) but ideal configurations proved hard to come by. Basically I set up a “topic” which is used as a one to many kind of broadcast, but messages were waiting for an acknowledgment from the phone and everything just started to lag for everybody. I’m sure this can be set up properly, and there’s other AMQPs I want to try out such as ZeroMQ and RabbitMQ, but it was just a quick test so I didn’t look too much into it.
On the browser the ideal way of doing push is with WebSockets, though not all browsers support it yet. There’s a couple workarounds around that, socket.io and web-socket-js. This demo was pretty cool of mouse cursors moving around in real-time, and the APE Ajax push engine also has some sweet demos.
Anyways, there’s plenty of info around on the web about comet and push. What I’m going to try to do here is walk you through my first exposures into node.js and Appcelerator Titanium Mobile to get a real-time mashup of tweets on a mapview:
Twitter Streaming API
They’ve had this API out for a few months now, so there’s plenty of libraries in a bunch of languages for easily accessing it. It basically involves keeping an HTTP connection open to their servers and continually receive data through that pipe. Twitter is beta testing User Streams, which is more suited for user twitter clients. Both of these APIs should eventually help out twitter with their load issues since constant polling by everybody can get pretty heavy. Here are some more advantages.
For the purposes of this demo, and since it’s all I have access to, we’ll be using the general purpose public streaming API. You’ll need a twitter account to be able to use the API, and they are currently using HTTP Auth on it (outside of their HTTP Auth deprecation schedule this month). The default access role grants you 10 boxes of one (lat/lon) degree, which is basically the size of a city. You can request the “locRestricted” role which allows 200 boxes of 10 degrees each. This almost covers the entire land-mass of the earth. They don’t have any way of just querying all geo-located tweets, not even with the firehose access role, so you have to construct your boxes yourself (I checked with their support).
Since my first test was with STOMP and ruby, here’s some lame dirty ruby to get a very crude list of 200 boxes around the continents in the format that twitter wants. I couldn’t fit Asia at all in there, and a bit of Africa got left out… oh well.
If you just have the default access roles, 10 cities, then you can try these out
Just copy paste the array from the output, and stick into the query below…
It’s all the rage now, and for very good reasons. Event based non-blocking stuff is just so awesome. I haven’t done much with node.js yet other than this demo, but hopefully I’ll get to use it more soon enough. So basically just go ahead and install node.js, it’s a super simple install. Naturally there’s a twitter streaming library for node.js, called twitter-node. Here’s the github page. Go and clone that somewhere. (I haven’t explored node.js package managers yet). Be sure to run the build script they have in there to install the base64 library you need. Grab the boxes array from above and put it in there for the location query (for some reason I couldn’t make it to 200 boxes with this library).
So basically what we are going to do is create a socket server with node.js on port 6969 and for every new event the twitter library sends us, we’ll go ahead and push that to the socket, which in turn will push it out to all the clients currently connected. I haven’t figured out how to close the socket properly if a client was an asshole and didn’t FIN, leaving the socket in a sort of limbo state. I don’t know if this even matters, but basically an exception will be thrown for each of those limbo handlers, each time we try to write to it.
That takes care of the server side… now to the client side.
Appcelerator Titanium Mobile
After trying most of them out, Appcelerator completely outshines them. Rhomobile is cool in that it’s ruby on rails-esque, write once, and push out to FIVE different platforms (iOS, Android, Blackberry, Windows, Symbian), but it’s just too slow and fugly (not completely native).
Anyways… after you get everything set up with Appcelerator, go ahead and create a new project in Titanium (iPhone or iPad, don’t matter) and put the following code in your app.js. We are just adding a mapview to the window, and some buttons to control the socket connection. When the ‘read’ event on the socket gets triggered, meaning a new JSON blob came in, we’ll parse that and create the corresponding annotation for the tweet. We can easily add a remote image to the annotation for the profile picture using Joe Stump‘s awesome tweetimag.es service.
That’s basically it. There’s a few bugs, but this isn’t about perfection, just a quick sample of these new technologies. Scary as it may be, this is pretty much where we are headed towards. A constant flow of real-time information, following us wherever we go. Can’t wait!
Is this particular application helpful? Maybe. I could see it being used to monitor emergency situations, or some type of big event. Problem is that very few people yet geo-tag their tweets. The average number of normal tweets was 750/second a month ago. Compare that to maybe 10 tweets a second of geo-tagged tweets…
With the access role I have and the boxes I use, I’m getting about 5 tweets a second on the map. Don’t go thinking that the iPhone can support that many total annotations on the map like in the screenshot above (it’s the simulator).
They have yet to add socket support to the Android side of Titanium, so as of now this won’t work on the Android (though it’ll probably work without modification once they do).
A bunch of JSON parsing exceptions happen on the ‘read’ event of the client socket. I imagine it’s because more than one tweet might come in the JSON payload per packet or whatever triggers the ‘read’, so it freaks out.
The bugs I encountered were that the pin drop animation isn’t working in the latest version of Titanium, though I know they are on it. Also if you terminate the app it’ll be the asshole I mentioned above and not send the FIN to the socket. A curious observation is that the socket remains connected when the app goes into the background state (probably just sleeping or something).
Please feel free to comment below any best practices on this (I’m a newb), or any questions you may have.