Twitter Streaming API + node.js + Appcelerator Titanium = Real-time tweet map

Let me walk you through my exploration of these new technologies as I get acquainted with them. The objective will be to display geo-tagged tweets around the world in real-time on a mapview, with the profile pic and tweet info on the annotation (see screenshot below). This can be accomplished in just a few lines of code, less than 100 with the help of some cool new frameworks and libraries.

Comet?

There’s a lot of buzz surrounding the real-time web nowadays, which involves pushing data and events from servers straight to clients just as it happens. There’s plenty of ways of achieving this, both on the browsers (websockets, long-polling, etc), and on mobile devices (sockets, AMQP clients, etc). The catch-all term is Comet.

Here’s a stackoverflow discussion on some of the options on the iPhone. I tried using the STOMP client they mention and setting up an Apache ActiveMQ server (a AMQP) but ideal configurations proved hard to come by. Basically I set up a “topic” which is used as a one to many kind of broadcast, but messages were waiting for an acknowledgment from the phone and everything just started to lag for everybody. I’m sure this can be set up properly, and there’s other AMQPs I want to try out such as ZeroMQ and RabbitMQ, but it was just a quick test so I didn’t look too much into it.

On the browser the ideal way of doing push is with WebSockets, though not all browsers support it yet. There’s a couple workarounds around that, socket.io and web-socket-js. This demo was pretty cool of mouse cursors moving around in real-time, and the APE Ajax push engine also has some sweet demos.

Anyways, there’s plenty of info around on the web about comet and push. What I’m going to try to do here is walk you through my first exposures into node.js and Appcelerator Titanium Mobile to get a real-time mashup of tweets on a mapview:


Twitter Streaming API

They’ve had this API out for a few months now, so there’s plenty of libraries in a bunch of languages for easily accessing it. It basically involves keeping an HTTP connection open to their servers and continually receive data through that pipe. Twitter is beta testing User Streams, which is more suited for user twitter clients. Both of these APIs should eventually help out twitter with their load issues since constant polling by everybody can get pretty heavy. Here are some more advantages.

For the purposes of this demo, and since it’s all I have access to, we’ll be using the general purpose public streaming API. You’ll need a twitter account to be able to use the API, and they are currently using HTTP Auth on it (outside of their HTTP Auth deprecation schedule this month). The default access role grants you 10 boxes of one (lat/lon) degree, which is basically the size of a city. You can request the “locRestricted” role which allows 200 boxes of 10 degrees each. This almost covers the entire land-mass of the earth. They don’t have any way of just querying all geo-located tweets, not even with the firehose access role, so you have to construct your boxes yourself (I checked with their support).

Since my first test was with STOMP and ruby, here’s some lame dirty ruby to get a very crude list of  200 boxes around the continents in the format that twitter wants. I couldn’t fit Asia at all in there, and a bit of Africa got left out… oh well.

If you just have the default access roles, 10 cities, then you can try these out

Just copy paste the array from the output, and stick into the query below…

node.js

It’s all the rage now, and for very good reasons. Event based non-blocking stuff is just so awesome. I haven’t done much with node.js yet other than this demo, but hopefully I’ll get to use it more soon enough. So basically just go ahead and install node.js, it’s a super simple install. Naturally there’s a twitter streaming library for node.js, called twitter-node. Here’s the github page. Go and clone that somewhere. (I haven’t explored node.js package managers yet). Be sure to run the build script they have in there to install the base64 library you need. Grab the boxes array from above and put it in there for the location query (for some reason I couldn’t make it to 200 boxes with this library).

So basically what we are going to do is create a socket server with node.js on port 6969 and for every new event the twitter library sends us, we’ll go ahead and push that to the socket, which in turn will push it out to all the clients currently connected. I haven’t figured out how to close the socket properly if a client was an asshole and didn’t FIN, leaving the socket in a sort of limbo state. I don’t know if this even matters, but basically an exception will be thrown for each of those limbo handlers, each time we try to write to it.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
var boxes = [-90, -60, -80, -50, -80, -60, -70, -50, -70, -60, -60, -50, -60, -60, -50, -50, -50, -60, -40, -50, -40, -60, -30, -50, -30, -60, -20, -50, -90, -50, -80, -40, -80, -50, -70, -40, -70, -50, -60, -40, -60, -50, -50, -40, -50, -50, -40, -40, -40, -50, -30, -40, -30, -50, -20, -40, -90, -40, -80, -30, -80, -40, -70, -30, -70, -40, -60, -30, -60, -40, -50, -30, -50, -40, -40, -30, -40, -40, -30, -30, -30, -40, -20, -30, -90, -30, -80, -20, -80, -30, -70, -20, -70, -30, -60, -20, -60, -30, -50, -20, -50, -30, -40, -20, -40, -30, -30, -20, -30, -30, -20, -20, -90, -20, -80, -10, -80, -20, -70, -10, -70, -20, -60, -10, -60, -20, -50, -10, -50, -20, -40, -10, -40, -20, -30, -10, -30, -20, -20, -10, -90, -10, -80, 0, -80, -10, -70, 0, -70, -10, -60, 0, -60, -10, -50, 0, -50, -10, -40, 0, -40, -10, -30, 0, -30, -10, -20, 0, -90, 0, -80, 10, -80, 0, -70, 10, -70, 0, -60, 10, -60, 0, -50, 10, -50, 0, -40, 10, -40, 0, -30, 10, -30, 0, -20, 10, -90, 10, -80, 20, -80, 10, -70, 20, -70, 10, -60, 20, -60, 10, -50, 20, -50, 10, -40, 20, -40, 10, -30, 20, -30, 10, -20, 20, -20, -20, -10, -10, -10, -20, 0, -10, 0, -20, 10, -10, 10, -20, 20, -10, 20, -20, 30, -10, 30, -20, 40, -10, 40, -20, 50, -10, -20, -10, -10, 0, -10, -10, 0, 0, 0, -10, 10, 0, 10, -10, 20, 0, 20, -10, 30, 0, 30, -10, 40, 0, 40, -10, 50, 0, -20, 0, -10, 10, -10, 0, 0, 10, 0, 0, 10, 10, 10, 0, 20, 10, 20, 0, 30, 10, 30, 0, 40, 10, 40, 0, 50, 10, -20, 10, -10, 20, -10, 10, 0, 20, 0, 10, 10, 20, 10, 10, 20, 20, 20, 10, 30, 20, 30, 10, 40, 20, 40, 10, 50, 20, -20, 20, -10, 30, -10, 20, 0, 30, 0, 20, 10, 30, 10, 20, 20, 30, 20, 20, 30, 30, 30, 20, 40, 30, 40, 20, 50, 30, -20, 30, -10, 40, -10, 30, 0, 40, 0, 30, 10, 40, 10, 30, 20, 40, 20, 30, 30, 40, 30, 30, 40, 40, 40, 30, 50, 40, -20, 40, -10, 50, -10, 40, 0, 50, 0, 40, 10, 50, 10, 40, 20, 50, 20, 40, 30, 50, 30, 40, 40, 50, 40, 40, 50, 50, -10, 40, 0, 50, 0, 40, 10, 50, 10, 40, 20, 50, 20, 40, 30, 50, 30, 40, 40, 50, 40, 40, 50, 50, 50, 40, 60, 50, -10, 50, 0, 60, 0, 50, 10, 60, 10, 50, 20, 60, 20, 50, 30, 60, 30, 50, 40, 60, 40, 50, 50, 60, 50, 50, 60, 60, -10, 60, 0, 70, 0, 60, 10, 70, 10, 60, 20, 70, 20, 60, 30, 70, 30, 60, 40, 70, 40, 60, 50, 70, 50, 60, 60, 70, -10, 70, 0, 80, 0, 70, 10, 80, 10, 70, 20, 80, 20, 70, 30, 80, 30, 70, 40, 80, 40, 70, 50, 80, 50, 70, 60, 80, -140, 10, -130, 20, -130, 10, -120, 20, -120, 10, -110, 20, -110, 10, -100, 20, -100, 10, -90, 20, -90, 10, -80, 20, -80, 10, -70, 20, -70, 10, -60, 20, -60, 10, -50, 20, -140, 20, -130, 30, -130, 20, -120, 30, -120, 20, -110, 30, -110, 20, -100, 30, -100, 20, -90, 30, -90, 20, -80, 30, -80, 20, -70, 30, -70, 20, -60, 30, -60, 20, -50, 30, -140, 30, -130, 40, -130, 30, -120, 40, -120, 30, -110, 40, -110, 30, -100, 40, -100, 30, -90, 40, -90, 30, -80, 40, -80, 30, -70, 40, -70, 30, -60, 40, -60, 30, -50, 40, -140, 40, -130, 50, -130, 40, -120, 50, -120, 40, -110, 50, -110, 40, -100, 50, -100, 40, -90, 50, -90, 40, -80, 50, -80, 40, -70, 50, -70, 40, -60, 50, -60, 40, -50, 50]//, -140, 50, -130, 60, -130, 50, -120, 60, -120, 50, -110, 60, -110, 50, -100, 60, -100, 50, -90, 60, -90, 50, -80, 60, -80, 50, -70, 60, -70, 50, -60, 60, -60, 50, -50, 60, -140, 60, -130, 70, -130, 60, -120, 70, -120, 60, -110, 70, -110, 60, -100, 70, -100, 60, -90, 70, -90, 60, -80, 70, -80, 60, -70, 70, -70, 60, -60, 70, -60, 60, -50, 70, -140, 70, -130, 80, -130, 70, -120, 80, -120, 70, -110, 80, -110, 70, -100, 80, -100, 70, -90, 80]//, -90, 70, -80, 80, -80, 70, -70, 80, -70, 70, -60, 80, -60, 70, -50, 80]
 
var sys = require('sys');
var TwitterNode = require('./twitter-node').TwitterNode;
 
var twit = new TwitterNode({
user: '<twitter_user>',
password: '<twitter_password>',
//track: ["inception"],
locations: boxes
});
 
twit.headers['User-Agent'] = 'node.js-thingy';
 
twit.addListener('error', function(error) {
sys.puts(error.message);
});
 
sys.puts("Start the twittah party");
 
twit.addListener('tweet', function(tweet) {
sys.puts("@" + tweet.user.screen_name + ": " + tweet.text);
});
 
var net = require("net");
net.createServer(function(socket){
twit.addListener('tweet', function(tweet) {
try {
socket.write(JSON.stringify(tweet));
}
catch (e) {
// socket.close(); // ???
// socket.destroy(); // ???
// socket.end(); // ???
 
sys.puts("Socket write error");
}
})
//socket.setTimeout(100);
socket.on("end", function () {
socket.end();
});
 
}).listen(6969);
 
twit.stream();
view raw tweet_proxy.js hosted with ❤ by GitHub
That takes care of the server side… now to the client side.

 

Appcelerator Titanium Mobile

I’m really impressed with this open source library. It’s event based javascript as well, with proxies and stuff to turn completely native. It makes so many things extremely simple that are usually a pain in the ass to do natively on both platforms. It’s still kind buggy and only supports iOS and Android at the moment, but the native builds really feel native. I like their approach to the cross-platform issue, where they try to keep things as unified as possible, but don’t let that limit themselves, so they also take advantage of each platform’s individual features. They have a module SDK so you can stick actual Objective-C or Java in there if necessary, and apparently they’ll be launching a module marketplace soon, along with Blackberry support. Here’s a great overview of the different cross-platform options.

After trying most of them out, Appcelerator completely outshines them. Rhomobile is cool in that it’s ruby on rails-esque, write once, and push out to FIVE different platforms (iOS, Android, Blackberry, Windows, Symbian), but it’s just too slow and fugly (not completely native).

Anyways… after you get everything set up with Appcelerator, go ahead and create a new project in Titanium (iPhone or iPad, don’t matter) and put the following code in your app.js. We are just adding a mapview to the window, and some buttons to control the socket connection. When the ‘read’ event on the socket gets triggered, meaning a new JSON blob came in, we’ll parse that and create the corresponding annotation for the tweet. We can easily add a remote image to the annotation for the profile picture using Joe Stump‘s awesome tweetimag.es service.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135
// create tab group
var tabGroup = Titanium.UI.createTabGroup();
 
/////////////////////////
// create base UI tab and root window
// We need a tabBar to be able to win.setToolbar items, even though we'll hide it
/////////////////////////
 
var win = Titanium.UI.createWindow({
title:'TweetFlow',
backgroundColor:'#fff',
tabBarHidden: true
});
var tab1 = Titanium.UI.createTab({
icon:'KS_nav_views.png',
title:'Map',
window:win
});
 
var mapview = Titanium.Map.createView({
mapType: Titanium.Map.STANDARD_TYPE,
region:{latitude:33.74511, longitude:-84.38993, latitudeDelta:15, longitudeDelta:15},
animate:true,
regionFit:true,
userLocation:false
});
 
win.add(mapview);
 
var connectButton = Titanium.UI.createButton({
title:'Connect',
width:75,
style:Titanium.UI.iPhone.SystemButtonStyle.BORDERED
});
var closeButton = Titanium.UI.createButton({
title:'Close',
width:75,
style:Titanium.UI.iPhone.SystemButtonStyle.BORDERED
});
var clearButton = Titanium.UI.createButton({
title:'Clear',
width:75,
style:Titanium.UI.iPhone.SystemButtonStyle.BORDERED
});
 
var flexSpace = Titanium.UI.createButton({
systemButton:Titanium.UI.iPhone.SystemButton.FLEXIBLE_SPACE
});
var fixedSpace = Titanium.UI.createButton({
systemButton:Titanium.UI.iPhone.SystemButton.FIXED_SPACE,
width:50
});
win.setToolbar([flexSpace,connectButton,flexSpace,closeButton,flexSpace,clearButton,flexSpace]);
 
 
/////////////////////////
// Button event listeners
/////////////////////////
mapview.addEventListener('click', function(e) {
if (e.annotation) {
e.annotation.leftView.image = "http://img.tweetimag.es/i/" + e.title + '_n.png';
}
});
 
connectButton.addEventListener('click', function() {
try {
socket.connect();
Titanium.API.info('Opened!');
} catch (e) {
Titanium.API.info('Exception: '+e);
}
});
 
closeButton.addEventListener('click', function() {
try {
socket.close();
} catch (e) {
Titanium.API.info('Exception: '+e);
}
});
 
clearButton.addEventListener('click', function() {
mapview.removeAllAnnotations();
});
 
tabGroup.addTab(tab1);
tabGroup.open();
 
 
/////////////////////////
// Socket event listener
/////////////////////////
 
var socket = Titanium.Network.createTCPSocket({
hostName:'localhost',
port:6969,
mode:Titanium.Network.READ_MODE
});
 
socket.addEventListener('read', function(e) {
//Titanium.API.info(e['from'] + ':' + e['data'].text);
try {
var tweet = JSON.parse(e.data.text);
if (tweet.geo && tweet.user.screen_name && tweet.text) {
//Titanium.API.info(tweet.geo);
var profileImageView = Titanium.UI.createImageView({
width:32,
height:32
});
var tweetAnnotation = Titanium.Map.createAnnotation({
latitude: parseFloat(tweet.geo.coordinates[0]),
longitude: parseFloat(tweet.geo.coordinates[1]),
title: tweet.user.screen_name,
subtitle: tweet.text,
pincolor: Titanium.Map.ANNOTATION_RED,
leftView: profileImageView,
animate:true
});
mapview.addAnnotation(tweetAnnotation);
}
} catch (exception) {
Titanium.API.info('Exception: '+ exception);
}
});
 
tabGroup.addEventListener('close', function(e) {
if (socket.isValid) {
socket.close();
}
});
view raw tweet_flow.js hosted with ❤ by GitHub

That’s basically it. There’s a few bugs, but this isn’t about perfection, just a quick sample of these new technologies. Scary as it may be, this is pretty much where we are headed towards. A constant flow of real-time information, following us wherever we go. Can’t wait!

Is this particular application helpful? Maybe. I could see it being used to monitor emergency situations, or some type of big event. Problem is that very few people yet geo-tag their tweets. The average number of normal tweets was 750/second a month ago. Compare that to maybe 10 tweets a second of geo-tagged tweets…

Notes:

With the access role I have and the boxes I use, I’m getting about 5 tweets a second on the map. Don’t go thinking that the iPhone can support that many total annotations on the map like in the screenshot above (it’s the simulator).

They have yet to add socket support to the Android side of Titanium, so as of now this won’t work on the Android (though it’ll probably work without modification once they do).

A bunch of JSON parsing exceptions happen on the ‘read’ event of the client socket. I imagine it’s because more than one tweet might come in the JSON payload per packet or whatever triggers the ‘read’, so it freaks out.

The bugs I encountered were that the pin drop animation isn’t working in the latest version of Titanium, though I know they are on it. Also if you terminate the app it’ll be the asshole I mentioned above and not send the FIN to the socket. A curious observation is that the socket remains connected when the app goes into the background state (probably just sleeping or something).

Please feel free to comment below any best practices on this (I’m a newb), or any questions you may have.

  • Pingback: Tweets that mention Twitter Streaming API + node.js + Appcelerator Titanium = Real-time tweet map -- Topsy.com

  • http://siedrix.com Siedrix

    When you include “var TwitterNode = require(‘./twitter-node’).TwitterNode;” where did you got that module?

  • http://ecito.com ecito
  • http://siedrix.com Siedrix

    thx

  • saperduper

    excellent! thanks for the post!

  • Life0fun

    so this is working only on desktop emulator environment where you can set up node.js server….In real world, you first need to setup the node.js server in public domain, then configure your phone to connect to it…and do you have any number on power consumption under this scenario ?

  • http://alexshulman.com Alex Shulman

    This is very cool. I actually just saw this, I wrote a very similar twitter mapper with node.js Socket.io twitter-node. I really like how you dealt with the popups.

    http://alexshulman.com/map

  • Dada

    Web design Bangladesh

    really like your blog site…it is a great article…simply fantastic work…keep it up…
    http://www.immensesystem.com/

  • Mim

    Web development Bangladesh

    fantastic work…love your blog…
    http://www.immensesystem.com/

  • http://halsotips.info/ halsotips

    helpfull article. thanks for sharing

  • Alexandre Costa

    Hi, first of all congratulations.. very good article. I have some dificulties.. what twiiter-node in GitHub can I get, there are more than one and my code of server don’t work. Can you help me?

    My e-mail is alexandreafc@gmail.com

  • Michael Seguin

    This just makes me smile… Well done.