July 26th 2010

You must have already heard about node.js, if not, it’s an evented input/output server based on Google V8 engine. That means that as soon as the server receives information it can respond and send relevant information back to your front-end. You can then treat this information the way you want. What is so cool about it is that if you know javascript you can code a node.js server and it really is a lot of fun!

In this article, I’ll cover the basics to be able to build a simple server that will respond to an “admin” user input and then broadcast a message to all users connected on the website. Based on that, you’re then free to apply it to whatever needs you may have.

Please note that some basic server knowledge is required, like installing software with command line and forward requests to specific ports.

The demo

First open the live feed demo page, then open the live feed admin page. Try to resize them so you can see both, then click on the display buttons on the admin page, watch the magic happen on the live feed page!

The basics

How does that real-time thing work? Well it’s actually quite simple. Every time a user connects to my demo page, I open an ajax connection, that ajax connection wait for the server to send data. Once it receives data, it’s processed, the connection is closed and a new one is opened. This can be called Comet, long polling, ajax push, HTTP Streaming and a bunch of other names. You can also do long-polling in PHP but since node.js is a dedicated server it does a better job at handling the load.

The setup

If you’ve never installed node.js on your server, you first need to do it. The documentation is here. Make sure you have g++ installed on your server. Simply run: “sudo apt-get install g++” if you’re on Ubuntu, it’ll either tell you it’s up to date or install it.

Here’s how I proceeded to install node.js, 4 easy steps:

[/sources] > git clone git://github.com/ry/node.git
[/sources] > cd node
[/sources/node] > ./configure
...wait...
[/sources/node] > make
...wait...
[/sources/node] > sudo make install
...wait...
'install' finished successfully (0.193s)
DONE

So you should now be setup with node.js on your server, let the fun begin.

The server (link to the source)

This demo uses two main javascript file, one for the server and one for the client. Let’s begin by tearing down the server javascript file.

The first thing you need to do is load the modules required for your app, in this case we need the http, sys, url and querystring module.

var http = require("http"),
    sys = require("sys"),
    url = require("url"),
    qs = require("querystring");

I then declared my global variables for this applications, they are as follow:

var ITEMS_BACKLOG = 20;

var urlMap = {
  '/real_time_feed' : function (req, res) {
    var since = parseInt(qs.parse(url.parse(req.url).query).since, 10);
    feed.query(since, function (data) {
      res.simpleJSON(200, data);
    });
  },
  '/send_feed_item' : function (req, res, json) {
    feed.appendMessage( json );
    res.simpleJSON(200, {});
  }
}

The ITEMS_BACKLOG is the max number of items the server keeps in memory, this way you can make sur it never runs out of memory, you’ll see where it’s being used later.

I then declare my urlMap, these are the URLs the server listens to and the code to execute upon calling. It’s fairly straight forward, there’s one very important thing you need to know tho, it’s that since node.js runs on a different port than your normal webserver (a webserver usually runs on port 80), you need to forward the calls on those URLs to your node.js server port.

In my case, I use NGINX as my webserver and node.js runs on port 8001. Here’s what my NGINX configuration file looks like:

# Send request to node.js on port 8001
location ~* ^(/real_time_feed|/send_feed_item)$
{
  proxy_pass http://localhost:8001;
}

You see that I redirect both URLs in my “urlMap” to the node.js server port. Might not be the most efficient way but since you cannot do cross-domain posting in ajax, it’s the only solution I found for now.

Then comes the meat, the server itself, it’s fairly easy to create, you just need to invoke the “createServer” method and make it listen to a specific port.

http.createServer(function (req, res) {
...
}).listen(8001);

All requests and responses will be processed in here, so you need some code in there right? Here it is:

http.createServer(function (req, res) {
  // Get the url and associate the function to the handler
  // or
  // Trigger the 404
  handler  = urlMap[url.parse(req.url).pathname] || notFound;

  var json = "";

  if(req.method === "POST"){
    // We need to process the post but we need to wait until the request's body is available to get the field/value pairs.
    req.body = '';

    req.addListener('data', function (chunk) {
      // Build the body from the chunks sent in the post.
        req.body = req.body + chunk;
      })
      .addListener('end', function () {
        json = JSON.stringify(qs.parse(req.body));
          handler(req, res, json);
      });
  }else{
    handler(req, res);
  }

  res.simpleJSON = function (code, obj) {
    var body = JSON.stringify(obj);
    res.writeHead(code, {
      "Content-Type": "text/json",
      "Content-Length": body.length
    });
    res.end(body);
  };

}).listen(8001);

So the first thing I do is go get the function that need to be invoked in my urlMap and assign it to my handler. Then depending on the type of request, I either get the posted data and send a response or just straight invoke the handler of the URL called and send a response.

Now you’ll see that in my urlMap I call the feed object, this object has 2 methods, one for querying and one for appending content to the feed.

The “query” method, don’t respond until there’s valid content to send, that’s why the ajax call keeps running, it’s just there waiting. The “appendMessage” method…well she append content to my “real_time_items” array.

// Handles the feed push and querying.
var feed = new function () {
  var real_time_items = [],
      callbacks = [];

  this.appendMessage = function (json) {
    // Append the new item.
    real_time_items.push( json );

    // Log it to the console
    sys.puts(new Date() + ": " + JSON.parse(json).type + " pushed");

    // As soon as something is pushed, call the query callback
    while (callbacks.length > 0)
      callbacks.shift().callback([JSON.parse(json)]);

    // Make sur we don't flood the server
    while (real_time_items.length > ITEMS_BACKLOG)
      real_time_items.shift();
  };

  this.query = function (since, callback) {
    var matching = [];

    for (var i = 0; i < real_time_items.length; i++) {
          var real_time_item = real_time_items[i];
          if (real_time_item.timestamp > since)
            matching.push(real_time_item)
    }

    if (matching.length != 0) {
      callback(matching);
    } else {
      callbacks.push({ timestamp: new Date(), callback: callback });
    }
  };
};

That’s basically it for the server!

The client (link to the source)

What you need to do on the client side is very simple, you only need to start an ajax call that restarts when it receive a response:

function longPoll_feed () {
  //make another request
  $.ajax({
    cache: false,
    dataType: 'json',
    type: "GET",
    url: "/real_time_feed",
    error: function () {
      //don't flood the servers on error, wait 10 seconds before retrying
      setTimeout(longPoll_feed, 10*1000);
    },
    success: function (json) {
      display_event(json);

      //if everything went well, begin another request immediately
      //the server will take a long time to respond
      //how long? well, it will wait until there is another message
      //and then it will return it to us and close the connection.
      //since the connection is closed when we get data, we longPoll again
      longPoll_feed();
    }
});
}

So that ajax call listens to what is sent on the “real_time_feed” url. Once it receives data, it processes them.

Since it’s JSON that is returned to my views, I can then treat it the way I want. This file need to be included in the HTML file that will receive the live content.

The big merge

Ok so now you have your server javascript file and your client javascript file. The first thing you need to do is create an HTML page in which you’ll include the client.js file, I my case, it’s the index.html file.

Now for the server.js file, you actually need to connect via command line and start the node.js server like so:

[scaron] > node /path/to/server.js
Server running at http://127.0.0.1:8001/

And finally, to be able to send content to the server in real time, you need to POST content to the url “/send_feed_item”. If you remember in my URL map, I have setup this url to append content to my feed. So a simple ajax post should do the trick to post content, that’s what I do in my admin page.

$.post(
  'http://nodejs.no-margin-for-errors.com/send_feed_item',
   json,
  function(){
     // Done!
  }
);

The sources

All the sources are available on github, feel free to fork them and play with it yourself.

The possibilities

Let’s just say you watch a live conference on ustream.tv and the speaker mention some website, with node.js he could be able to send you to this website in real time as long as you are connected to his server or you could be watching live news on a website and they mention traffic, you could then be presented a Google traffic map live as they talk about it on the stream.

The possibilites are endless, this post covers only the basics, I’ll let your mind do the rest 😉