Deliver real-time information to your users using node.js

July 26th 2010

You must have already heard about node.js, if not, it’s an evented input/output server based on Google V8 engine. That means that as soon as the server receives information it can respond and send relevant information back to your front-end. You can then treat this information the way you want. What is so cool about it is that if you know javascript you can code a node.js server and it really is a lot of fun!

In this article, I’ll cover the basics to be able to build a simple server that will respond to an “admin” user input and then broadcast a message to all users connected on the website. Based on that, you’re then free to apply it to whatever needs you may have.

Please note that some basic server knowledge is required, like installing software with command line and forward requests to specific ports.

The demo

First open the live feed demo page, then open the live feed admin page. Try to resize them so you can see both, then click on the display buttons on the admin page, watch the magic happen on the live feed page!

The basics

How does that real-time thing work? Well it’s actually quite simple. Every time a user connects to my demo page, I open an ajax connection, that ajax connection wait for the server to send data. Once it receives data, it’s processed, the connection is closed and a new one is opened. This can be called Comet, long polling, ajax push, HTTP Streaming and a bunch of other names. You can also do long-polling in PHP but since node.js is a dedicated server it does a better job at handling the load.

The setup

If you’ve never installed node.js on your server, you first need to do it. The documentation is here. Make sure you have g++ installed on your server. Simply run: “sudo apt-get install g++” if you’re on Ubuntu, it’ll either tell you it’s up to date or install it.

Here’s how I proceeded to install node.js, 4 easy steps:

[/sources] > git clone git://github.com/ry/node.git
[/sources] > cd node
[/sources/node] > ./configure
...wait...
[/sources/node] > make
...wait...
[/sources/node] > sudo make install
...wait...
'install' finished successfully (0.193s)
DONE

So you should now be setup with node.js on your server, let the fun begin.

The server (link to the source)

This demo uses two main javascript file, one for the server and one for the client. Let’s begin by tearing down the server javascript file.

The first thing you need to do is load the modules required for your app, in this case we need the http, sys, url and querystring module.

var http = require("http"),
    sys = require("sys"),
    url = require("url"),
    qs = require("querystring");

I then declared my global variables for this applications, they are as follow:

var ITEMS_BACKLOG = 20;

var urlMap = {
  '/real_time_feed' : function (req, res) {
    var since = parseInt(qs.parse(url.parse(req.url).query).since, 10);
    feed.query(since, function (data) {
      res.simpleJSON(200, data);
    });
  },
  '/send_feed_item' : function (req, res, json) {
    feed.appendMessage( json );
    res.simpleJSON(200, {});
  }
}

The ITEMS_BACKLOG is the max number of items the server keeps in memory, this way you can make sur it never runs out of memory, you’ll see where it’s being used later.

I then declare my urlMap, these are the URLs the server listens to and the code to execute upon calling. It’s fairly straight forward, there’s one very important thing you need to know tho, it’s that since node.js runs on a different port than your normal webserver (a webserver usually runs on port 80), you need to forward the calls on those URLs to your node.js server port.

In my case, I use NGINX as my webserver and node.js runs on port 8001. Here’s what my NGINX configuration file looks like:

# Send request to node.js on port 8001
location ~* ^(/real_time_feed|/send_feed_item)$
{
  proxy_pass http://localhost:8001;
}

You see that I redirect both URLs in my “urlMap” to the node.js server port. Might not be the most efficient way but since you cannot do cross-domain posting in ajax, it’s the only solution I found for now.

Then comes the meat, the server itself, it’s fairly easy to create, you just need to invoke the “createServer” method and make it listen to a specific port.

http.createServer(function (req, res) {
...
}).listen(8001);

All requests and responses will be processed in here, so you need some code in there right? Here it is:

http.createServer(function (req, res) {
  // Get the url and associate the function to the handler
  // or
  // Trigger the 404
  handler  = urlMap[url.parse(req.url).pathname] || notFound;

  var json = "";

  if(req.method === "POST"){
    // We need to process the post but we need to wait until the request's body is available to get the field/value pairs.
    req.body = '';

    req.addListener('data', function (chunk) {
      // Build the body from the chunks sent in the post.
        req.body = req.body + chunk;
      })
      .addListener('end', function () {
        json = JSON.stringify(qs.parse(req.body));
          handler(req, res, json);
      });
  }else{
    handler(req, res);
  }

  res.simpleJSON = function (code, obj) {
    var body = JSON.stringify(obj);
    res.writeHead(code, {
      "Content-Type": "text/json",
      "Content-Length": body.length
    });
    res.end(body);
  };

}).listen(8001);

So the first thing I do is go get the function that need to be invoked in my urlMap and assign it to my handler. Then depending on the type of request, I either get the posted data and send a response or just straight invoke the handler of the URL called and send a response.

Now you’ll see that in my urlMap I call the feed object, this object has 2 methods, one for querying and one for appending content to the feed.

The “query” method, don’t respond until there’s valid content to send, that’s why the ajax call keeps running, it’s just there waiting. The “appendMessage” method…well she append content to my “real_time_items” array.

// Handles the feed push and querying.
var feed = new function () {
  var real_time_items = [],
      callbacks = [];

  this.appendMessage = function (json) {
    // Append the new item.
    real_time_items.push( json );

    // Log it to the console
    sys.puts(new Date() + ": " + JSON.parse(json).type + " pushed");

    // As soon as something is pushed, call the query callback
    while (callbacks.length > 0)
      callbacks.shift().callback([JSON.parse(json)]);

    // Make sur we don't flood the server
    while (real_time_items.length > ITEMS_BACKLOG)
      real_time_items.shift();
  };

  this.query = function (since, callback) {
    var matching = [];

    for (var i = 0; i < real_time_items.length; i++) {
          var real_time_item = real_time_items[i];
          if (real_time_item.timestamp > since)
            matching.push(real_time_item)
    }

    if (matching.length != 0) {
      callback(matching);
    } else {
      callbacks.push({ timestamp: new Date(), callback: callback });
    }
  };
};

That’s basically it for the server!

The client (link to the source)

What you need to do on the client side is very simple, you only need to start an ajax call that restarts when it receive a response:

function longPoll_feed () {
  //make another request
  $.ajax({
    cache: false,
    dataType: 'json',
    type: "GET",
    url: "/real_time_feed",
    error: function () {
      //don't flood the servers on error, wait 10 seconds before retrying
      setTimeout(longPoll_feed, 10*1000);
    },
    success: function (json) {
      display_event(json);

      //if everything went well, begin another request immediately
      //the server will take a long time to respond
      //how long? well, it will wait until there is another message
      //and then it will return it to us and close the connection.
      //since the connection is closed when we get data, we longPoll again
      longPoll_feed();
    }
});
}

So that ajax call listens to what is sent on the “real_time_feed” url. Once it receives data, it processes them.

Since it’s JSON that is returned to my views, I can then treat it the way I want. This file need to be included in the HTML file that will receive the live content.

The big merge

Ok so now you have your server javascript file and your client javascript file. The first thing you need to do is create an HTML page in which you’ll include the client.js file, I my case, it’s the index.html file.

Now for the server.js file, you actually need to connect via command line and start the node.js server like so:

[scaron] > node /path/to/server.js
Server running at http://127.0.0.1:8001/

And finally, to be able to send content to the server in real time, you need to POST content to the url “/send_feed_item”. If you remember in my URL map, I have setup this url to append content to my feed. So a simple ajax post should do the trick to post content, that’s what I do in my admin page.

$.post(
  'http://nodejs.no-margin-for-errors.com/send_feed_item',
   json,
  function(){
     // Done!
  }
);

The sources

All the sources are available on github, feel free to fork them and play with it yourself.

The possibilities

Let’s just say you watch a live conference on ustream.tv and the speaker mention some website, with node.js he could be able to send you to this website in real time as long as you are connected to his server or you could be watching live news on a website and they mention traffic, you could then be presented a Google traffic map live as they talk about it on the stream.

The possibilites are endless, this post covers only the basics, I’ll let your mind do the rest ;)

45 Responses to “Deliver real-time information to your users using node.js”


  1. […] This post was mentioned on Twitter by Stephane Caron, Dominic Mercier, Romain Tripault, rtmate3, RTmate2 and others. RTmate2 said: RT @PosAbsolute Deliver real-time information to your users using node.js http://bit.ly/8Z0kBn by @scaron […]

  2. Great post!

    Have you experience it with a real case project? If yes, please share the URL. Would be nice to see how you implement it.

    I see you are using NGINX. His it faster than Apache? Why are you preffering NGINX from Apache?
    Thanks

  3. Stephane says:

    @Dominic: I am working on a real case project using node, I just can’t disclose the details yet.

    As for NGINX, I use it because it’s using a lot less memory than apache. I’ve set my server up so the static files are served by NGINX and the dynamic served by apache. This way I get the low memory footprint and speed of NGINX but don’t loose any of the features apache can provide.

  4. Deliver real-time information to your users using node.js…

    This article cover the basics to be able to build a simple node.js server that will respond to an “admin” user input and then broadcast a message to all users connected on the website….

  5. […] Deliver real-time information to your users using node.js (tags: javascript node.js rtw programming) Categories: Links Comments (0) Trackbacks (0) Leave a comment Trackback […]

  6. […] Stephane Caron covers the basics of delivering real-time information to your users using node.js. […]

  7. Hey Stephane,

    ” I’ve set my server up so the static files are served by NGINX and the dynamic served by apache. This way I get the low memory footprint and speed of NGINX but don’t loose any of the features apache can provide.”

    How did you set this up? If you were using WordPress or Drupal, is this still achievable?

  8. Stephane says:

    @Kevin: It’s fairly easy. To do this, you need to run NGINX on port 80, the default web port and run apache on a secondary port, 8080 let’s say.

    That means that NGINX will receive all the requests, in the configuration file I just need to forward the calls to the apache port if it’s not for a static file.

    I can send you a sample config file if you want.

  9. Mauro says:

    Great tutorial! I have a question: this work only if client and server are on the same machine, right? I’m wondering if it’s possible to do the same with remote client, using JSONP maybe. Is it possible?

  10. Stephane says:

    @Mauro: Not really, the server.js file could be on a server dedicated for node.js and the client.js file could be on another server.

    The client.js file don’t directly communicate with the server.js.

    Hope it clears things up :)

  11. Mauro says:

    Ok, but your client use an ajax request to retrieve the data so if I use it on a site that is outsite my domain is a same-origin policy violation. Am I wrong? What I’m trying to do is a widget that a user can embed in his webpage and that is updated in real time when I push new data from the node.js server. Dont know if this make sense…

  12. Stephane says:

    @Mauro: It does make sense, and you are right, it needs to be under the same domain. There are some tricks to do cross-domain AJAX request, but they are not needed in that case.

  13. stoimen says:

    Great tutorial! I’m trying to write something very similar now and this can help me definitely!

  14. Hello

    Do you like travelling? So do I. What’s the most annoying thing in travelling? Expenses and time you must spend getting from one point to another. It’s difficult to find solution with time but you can do something with prices. You can choose bilety autokarowe that are really cheap and anyone can afford to buy them.

    Best regards

  15. Jolene says:

    Great read! You may want to follow up to this topic!

    cigars

  16. Ryan says:

    I’m trying to run the example and it is not working….has something changed? I’ve tried in FF and IE

  17. crai says:

    As of 19 Apr 2011, the demo doesn’t work, tested in FF 4.0.1 windows7

  18. Stephane says:

    Hi there,

    It’s because the node server was down. I’ve just brought it back up.

    Thanks!

  19. Liam Boyle says:

    Great article,I hope to be able to at least implement node.js for the purposes of a tutorial with knockout.js in the future, and this will help loads!

    WebDevelopment

  20. Santhosh T G V R S says:

    Great Post, but i have a couple of questions here.

    1. I copied the entire source folder and i navigated to js folder and typed “node server.js”. Now, i browsed to “localhost:8001″ and the result i got is “Not Found”. This is because the code has only two checks one is “/real_time_feed” and another is “/send_feed_item”. Do we need to implement for “/index.htm” or “/”, “/admin.html”, “/css/master.css” and “/js/client.js” also? For now i changed server.js to handle those requests as well.

    This is how i changed the code..

    var urlMap = {
    ‘/real_time_feed’ : function (req, res) {
    var since = parseInt(qs.parse(url.parse(req.url).query).since, 10);
    feed.query(since, function (data) {
    res.simpleJSON(200, data);
    });
    },
    ‘/send_feed_item’ : function (req, res, json) {
    feed.appendMessage( json );
    res.simpleJSON(200, {});
    },
    ‘/’ : function (req, res) {fs.readFile(“../index.html”, function(err, data)
    {
    if (err) {
    sys.puts(“Error loading ” + “index.html”);
    } else {
    headers = { “Content-Type”: “text/html”
    , “Content-Length”: data.length
    };
    headers[“Cache-Control”] = “public”;
    sys.puts(“static file ” + “index.html” + ” loaded”);
    res.writeHead(200, headers);
    res.end(data);
    }
    }
    )
    },
    ‘/admin.html’ : function (req, res) {fs.readFile(“../admin.html”, function(err, data)
    {
    if (err) {
    sys.puts(“Error loading ” + “admin.html”);
    } else {
    headers = { “Content-Type”: “text/html”
    , “Content-Length”: data.length
    };
    headers[“Cache-Control”] = “public”;
    sys.puts(“static file ” + “admin.html” + ” loaded”);
    res.writeHead(200, headers);
    res.end(data);
    }
    }
    )
    },
    ‘/css/master.css’ : function (req, res) {fs.readFile(“../css/master.css”, function(err, data)
    {
    if (err) {
    sys.puts(“Error loading ” + “master.css”);
    } else {
    headers = { “Content-Type”: “text/css”
    , “Content-Length”: data.length
    };
    headers[“Cache-Control”] = “public”;
    sys.puts(“static file ” + “master.css” + ” loaded”);
    res.writeHead(200, headers);
    res.end(data);
    }
    }
    )
    },
    ‘/js/client.js’ : function (req, res) {fs.readFile(“client.js”, function(err, data)
    {
    if (err) {
    sys.puts(“Error loading ” + “client.js”);
    } else {
    headers = { “Content-Type”: “text/javascript”
    , “Content-Length”: data.length
    };
    headers[“Cache-Control”] = “public”;
    sys.puts(“static file ” + “client.js” + ” loaded”);
    res.writeHead(200, headers);
    res.end(data);
    }
    }
    )
    }
    }

    2. server.js is running on localhost and port no 8001. now, how to redirect requests to node? I mean in this case i’m browsing “http://nodejs.no-margin-for-errors.com/”, but how come the node is listening to this?

  21. temon says:

    hi Stephane, I have tried to practice your tutorial. but I get some error here, and I dont know how fix this. pleas help me.

    from the client, http://localhost, I get this json :
    [{“type”:”google_map”,”content[lat]”:”37.331693″,”content[long]”:”-122.0307642″}]

    and if I see in your demo page is I should get the json like this :
    [{“type”:”google_map”,”content”:{“lat”:”37.331693″,”long”:”-122.0307642″}}]

    but if I see the post data is same like this :
    type=google_map&content%5Blat%5D=37.331693&content%5Blong%5D=-122.0307642

    I dont know, why the obj json changes to array in the respos json.

    I using node.js v0.4.11and use nginx too.

    thanks.

  22. It?s actually a nice and useful piece of information. I?m satisfied that you shared this helpful information with us. Please stay us informed like this. Thank you for sharing.

  23. I don?t even know the way I ended up here, however I assumed this publish was once good. I do not recognize who you’re however definitely you are going to a well-known blogger for those who aren’t already ;) Cheers!

  24. girley says:

    Great tutorial!
    But I got an issues when I tried the source codes on my local machine. I run server.js and view the result on the browser but Not Found appeared on my screen. I don’t know what’s the problem.I’m just a beginner in nodejs and right now we’re planning to develop an app using nodejs.Can you help me with this?Thank you so much!

  25. […] Read full article Categories: All, Async I/O, NodeJS, Real-Time Web Tags: JavaScript, Node.JS, NodeJS /* […]

  26. Z says:

    On the demo, my JS console says:
    “Failed to load resource: the server responded with a status of 502 (Bad Gateway)”
    “2event.layerX and event.layerY are broken and deprecated in WebKit. They will be removed from the engine in the near future.”
    and it doesn’t work. Any idea how come?

    Z

  27. pmb says:

    There is no way to POST to /send_feed_item from console, which makes this whole thing very useless.

  28. Ron says:

    Hi – tried the demo in 3 different browsers, but nothing at all happened. Does not seem to work…?

  29. Great tutorial
    I’m trying to write something very similar now and this can help me definitely!

    Thanks

  30. Stephane says:

    Hey guys,

    The nodeJS server was down, it’s now back up, the demos should work as intended.

    @pmb: This is intended as being an introduction to nodeJS nothing advanced, you’re free to take the code and build on it.

    Thanks

  31. Remy says:

    This is a CLIENT-side Polling, all clients makes unnecesary requests like this:

    Are there something new?———>NO
    Are there……
    .
    .
    1000000…….
    .
    .Are there something new?———>YES!! (finally)

    Please I need a server side poolling in nodejs, thanks.

  32. xbox.com says:

    video game xbox 360…

    To get strong one should have faith in one’s self and rely upon one’s strength. It will require strong heart to overcome every hardship that will come. Trust yourself and believe….

  33. cenobyte says:

    demo is not working…

  34. gronux says:

    don’t work error
    502 Bad Gateway

  35. Javi says:

    Hi!

    Demo is not working now.. (502 error).

    Thanks,

  36. Anil says:

    How to run this code under apache node.js enviornment

  37. I was looking for great explanation/tutorial on node.js and finally found one. Thanks for this great resource and keep up the good work!

  38. abhijit says:

    Demo is not working. I am getting an error while starting client by running node client.js. I ‘ve aised that issue on github.

  39. Horten says:

    Demo ist not working 502 bad gateway

  40. Taranov.Mike says:

    hello. Read your source. It look like while(1) read_from_server(); I learn JS two days, but can tell you that this is not JS way. I`m talking about that:
    “What you need to do on the client side is very simple, you only need to start an ajax call that restarts when it receive a response”

    Best choise (imho) use async_calls from server side.(node.js+now.js+socket.io). What you think about it?
    Best regards.Mike.

  41. Thanks for finally writing about >Deliver real-time information to your
    users using node.jsStéphane Caron – No Margin For
    Errors <Loved it!

Leave a Reply