Refactoring an express.js application without IDE

Refactoring is scaring.

Without support from an IDE, it becomes even difficult. However, git can help. I have done refactoring for an express.js web application several times recently. I summaries the following rules to make the process less scaring.

Rule 1. branch before refactoring

If the refactoring is not going well, you can always go back.  In most cases, I have to touch the model, the view, and the controller. If you named your files according to RESTful resource name, it is very likely you need to update the file names.

Rule 2. rename files and then commit before touch the codes inside files

After add new files and rm old ones, git will recognized the changes are renaming files. If you make changes inside a renamed file, then the renaming will be lost in git history.

Rule 3. refactoring model first, then the views and client side JavaScript files, and finally the controller

In this sequence, you will be able to have a testable application after each code modification. When M-V-C are updated, and all tests passed, you can merge the refactoring branch back to the original one.

PUT to update a resource state

It is common that a resource's behavior changes according to its state. A online survey is such a resource. A survey can have two states: idle and active. In idle state, a survey does not accept user inputs, and it does in active state.  The state transition can be seen in the follow figure.

Survey state transition diagram

Now if we want to expose the status of the survey as a resource, then we will have the following interfaces.

Method	URL	Details
GET	/surveys/:id/state	Get the current state of a survey
PUT	/surveys/:id/state	Set the state of a survey

The PUT request can have the state to set in the request body. The response can be 200 with the set result. The response can also be 4xx, if the client has erred. The error can be 403 forbidden, 404 survey not found, or 409 conflict if the state to set is not supported.

I am surprised to see that POST was used to set survey state in Google's survey API. The interfaces are:

Method	URL	Details
POST	/surveys/:id/start	set the survey to be active
POST	/surveys/:id/stop	set the survey to be idle

This design has two problems compared to the previous design:

1. /surveys/:id/start and /surveys/:id/stop are not resources; and
2. POST is not idempotent, but PUT is. 

An Express middleware to check if a resource exists

After writing a similar code snippets for tens of times, I decided to write a middleware to handle it. The scenario is quite common. You want to check if an resource exists before going to the next step of processing. In my applications,  it is a query in the MongoDB.

/**
 * A middleware to check if id exists in collection
 * app.get('/resources/:id/', exist('id', collection), function(req, res){...}}
 * @param  {String} id            the parameter name of item id in req object
 * @param  {Model} collection     the collection model
 * @return {Function}             the middleware
 */
function exist(id, collection) {
  return function (req, res, next) {
    collection.findById(req.params[id]).exec(function (err, item) {
      if (err) {
        console.error(err);
        return res.send(500, 'Something wrong!');
      }

      if (!item) {
        return res.send(404, 'item ' + req.params[id] + ' not found');
      }
      
      req[req.params[id]] = item;
      next();
    });
  };
}

Introduction to REST prezi updated

I updated the prezi of an introduction to REST for next week's MSU Web Dev Cafe meeting next week. The major change was to add a case study to tell of a design is RESTful.

Forcing clients to reload on new application releases

Many Web application clients get data via Ajax. The resources including the CSS and JavaScript files  are never reloaded once loaded in this way. However, some new features require the clients to retrieve the updated resources from the server. This can be achieved by maintaining a service generated release number on the client, and compare that number with the current release number on the server via Ajax. If the release number is updated, then run

window.location.reload(true);

You will need to either disable the the cache for the resource of release number, or set cache to false of the Ajax GET request.

Resource representation generation on first GET retrieval

Intent

Provide a resource type that has a new resource representation instance generated on the first retrieval. In HTTP, this is often a GET request to a URL that the resource provides.

Also Known As

The closest pattern I can find is the Multiton pattern.

Motivation

I have implement this pattern in two scenarios:

An application provides the remote control of a specific CCD detector. The detector scans a sample and acquires an image on every scan spot. The native image format is not supported by Web browsers. The users can view the scan process on a page. The page will retrieve a new image once the acquisition finishes on a scan spot. The image is then converted to PNG, and sent to the client. The PNG image is saved, and all later requests will be served directly without conversion. The challenge, in this scenario, is that the application should convert the image to PNG only once. That implies that all the requests of that image before the PNG file is available are served together.

In the other scenario, a client retrieves a user's thumbnail photo from an application. The application gets a photo from an Active Directory when the photo is requested from the first time. The application saves the photo and serves it locally thereafter. Similar to the first scenario, the application should retrieve the photo from the AD service only once.

Design

When a resource representation is requested but is not available in the local file system, the request is put into an array storing all the requests for the same representation. Such an array needs to be put into a hash table, and its key is the resource's identifier. When a resource is requested for the first time, the resource is not available in the local file system, and the corresponding key is not in the hash table. Then the key will be created in the hash table, and the first request is pushed into the array. All following requests of the same resource are pushed into the array when the application is generating the resource representation. When the representation is ready, it is saved in the file system, and the key is removed from the hash table. All the requests in the array are served in a batch. The design can be implemented in various programming languages. There is a big difference between the implementation in a non-event-driven programming language like Java and that in an event-driven programming language like node.js.

Challenge 1: synchronization of the hash table

Adding and removing similar resource requests into the hash table have to be synchronized.

Java	node.js
We will have to use a concurrent util class like java.util.concurrent.ConcurrentHashMap(String, List).	A simple object like {"resource-identifier": []} will work.

Challenge 2: asynchronous processing

When the application is generating or retrieving the resource, we want the thread previously allocated to the request to be freed, and the handling of the request continues when the resource is finally available. Before Servlet 3, we will have to use something like Jetty continuation for this in Java. On the contrary, because node.js by nature has only a single thread, the processing is by default asynchronous.

Java	node.js
We will need put a Jetty continuation or a Servlet 3 AsyncContext into the hash map, and write the response from there when the resource is available.	Just put the standard http.ServerResponse instances in the array.

In order to improve performance, we can also add cache control to these generated resources in addition to the copy in the file system.

SSH connection drop issue for Ubuntu virtualbox guest on Windows 7 host with NAT network

The problem was really annoying when I need to ssh to some Linux servers from the Ubuntu virtualbox guest. As I remember, it started after I updated the VirtualBox platform to 4.3.26 or updated the guest from 12.04 to 14.04. The fix was surprisingly simple. See https://forums.virtualbox.org/viewtopic.php?f=6&t=66086#p313103

With the default Intel adapter, ssh connections were always dropped. When switching to the first non-Intel adapter, the issue was gone.