Tuesday, March 17, 2015

request.get(...).pipe(response) is not RESTful

Request is a nice package for HTTP client programming on node.js. It provides API's that can directly stream. So you can do things like
request('http://google.com/doodle.png').pipe(fs.createWriteStream('doodle.png'))
When programming a service proxy, it is straightforward to do something like
app.get('/proxied/service', function (req, res) {
    request.get('http://the/real/service').pipe(res);
})
It is so neat and powerful. However, it is not RESTful, and in some cases it could cause errors that is hard to figure out. When the client of the proxied service gets the response, it always expects some representation from your service not the real service. When the representation of 'http://the/real/service' was sent to the client, the client will be confused. For example, the authorization header, the cache-control header, or the cookie header can all have side effects.

In an application I develop, I had a proxied service just like the above. It provided a list in JSON that was fetched from the other service hosted by a different application. It ran well in test and first a few month after released, until one day a user complained that he cannot access the list that should appear in the suggestion of a text input. I found the service was down. However, the user should still be able to type in whatever s/he thought was right, and then save it on the application. But the issue was that the application rejects such a request for authentication reason if the user's hand was not quick enough.  It took me a while trying to reproduce the problem on my browser, and then figure it out.

The problem was that the remote service was down, and the client's browser got a "net::ERR_EMPTY_RESPONSE" error that basically means the server closed the connection without sending any data. That was only first half of the story. Such an error will force the browser to clean up the cookies set by the application, and all following requests from the same page will be rejected. So the proper way is still to provide a timeout, and an error handler like
   request({
      url: service.device.url,
      timeout: 30 * 1000,
      ...
    }, function (err, response, resBody) {
      if (err) {
        return res.json(503, ...);
      }
     ...
    });

500, 502, 503 are good codes for this case.