How to make nodejs applications modular

Modules come in multiple flavors. To name a few, function, objects, classes, configuration metadata, initialization data or server can all be treated as modules. We will explore ways to modularize Routes, Middleware, Services, Models and utility libraries in a realistic setting.

This article is more technical, “How to modularize nodejs applications” offers a more theoretical approach to think about modularization.

As a recap, two key elements are going to be leveraged to modularize components as stated in the first article, at a glance module.exports ES5 utility, its ESNext import/export utility equivalents, and usage of the index file at every top-level directory.

The technique we are going to explore is based on two enhancements: make most of the things importable(or exportable depending on perspective), and adding an index file to every level of the project directory structure.

If you haven't read it yet, the followup to article is Overview on testing nodejs applications. You may give it a bird-eye-view or test drive to get a sense of how testing nodejs will look like.

In this article we will talk about:

Even though this blogpost was designed to offer complementary materials to those who bought my Testing nodejs Applications book, the content can help any software developer to tuneup working environment. You use this link to buy the book. Testing nodejs Applications Book Cover

Show me the code

var express = require('express'),
    path = require('path'),
    n = path.normalize,
    j = path.join,
    app = express();

/**Data Layer*/
var mongodb = require("mongodb");
mongoose.connect('mongodb://localhost:27017/devdb');
var User = require('./models').User; 

/**
 * Essential Middelewares 
 */
app.use(express.logger());
app.use(express.cookieParser());
app.use(express.session({ secret: 'angrybirds' }));
app.use(express.bodyParser());
app.use((req, res, next) => { /** Adding CORS support here */ });

app.use((req, res) => res.sendFile(n(j(__dirname, 'index.html'))));


/** .. more routes + code for app ... */
app.get('/', function (req, res) {
  return res.send('Hello World!');
});


/** code that initialize everything, then comes this route*/
app.get('/users/:id', function(req, res, next){
  User.findById(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
});

/**
 * More code, more time, more developers 
 * Then you realize that you actually need:
 */ 
app.get('/admin/:id', function(req, res, next){
  User.findById(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
});
/**
 * This would work just fine, but we may also have a requirement to listen to Twitter changes 
app.listen(port, function () {
  console.log('Example app listening on port 3000!')
});
*/

var server = require('http').createServer(app);
server.listen(app.get('port'), () => console.log(`Listening on ${ process.env.PORT || 8080 }`));
var wss = require('socket.io')(server);
//Handling realtime data
wss.on('connection'(socket, event) => {
    socket.on('error', () => {});
    socket.on('pong', () => {});
    socket.on('disconnect', () => {});
    socket.on('message', () => {});
});

Example:

What can possibly go wrong?

How do we go from the code posted above to a code that looks like as in the snippet below?

Looking at the code above, one would wonder how easy, or hard, maintenance would turn out to be when asked to add or sunset a feature from a program similar to the one presented above.

When trying to figure out how to approach modularization, the following key points may be a challenge:

When done right, ideally, we will have a code that looks like the following at the end of the exercise.

var express = require('express'),
  app = express(), 
  http = require('http'),
  server = http.createServer(app);

...
require('./config')
require('./utils/mongodb');
require('./utils/middleware')(app);
require('./routes')(app);
require('./realtime')(app, server);
...

server.listen(app.get('port'));
module.exports.server = server;

Example:

The following sections will explore more on making points stated above work.

The need to have a modular system

The need to have a Layered Architecture

Large codebases tend to be hard to maintain and nodejs applications are not an exception to this reality. Updates in 3rd party integrations, the evolution of language or libraries are some of the reasons to revisit codebase under hibernation.

When talking about the layered architecture, we will be referencing two additional layers: the introduction of the controller, the model, and later on the service layer.

A layered architecture makes sure systems components are decoupled, and changes in any of the components of the integrated system do not directly translate into a rewrite of dependent modules.

For instance, layered architectures make sure swapping a payment system from an expensive one to a cheaper one does not cause a service disruption. It also makes sure both systems can actually work in tandem.

Another example that may strike your interest is when there is a change in a route handler, the route declaration should stay intact. Ideally, write once and forget.

Every step of the way, while making our application layered, we will keep in mind above mentioned details.

The need to make nodejs server modular

Disambiguation: Server. The server is not in terms of the computer that runs the code, but the actual code that serves as an entry point(server) to the node application.

The nodejs server should be easy to test, like any other application component. To be able to test a nodejs server in isolation, we have to be able to import a nodejs server as a module. A server, by definition, requires the use of network resources. Those resources are expensive read/write operations, may introduce unwanted side effects, when not mocked out in unit tests. Hence, another reason to make nodejs server modular, is a need to mock expensive read/write operations that may impede test performance.

More on how to modularize nodejs servers is explained in Modularization of nodejs servers

Modularization of nodejs server

Modularization of nodejs server, as for any other module forming procedure, requires 2 steps. The first step is to make sure the server is exportable as any other object. The second step is to provide access to it, via index.

//in server.js 
var express = require('express'),
    app = express();
var server = require('http').createServer(app);
    server.listen(app.get('port'), () => console.log(`Listening on ${ process.env.PORT || 8080 }`));

//Modularizing the server 
module.export = server;

//Adding server export in index.js
module.export = require('./server');

Once this process is done, it becomes possible to import the server be in unit tests, or other sections.

The above code has a caveat: every time we require the server, the script will automatically start listening. To prevent this from happening, the server initialization code can be moved into a configurable function, also known as a thunk.

That modification can make the code look as in following example:

module.export.server = function(app){
  return function(){
    var m = `Listening on ${ process.env.PORT || 8080 }`;
    var server = require('http').createServer(app);
    server.listen(app.get('port'), () => console.log(m));
    return server;
  }
}

This modification makes sure, that as long as the function var server = require('./server')(app) has not been called yet, the server.listen will not be listening.

The need to make expressjs routes modular

The need for a router and route handlers. The router is obviously a cornerstone in expressjs application.

For the nodejs only use case, the introduction of a router may be subtle, especially when there is no clear reason to respond to client requests, for instance on command line applications or scripts designed to be run by a server process.

However, in case we need to respond to some requests, it is possible that some request handlers start looking a little like copycats, past a certain threshold. Whence, the need to make route handlers less repetitive and more reusable.

expressjs route handler comes into the picture, to simplify and avoid boilerplate when it comes to HTTP request/response operations.

Separation of route handler from the route ~ or why tightly coupled handlers are not a good idea.

The route constitutes one part of the application that will not change quite often. The frequency of change depends on how modular the route handler turns out to be.

When the time comes to test a route, it may be hard to isolate one route from the rest of the pack. Since isolated tests require the availability of one piece of code under test to be available for test, we have no other choice but to make routes importable/exportable.

The next step showcase how modularization of a route can be achieved in a few steps.

Modularization of expressjs Routes

Modularization of expressjs route, as for any other module forming procedure, requires 2 steps at a minimum. The first step is to make sure a route is exportable. The second step is to provide access to it via an index file.

To make the route “exportable” means to extract the route out of the route's definition.

More on how to modularize router, with manifest routes technique, is explained in Modularizing nodejs applications ~ Manifest routes

//Get this construct off server.js 
app.get('/users/:id', function(req, res, next){
  User.findById(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
});

// First iteration makes function exportable ~ in route/user.js
var router = require('express').Router();
router.get('/users/:id', function(req, res, next){
  User.findById(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
});
module.export = router;

// Second iteration ~ in route/index.js
module.export = require('./route/user');

It is always possible to do better, while at the same time staying within the limit of acceptable refactoring. One of those things we can consider is naming and extracting the route handler from the route definition. This process does the groundwork to move transform route handlers into controllers.

// Router definition ~ in route/user.js 
router.get('/users/:id', function(req, res, next){
  User.findById(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
});

//Naming and extracting route handler results in: 
function getUsersById(req, res, next){
  User.findById(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
}
router.get('/users/:id', getUsersById);

The need of a middleware

In expressjs context and nodejs in general, the middleware provides a way to do validation/verifications before continuing with the rest of the router.

Some of the commonly used middleware are:CORS, authentication(passportjs), JSON(transforming a body into a JSON object), route logging, to name a few.

Modularization of middleware

Modularization of middleware, as for any other module forming procedure, requires 2 steps at a minimum. The first step is to make sure the middleware is exportable as any other function. The second step is to provide access to it, via index.

The extra step that may be optional in certain instances, is to identify opportunities to create a new middleware and attach middleware from one point instead of multiple imports.

/**
 * Essential Middelewares 
 */
app.use(express.logger());
app.use(express.cookieParser());
app.use(express.session({ secret: 'angrybirds' }));
app.use(express.bodyParser());
//Opportunity to modularize 
app.use((req, res, next) => { /** Adding CORS support here */ });

//First step ~ extract cors middleware + move to middleware/cors.js 
function cors(req, res, next) => { /** Adding CORS support here */ }
app.use(cors);

//Second step ~ Export via middlware/index.js
module.export = require('./cors');

The extra step would be to group middleware calls into a thunk. This kind of module makes it possible to configure a function and use it later on.

//Add initialization function in utils/middleware.js
module.export = function middleware(express, app){
  //optional
  //return function(){ 
    app.use(express.logger());
    app.use(express.cookieParser());
    app.use(express.session({ secret: 'angrybirds' }));
    app.use(express.bodyParser());
    app.use(cors());
  //}
};

//How to use the new middleware in server.js
require('./utils/middleware')(express, app);

The need of a controller

Modularization of routes leaves one remnant that can be improved upon: the request handler is tightly coupled with the route it serves. That makes re-usability a little hard, isolated testing a bit hacky and developers' life a little more miserable.

One further step to modularization is the introduction of a standalone handler. Generally speaking, with some exceptions, when a route handler is extracted from the rest of the route, the resulting code qualifies as a controller. A controller can be interchangeable between related and unrelated routes alike.

The need of a controller, or a controller layer, is independent of a router, or routing system adopted. This makes it possible to pass around controllers to various routing systems, as long as they adhere to some sort of similar request handler function signature.

Modularization of controllers

Three steps are required to make controllers modular. The first two steps have been describing in the introduction: introduction of import/export constructs and the index. The first step includes the ability to make route handler exportable, and by that to eject the handler out of route definition.

Modularization of expressjs route handlers as controllers.

Modularization of expressjs controller, as for any other module forming procedure, requires 2 steps at a minimum. The first step is to make sure a controller is exportable. The second step is to provide access to it, via index.

In the case of expressjs, which is really not applicable to controllers outside expressjs framework, the controller is basically the request/response handler.

//Ejecting handler out of route definition + export to controller/user.js 
//The filename can also be user/controller.js for organization by feature projects 
function getUsersById(req, res, next){
  User.findById(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
}

//First interation ~ export handler as a controller in controller/user.js  
module.export = getUserById;

// Second iteration ~ in user/controller/index.js
//OR controller/user/index.js for organization by category projects 
module.export = require('./user/controller');

Following this logic, a controller is nothing but the ejected route handler.

The need of a model layer

The model is needed to translate application business logic into well-organized data structures within a program. That idea is mainly why a model is needed.

Having a data model layer is a whole new story. Managing models become cumbersome in some circumstances. That is why there is a plethora of ORM/ODM and whatnot. These tools bring DRY to model management.

Separation of Model from Route. A tightly coupled model/router combo is not always a good idea, particularly when the application has a tendency to grow beyond database fetch and dump scenarios. Keeping models tightly coupled to the route will make it difficult to test, scale, and reduce code duplications. Code duplications are known to increase technical debt.

More on how to modularize the router is explained in How to modularize expressjs routes

Modularization of model layers

Modularization of the model, as for any other module forming procedure, requires 2 steps at a minimum. The first step is to make sure the model is exportable as any other function or object. The second step is to provide access to it, via index.

It worths to mention that model definitions may come with accompanying schema definition. This is usually the case when coupling nodejs application to a mongodb database, via mongoose DRM(Document Relational Mapping).

//In utils/mongodb.js we initialize the mongodb 
var mongodb = require("mongodb");
mongoose.connect('mongodb://localhost:27017/devdb');
module.export = mongodb;

//Can be exported in utils/index.js 
module.export = require('./utils/mongodb');

//How to use this
require('./utils/mongodb');

The model layer is composed of the database connection and initialization alone. There is also schema definition that can be done. The order in which we load and compile schema to modules is as following: connect to the database first, then load models.

The need for real-time data

The need for real-time data — most modern applications require a stream of information to be pushed to customers near real-time. Some applications are designed with real-time capabilities at their core, others have the real-time element as an add-on.

For instance, a stock trading application, cannot afford to lose a minute sending updates to customers who need such invaluable information as soon as a change happens on stock prices. The same applies to maps and instant communication applications. Realtime capability is a core feature in such applications.

WebSocket is an extension of the HTTP protocol that makes near real-time magic happen.

Decoupling the real-time portion of the application makes maintenance a little easier to test, deploy and maintain.

Modularization of WebSocket handlers

How to modularize the WebSocket?

Modularization of WebSocket handlers, as for any other module forming procedure, requires 2 steps at a minimum. The first step is to make sure the handlers are exportable as any other function. The second step is to provide access to them, via index.

We have to make sure we isolate WebSocket objects, as the event model of the WebSocket implementation requires the availability of a network, which may not be available in a unit testing context.

Even if the networks were available, unit testing on a network is expensive and should be mocked out to make sure we have small and fast test cases.

More on how to modularize WebSocket is available on How to modularize socket.io/expressjs application 

var express = require('express'),
    app = express(), 
    http = require('http'),
    server = http.createServer(app),
    wss = require('socket.io')(server);
//Handling realtime data
wss.on('connection', (socket, event) => {
    socket.on('error', () => {});
    socket.on('pong', () => {});
    socket.on('disconnect', () => {});
    socket.on('message', () => {});
});

//in ./realtime.js 
module.export = function(app, server){
  var wss = require('socket.io')(server);
  //Handling realtime data
  wss.on('connection', (socket, event) => {
      socket.on('error', onError);
      socket.on('pong', onPong);
      socket.on('disconnect', onDisconnet);
      socket.on('message', onMessage);
  });
}

//How to use: in server.js 
require('./realtime')(app, server)

The need for a service layer

The MVC approach decouples the model from the view, and both are supposed to be held together by a controller. However, using models inside a controller assumes that the database or ORM used to negotiate data with the underlying database will not change over time. That is not always the case.

The API from the same ORM/ODM may change over time. The ORM/ODM maybe sunset at one point. Data that used to come from one kind of database, may be moved to another database.

All of the above changes require our model to be resilient and ready to adapt to change over time.

The examples provided above are all related to the data model, but integration with third-party services can well fall under the same category.

To sum up, there a clear need to abstract business logic in a service layer, that we changes are under the limits of what we can control. There is a clear need for the modularization of business logic under the service banner.

Modularization of the service layer

There are 3 steps we can take to make the service layer, modular. There are two steps already mentioned in the introduction. The last and more critical is to group related individual model handling modules, or grouping third party integration links into a service.

The following module can be used a service:

//initial request handler 
module.exports = function(req, res, next){
 return User.findById(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
}

//service declaration
function UserService(){}

UserService.prototype.getUser = function(id, next){
  return User.findById(id, function(error, user){
    return next(error, user);
  });
};
module.exports = UserService;

//request|controller handler 
module.exports = function(req, res, next){
 return new UserService().getUser(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
}

The signature of findById may change in the future, or the function is renamed. Whatever happens, we have one single place in our library that is subject to such a change.

More on how to modularize the service layer Modularize nodejs service layer

The need for a configuration layer

The service layer, the controller layer, the model layer, servers, and whatnot all require pre-determined environment variables that enable those layers to work from one environment to the next. Those variables are known under the configurations moniker. Those variables are sensitive to unintentional leaks to the public. Modularization of such a layer, in part, resolves this issue.

When it comes to databases, there is a need to modularize database connection strings and secrets and get them out of the codebase. This idea can be extended to other secrets and variables needed to run the application.

Modularization of configuration layer

There is a library that brings server configurations in mists of the application. The beauty of this library is its ability to completely separate configuration from code.

//in utils/config.js 
var dotenv = require('dotenv'),
  dotenvExpand = require('dotenv-expand');

module.export = function(){
  var config = dotenv.config();
  dotenvExpand(config);
  return config;
}

//in utils/index.js
module.export = require('./config')();

//Usage 
let config = require('./config');

More on how to modularize configurations variables Modularize nodejs application configuration

To sum up

As recapitulation, the code we exposed at the beginning of the current article may end up looking something like this:

var express = require('express')
var app = express();
//@todo add modules here to make the server a bit smaller and granular 
...
require('./config')
require('./utils/mongodb');
require('./utils/middleware')(express, app);
require('./routes')(app);
require('./realtime')(app, server)
...
module.exports.server = server;

Example:

Conclusion

In this article, we revisited how modularization can be achieved by leveraging the power of module.exports( or export in ES7+). The variety and flavors of components candidate to modularize, make it imperative that the modularization has to be minimalist, that is the reason why we leveraged the index file to make sure we do not overload already complex architectures. There are additional deeper complementary materials in the “Testing nodejs applications” book, around modularization and project layout.

References