How to modularize nodejs applications

divide et impera

One of the key issues working with large-scale nodejs applications is the management of complexity. Modularization shifts focus to transform the codebase into reusable, easy-to-test modules. This article explores some techniques used to achieve that.

This article is more theoretical, “How to make nodejs applications modular” may help with that is more technical.

In this article we will talk about:

Even though this blog post was designed to offer complementary materials to those who bought my Testing nodejs Applications book, the content can help any software developer to tuneup working environment. You use this link to buy the book. Testing nodejs Applications Book Cover

Show me the code

This piece of code is going to go through modularization in “How to make nodejs applications modular” blog. As for now, we will highlight failures and points of interest down below.

var express = require('express');
var app = express();

/**Data Layer*/
var mongodb = require("mongodb");
mongoose.connect('mongodb://localhost:27017/devdb');
var User = require('./models').User; 

/**
 * Essential Middelewares 
 */
app.use(express.logger());
app.use(express.cookieParser());
app.use(express.session({ secret: 'angrybirds' }));
app.use(express.bodyParser());
app.use((req, res, next) => { /** Adding CORS support here */ });

app.use((req, res) => res.sendFile(path.normalize(path.join(__dirname, 'index.html'))));


/** .. more routes + code for app ... */
app.get('/', function (req, res) {
  return res.send('Hello World!');
});


/** code that initialize everything, then comes this route*/
app.get('/users/:id', function(req, res, next){
  User.findById(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
});

/**
 * More code, more time, more developers 
 * Then you realize that you actually need:
 */ 
app.get('/admin/:id', function(req, res, next){
  User.findById(req.params.id, function(error, user){
    if(error) return next(error);
    return res.status(200).json(user);
  });
});
/**
 * This would work just fine, but we may also have a requirement to listen to Twitter changes 
app.listen(port, function () {
  console.log('Example app listening on port 3000!')
});
*/

var server = require('http').createServer(app);
server.listen(app.get('port'), () => console.log(`Listening on ${ process.env.PORT || 8080 }`));
var wss = require('socket.io')(server);
//Handling realtime data
wss.on('connection'(socket, event) => {
    socket.on('error', () => {});
    socket.on('pong', () => {});
    socket.on('disconnect', () => {});
    socket.on('message', () => {});
});

Example:

What can possibly go wrong?

When trying to navigate strategies around modularization of nodejs applications, the following points may be a challenge:

The following sections will explore more on making points stated above work.

Modules

In nodejs context, anything from a variable to function, to classes, or an entire library qualifies to become modules.

A module can be seen as an independent piece of code dedicated to doing one and only one task at a time. The amalgamation of multiple tasks under one abstract task, or one unit of work, is also good module candidates. To sum up, modules come in function, objects, classes, configuration metadata, initialization data, servers, etc.

Modularization is one of the techniques used to break down a large software into smaller malleable, more manageable components. In this context, a module is treated as the smallest independent composable piece of software, that does only one task. Testing such a unit in isolation becomes relatively easy. Since it is a composable unit, integrating it into another system becomes a breeze.

Leveraging exports

To make a unit of work a module, nodejs exposes import/export, or module.exports/require, utilities. Therefore, modularization is achieved by leveraging the power of module.exports in ES5, equivalent to export in ES7+. With that idea, the question to “Where to start with modularization?” becomes workable.

Every function, object, class, configuration metadata, initialization data, or the server that can be exported, has to be exported. That is how Leveraging module.exports or import/export utilities to achieve modularity looks like.

After each individual entity becomes exportable, there is a small enhancement that can make importing the entire library, or modules, a bit easier. Depending on project structure, be feature-based or kind-based.

At this point, we may ask ourselves if the technique explained above can indeed scale. Simply put, Can the techniques explained above scale?

The large aspect of large scale application combines Lines of Code(20k+ LoC), number of features, third party integrations, and the number of people contributing to the project. Since these parameters are not mutually exclusive, a one-person project can also be large scale, it has to have fairly large lines of code involved or a sizable amount of third-party integrations.

nodejs applications, as a matter of fact like any application stack, tend to be big and hard to maintain past a threshold. There is no better strategy to manage complexity than breaking down big components into small manageable chunks.

Large codebases tend to be hard to test, therefore hard to maintain, compared to their smaller counterparts. Obviously, nodejs applications are no exception to this.

Leveraging the index

Using the index file at every directory level makes it possible to load modules from a single instruction. Modules at this point in time, are supposed to be equivalent or hosted in the same directory. Directories can mirror categories(kind) or features, or a mixture of both. Adding the index file at every level makes sure we establish control over divided entities, aka divide and conquer.

Divide and conquer is one of the old Roman Empire Army techniques to manage complexity. Dividing a big problem into smaller manageable ones, allowed the Roman Army to conquer, maintain and administer a large chunk of the known world in middle age.

Scalability

How the above techniques can be applied at scale

The last question in this series would be to know if the above-described approach can scale. First, the key to scalability is to build things that do not scale first. Then when scalability becomes a concern, figure out how to address those concerns. So, the first iteration would be supposed to not be scalable.

Since the index is available to every directory, and the index role becomes to expose directory content to the outer world, it doe not matter if the directory count yields 1 or 100 or 1000+. A simple call to the parent directory makes it possible to have access to 1, 100, or 1000+ libraries.

From this vantage point, introduction of the index at every level of the directory comes with scalability as a “cherry on top of the cake”.

Where to go from here

This post focused on the theoretical side of the modularization business. The next step is to put techniques described therein put to test in the next blog post.

Conclusion

Modularization is a key strategy to crafting reusable composable software components. It brings elegance to the codebase, reduces copy/paste occurrences(DRY), improves performance, and makes the codebase testable. Modularization reduces the complexity associated with large-scale nodejs applications.

In this article, we revisited how to increase key layers testability, by leveraging basic modularization techniques. Techniques discussed in this article, are applicable to other aspects of the nodejs application. There are additional complimentary materials in the “Testing nodejs applications” book.

References

tags: #snippets #code #annotations #question #discuss