How to modularize redis clients

Modularization of redis for testability

To take advantage of multicore systems, nodejs — being a single-threaded JavaScript runtime — spins up multiple processes to guarantee parallel processing capabilities. That works well until inter-process communication becomes an issue.

That is where key-stores such as redis come into the picture, to solve the inter-process communication problem while enhancing real-time experience.

This article is about showcasing how to achieve leverage modular design to provide testable and scalable code.

In this article we will talk about:

Even though this blog post was designed to offer complementary materials to those who bought my Testing nodejs Applications book, the content can help any software developer to tuneup working environment. You use this link to buy the book. Testing nodejs Applications Book Cover

Show me the code

Introducing extra components makes it hard to test a system in isolation. This example highlights some of the moving parts we will be discussing in this article:

//creating the Server -- alternative #1 
var app = express();
var server = Server(app);

//creating the Server -- alternative #2
var express = require('express'),
    app = express(),
    server = require('http').createServer(app);

//Initialization of WebSocket Server + Redis Pub/Sub    
var wss = require("socket.io")(server),
	redis = require('redis'), 
	rhost = process.env.REDIS_HOST,
	rport = process.env.REDIS_PORT,
	pub = redis.createClient(rport, rhost), 
  sub = redis.createClient(rport, rhost);
  
//HTTP session middleware thing
function middleware(req, res, next){
 ...
 next();
}

//exchanging session values 
wss.use(function(socket, next){
 	middleware(socket.request, socket.request.res, next);
});

//express uses middleware for session management
app.use(middleware);
    
//somewhere
wss.sockets.on("connection", function(socket) {
 
 //socket.request.session 
 //Now it's available from Socket.IO sockets too! Win!
 socket.on('message', (event) => {
	 var payload = JSON.parse(event.payload || event),
	 	user = socket.handshake.user || false;
	 
	 //except when coming from pub  			
	 pub.publish(payload.conversation, payload)); 
 });

 //redis listener
 sub.on('message', function(channel, event) {
	var payload = JSON.parse(event.payload || event),
		user = socket.handshake.user || false;
    wss.
      sockets.
      in(payload.conversation).
      emit('message', payload);
 });

Example:

What can possibly go wrong?

One way is to create One instance (preferably while loading top-level module), and inject that instance into dependent modules – Managing modularity and redis connections in nodejs. – The other way: node module loader caches loaded modules. Which provides a singleton by default.

The need to have a redis powered pub/sub

JavaScript, and nodejs in particular, is a single-threaded language — but has other ways to provide parallel computing.

It is possible to spin up any number of processes depending on application needs. The process to process communication becomes an issue, and when one process mutates the state of a shared object, for instance, any other process on the same server would have to be informed about the update.

Unfortunately, that is not feasible. pub/sub mechanisms that redis brings to the table, make it possible to solve problems similar to this one.

How to modularize redis clients for testability

pub/sub implementations make the code intimidating, especially when the time comes to test.

We assume that existing code has little to no test, and most importantly, not modularized. Or well tested, and well modularized, but the addition of real-time handling adds a need to leverage pub/sub to provide near real-time experience.

The first and easy thing to do in such a scenario is to break code blocks into smaller chunks that we can test in isolation.

Past these steps, other refactoring techniques can take over.

// hard to mock when located in [root]/index.js  
var redis = require('redis'), 
	rhost = process.env.REDIS_HOST,
	rport = process.env.REDIS_PORT,
	pub = redis.createClient(rport, rhost), 
  sub = redis.createClient(rport, rhost);

// Easy to mock with introduction of createClient factory
// in /lib/util/redis.js|redis-helper.js
module.exports = function(redis){
    return redis.createClient(port, host);
}

How to modularize redis clients for reusability

The example provided in this article scratches the surface on what can be achieved when integrating redis into a project.

What would be the chain of events if, for some reason, redis server goes down. Would that affect the overall health and usability of the whole application?

If the answer is yes, or not sure, that gives a pretty good indication of the need to isolate usage of redis, and make sure its modularity is sound and failure-proof.

Modularization of the redis can be seen from two angles: to publish a set of events to the shared store, subscribing to the shared store for updates on events of our interest.

By making the redis integration modular, we also have to think about making sure redis server downtime/failure, does not translate into a cascading effect that may bring the application down.

//in app|server|index.js   
var client = require("redis").createClient(); 
var app = require("./lib")(client);//<- Injection

//injecting redis into a route
var createClient = require('./lib/util/redis');
module.exports = function(redis){
  return function(req, res, next){
    var redisClient = createClient(redis);
    return res.status(200).json({message: 'About Issues'});
  };
};

//usage
var getMessage = require('./')(redis);

How to modularize redis clients for composability

In the previous two sections, we have seen how pub/sub enhanced by a redis server brings near real-time experience to the program.

The problem we faced in both sections, is that redis is tightly coupled to all modules, even those that do not need to use it.

Composability becomes an issue when we need to avoid having a single point of failure in the program, as well as providing a test coverage deep enough to prevent common use cases of failures.

// in /lib/util/redis
const redis = require('redis');
module.exports = function(options){
  return options ?  {} : redis;
}

The above small factory may look a little weird, but it makes it possible to offset initialization to a third-party service and becomes possible to mock when testing.

Techniques to modularize redis powered pub/sub

The need to modularize the pub/sub code has been discussed in previous segments.

The issue we still have at this time is at pub/sub handler level. As we may have noticed already, testing pub/sub handlers is challenging especially when not having an up and running redis instance.

Modularizing that two kinds of handlers provide an opportunity to test pub/sub handlers in isolation. It also makes it possible to share the handlers with other systems that may need exactly the same kind of behavior.

The need to lose coupling WebSocket with redis pub/sub system

One example of decoupling pub/subfrom redis and make its handlers re-usable, can be seen when the WebSocket server has to leverage socket server events.

For example, on a new message read on the socket, the socket server should notify other processes that there is in fact a new message on the socket.

The pub is the right place to post this kind of notification. On a new message posted in the store, the WebSocket server may need to respond to a particular user. and so forth.

How to modularize WebSocket redis communications

There is a use case where an infinite same message can be ping-pong-ed between pub and sub.

To make sure such a thing doesn't happen, a communication protocol should be initialized. For example, when a message is published to the store by a WebSocket and the message is destined to all participating processes, a corresponding listener should read from the store and forward the message to all participating sockets, In such a way a socket that receives the message simply publishes it but does not answer to the sender right away.

Subscribed sockets, can then read from the store, and forward the message to the right receiver.

There is an entire blog dedicated to modularizing nodejs WebSockets here

How modularize redis configuration

The need to configure a server comes not only for redis server but also for any other server or service.

In this particular instance, we will see how we can include redis configuration into an independent module that can then be used with the rest of the configurations.

//from the example above 
const redis = require("redis"); 
const port = process.ENV.REDIS_PORT || "6379";
const host = process.ENV.REDIS_HOST || "127.0.0.1";
module.exports = redis.createClient(port, host);

//abstracting configurations in lib/configs
module.exports = Object.freeze({ 
  redis: {
    port: process.ENV.REDIS_PORT || "6379",
    port: process.ENV.REDIS_HOST || "127.0.0.1"
  }
});

//using an abstracted configurations
const configs = require('./lib/configs');
module.exports = redis.createClient(
  configs.redis.port, 
  configs.redis.host
);

This strategy to rethink, application structure has been found here

Conclusion

Modularization is a key strategy in crafting re-usable composable software. Modularization brings not only elegance but makes copy/paste detectors happy, and at the same time improves both performance and testability.

In this article, we revisited how to aggregate WebSocket code into composable and testable modules. The need to group related tasks into modules involves the ability to add support of Pub/Sub on demand and using various solutions as project requirements evolve. There are additional complimentary materials in the “Testing nodejs applications” book.

References + Reading List

tags: #snippets #redis #nodejs #modularization