redis &mdash; Simple Engineering https://getsimple.works/tag:redis Tue, 28 Apr 2026 14:53:47 +0000 How to modularize redis clients https://getsimple.works/how-to-modularize-redis-clients?pk_campaign=rss-feed <![CDATA[Modularization of redis for testability To take advantage of multicore systems, nodejs -- being a single-threaded JavaScript runtime -- spins up multiple processes to guarantee parallel processing capabilities. That works well until inter-process communication becomes an issue. That is where key-stores such as redis come into the picture, to solve the inter-process communication problem while enhancing real-time experience. This article is about showcasing how to achieve leverage modular design to provide testable and scalable code. In this article we will talk about: How to modularize redis clients for reusability How to modularize redis clients for testability How to modularize redis clients for composability The need to have a redis powered pub/sub Techniques to modularize redis powered pub/sub The need to coupling WebSocket with redis pub/subsystem How to modularize WebSocket redis communications How to modularize redis configuration Even though this blog post was designed to offer complementary materials to those who bought my Testing nodejs Applications book, the content can help any software developer to tuneup working environment. You use this link to buy the book. Testing nodejs Applications Book Cover Show me the code Introducing extra components makes it hard to test a system in isolation. This example highlights some of the moving parts we will be discussing in this article: //creating the Server -- alternative #1 var app = express(); var server = Server(app); //creating the Server -- alternative #2 var express = require('express'), app = express(), server = require('http').createServer(app); //Initialization of WebSocket Server + Redis Pub/Sub var wss = require("socket.io")(server), redis = require('redis'), rhost = process.env.REDISHOST, rport = process.env.REDISPORT, pub = redis.createClient(rport, rhost), sub = redis.createClient(rport, rhost); //HTTP session middleware thing function middleware(req, res, next){ ... next(); } //exchanging session values wss.use(function(socket, next){ middleware(socket.request, socket.request.res, next); }); //express uses middleware for session management app.use(middleware); //somewhere wss.sockets.on("connection", function(socket) { //socket.request.session //Now it's available from Socket.IO sockets too! Win! socket.on('message', (event) = { var payload = JSON.parse(event.payload || event), user = socket.handshake.user || false; //except when coming from pub pub.publish(payload.conversation, payload)); }); //redis listener sub.on('message', function(channel, event) { var payload = JSON.parse(event.payload || event), user = socket.handshake.user || false; wss. sockets. in(payload.conversation). emit('message', payload); }); Example: What can possibly go wrong? Having redis.createClient() everywhere, makes it hard to mock creation/deletion of redis instances(pub/sub) is out of control One way is to create One instance (preferably while loading top-level module), and inject that instance into dependent modules - Managing modularity and redis connections in nodejs. - The other way: node module loader caches loaded modules. Which provides a singleton by default. The need to have a redis powered pub/sub JavaScript, and nodejs in particular, is a single-threaded language -- but has other ways to provide parallel computing. It is possible to spin up any number of processes depending on application needs. The process to process communication becomes an issue, and when one process mutates the state of a shared object, for instance, any other process on the same server would have to be informed about the update. Unfortunately, that is not feasible. pub/sub mechanisms that redis brings to the table, make it possible to solve problems similar to this one. How to modularize redis clients for testability pub/sub implementations make the code intimidating, especially when the time comes to test. We assume that existing code has little to no test, and most importantly, not modularized. Or well tested, and well modularized, but the addition of real-time handling adds a need to leverage pub/sub to provide near real-time experience. The first and easy thing to do in such a scenario is to break code blocks into smaller chunks that we can test in isolation. In essence, the pub and sub are both redis clients, that have to be created independently so that they run in two separate contexts and processes. We may be tempted to use pub and sub-objects as the same client, that would be detrimental and create race conditions from the get-go. Delegating pub/sub-creation to a utility function makes it possible to mock the clients. The utility function should accept injected redis. It is possible to go the extra mile and delegate redis instance initialization in its own factory. That way, it becomes even easier to mock the redis instance itself. Past these steps, other refactoring techniques can take over. // hard to mock when located in [root]/index.js var redis = require('redis'), rhost = process.env.REDISHOST, rport = process.env.REDISPORT, pub = redis.createClient(rport, rhost), sub = redis.createClient(rport, rhost); // Easy to mock with introduction of createClient factory // in /lib/util/redis.js|redis-helper.js module.exports = function(redis){ return redis.createClient(port, host); } How to modularize redis clients for reusability The example provided in this article scratches the surface on what can be achieved when integrating redis into a project. What would be the chain of events if, for some reason, redis server goes down. Would that affect the overall health and usability of the whole application? If the answer is yes, or not sure, that gives a pretty good indication of the need to isolate usage of redis, and make sure its modularity is sound and failure-proof. Modularization of the redis can be seen from two angles: to publish a set of events to the shared store, subscribing to the shared store for updates on events of our interest. By making the redis integration modular, we also have to think about making sure redis server downtime/failure, does not translate into a cascading effect that may bring the application down. //in app|server|index.js var client = require("redis").createClient(); var app = require("./lib")(client);//<- Injection //injecting redis into a route var createClient = require('./lib/util/redis'); module.exports = function(redis){ return function(req, res, next){ var redisClient = createClient(redis); return res.status(200).json({message: 'About Issues'}); }; }; //usage var getMessage = require('./')(redis); How to modularize redis clients for composability In the previous two sections, we have seen how pub/sub enhanced by a redis server brings near real-time experience to the program. The problem we faced in both sections, is that redis is tightly coupled to all modules, even those that do not need to use it. Composability becomes an issue when we need to avoid having a single point of failure in the program, as well as providing a test coverage deep enough to prevent common use cases of failures. // in /lib/util/redis const redis = require('redis'); module.exports = function(options){ return options ? {} : redis; } The above small factory may look a little weird, but it makes it possible to offset initialization to a third-party service and becomes possible to mock when testing. Techniques to modularize redis powered pub/sub The need to modularize the pub/sub code has been discussed in previous segments. The issue we still have at this time is at pub/sub handler level. As we may have noticed already, testing pub/sub handlers is challenging especially when not having an up and running redis instance. Modularizing that two kinds of handlers provide an opportunity to test pub/sub handlers in isolation. It also makes it possible to share the handlers with other systems that may need exactly the same kind of behavior. The need to lose coupling WebSocket with redis pub/sub system One example of decoupling pub/sub from redis and make its handlers re-usable, can be seen when the WebSocket server has to leverage socket server events. For example, on a new message read on the socket, the socket server should notify other processes that there is in fact a new message on the socket. The pub is the right place to post this kind of notification. On a new message posted in the store, the WebSocket server may need to respond to a particular user. and so forth. How to modularize WebSocket redis communications There is a use case where an infinite same message can be ping-pong-ed between pub and sub. To make sure such a thing doesn't happen, a communication protocol should be initialized. For example, when a message is published to the store by a WebSocket and the message is destined to all participating processes, a corresponding listener should read from the store and forward the message to all participating sockets, In such a way a socket that receives the message simply publishes it but does not answer to the sender right away. Subscribed sockets, can then read from the store, and forward the message to the right receiver. There is an entire blog dedicated to modularizing nodejs WebSockets here How modularize redis configuration The need to configure a server comes not only for redis server but also for any other server or service. In this particular instance, we will see how we can include redis configuration into an independent module that can then be used with the rest of the configurations. //from the example above const redis = require("redis"); const port = process.ENV.REDISPORT || "6379"; const host = process.ENV.REDISHOST || "127.0.0.1"; module.exports = redis.createClient(port, host); //abstracting configurations in lib/configs module.exports = Object.freeze({ redis: { port: process.ENV.REDISPORT || "6379", port: process.ENV.REDISHOST || "127.0.0.1" } }); //using an abstracted configurations const configs = require('./lib/configs'); module.exports = redis.createClient( configs.redis.port, configs.redis.host ); This strategy to rethink, application structure has been found here Conclusion Modularization is a key strategy in crafting re-usable composable software. Modularization brings not only elegance but makes copy/paste detectors happy, and at the same time improves both performance and testability. In this article, we revisited how to aggregate WebSocket code into composable and testable modules. The need to group related tasks into modules involves the ability to add support of Pub/Sub on demand and using various solutions as project requirements evolve. There are additional complimentary materials in the "Testing nodejs applications" book. References + Reading List Testing nodejs Applications book redis pubsub library ~ A Github cloned library Building a Chat Server with node and redis - tests ~ Matthew Daly Blog High Volume, low latency difficulties nodejs/pubsub/redis ~ StackOverflow Question examples using redis-store with socket.io ~ StackOverflow Question Using redis as PubSub over socket.io ~ StackOverflow Question socket.io-redis ~ Github Library socket.io within concurrent processes ~ Tolle Iv Blog Modularize Your Chat App(or How to Write a nodejs Express App in More Than One File) ~ Philosophy is Thinking Medium bacon.js + nodejs + mongodb: Functional Reactive Programming on the Server ~ Carbon Five Blog NODE.JS DATABASES: USING REDIS FOR FUN AND PROFIT tags: #snippets #redis #nodejs #modularization ]]> Modularization of redis for testability

To take advantage of multicore systems, nodejs — being a single-threaded JavaScript runtime — spins up multiple processes to guarantee parallel processing capabilities. That works well until inter-process communication becomes an issue.

That is where key-stores such as redis come into the picture, to solve the inter-process communication problem while enhancing real-time experience.

This article is about showcasing how to achieve leverage modular design to provide testable and scalable code.

In this article we will talk about:

  • How to modularize redis clients for reusability
  • How to modularize redis clients for testability
  • How to modularize redis clients for composability
  • The need to have a redis powered pub/sub
  • Techniques to modularize redis powered pub/sub
  • The need to coupling WebSocket with redis pub/subsystem
  • How to modularize WebSocket redis communications
  • How to modularize redis configuration

Even though this blog post was designed to offer complementary materials to those who bought my Testing nodejs Applications book, the content can help any software developer to tuneup working environment. You use this link to buy the book. Testing nodejs Applications Book Cover

Show me the code

Introducing extra components makes it hard to test a system in isolation. This example highlights some of the moving parts we will be discussing in this article:

//creating the Server -- alternative #1 
var app = express();
var server = Server(app);

//creating the Server -- alternative #2
var express = require('express'),
    app = express(),
    server = require('http').createServer(app);

//Initialization of WebSocket Server + Redis Pub/Sub    
var wss = require("socket.io")(server),
	redis = require('redis'), 
	rhost = process.env.REDIS_HOST,
	rport = process.env.REDIS_PORT,
	pub = redis.createClient(rport, rhost), 
  sub = redis.createClient(rport, rhost);
  
//HTTP session middleware thing
function middleware(req, res, next){
 ...
 next();
}

//exchanging session values 
wss.use(function(socket, next){
 	middleware(socket.request, socket.request.res, next);
});

//express uses middleware for session management
app.use(middleware);
    
//somewhere
wss.sockets.on("connection", function(socket) {
 
 //socket.request.session 
 //Now it's available from Socket.IO sockets too! Win!
 socket.on('message', (event) => {
	 var payload = JSON.parse(event.payload || event),
	 	user = socket.handshake.user || false;
	 
	 //except when coming from pub  			
	 pub.publish(payload.conversation, payload)); 
 });

 //redis listener
 sub.on('message', function(channel, event) {
	var payload = JSON.parse(event.payload || event),
		user = socket.handshake.user || false;
    wss.
      sockets.
      in(payload.conversation).
      emit('message', payload);
 });

Example:

What can possibly go wrong?

  • Having redis.createClient() everywhere, makes it hard to mock
  • creation/deletion of redis instances(pub/sub) is out of control

One way is to create One instance (preferably while loading top-level module), and inject that instance into dependent modules – Managing modularity and redis connections in nodejs. – The other way: node module loader caches loaded modules. Which provides a singleton by default.

The need to have a redis powered pub/sub

JavaScript, and nodejs in particular, is a single-threaded language — but has other ways to provide parallel computing.

It is possible to spin up any number of processes depending on application needs. The process to process communication becomes an issue, and when one process mutates the state of a shared object, for instance, any other process on the same server would have to be informed about the update.

Unfortunately, that is not feasible. pub/sub mechanisms that redis brings to the table, make it possible to solve problems similar to this one.

How to modularize redis clients for testability

pub/sub implementations make the code intimidating, especially when the time comes to test.

We assume that existing code has little to no test, and most importantly, not modularized. Or well tested, and well modularized, but the addition of real-time handling adds a need to leverage pub/sub to provide near real-time experience.

The first and easy thing to do in such a scenario is to break code blocks into smaller chunks that we can test in isolation.

  • In essence, the pub and sub are both redis clients, that have to be created independently so that they run in two separate contexts and processes. We may be tempted to use pub and sub-objects as the same client, that would be detrimental and create race conditions from the get-go.
  • Delegating pub/sub-creation to a utility function makes it possible to mock the clients.
  • The utility function should accept injected redis. It is possible to go the extra mile and delegate redis instance initialization in its own factory. That way, it becomes even easier to mock the redis instance itself.

Past these steps, other refactoring techniques can take over.

// hard to mock when located in [root]/index.js  
var redis = require('redis'), 
	rhost = process.env.REDIS_HOST,
	rport = process.env.REDIS_PORT,
	pub = redis.createClient(rport, rhost), 
  sub = redis.createClient(rport, rhost);

// Easy to mock with introduction of createClient factory
// in /lib/util/redis.js|redis-helper.js
module.exports = function(redis){
    return redis.createClient(port, host);
}

How to modularize redis clients for reusability

The example provided in this article scratches the surface on what can be achieved when integrating redis into a project.

What would be the chain of events if, for some reason, redis server goes down. Would that affect the overall health and usability of the whole application?

If the answer is yes, or not sure, that gives a pretty good indication of the need to isolate usage of redis, and make sure its modularity is sound and failure-proof.

Modularization of the redis can be seen from two angles: to publish a set of events to the shared store, subscribing to the shared store for updates on events of our interest.

By making the redis integration modular, we also have to think about making sure redis server downtime/failure, does not translate into a cascading effect that may bring the application down.

//in app|server|index.js   
var client = require("redis").createClient(); 
var app = require("./lib")(client);//<- Injection

//injecting redis into a route
var createClient = require('./lib/util/redis');
module.exports = function(redis){
  return function(req, res, next){
    var redisClient = createClient(redis);
    return res.status(200).json({message: 'About Issues'});
  };
};

//usage
var getMessage = require('./')(redis);

How to modularize redis clients for composability

In the previous two sections, we have seen how pub/sub enhanced by a redis server brings near real-time experience to the program.

The problem we faced in both sections, is that redis is tightly coupled to all modules, even those that do not need to use it.

Composability becomes an issue when we need to avoid having a single point of failure in the program, as well as providing a test coverage deep enough to prevent common use cases of failures.

// in /lib/util/redis
const redis = require('redis');
module.exports = function(options){
  return options ?  {} : redis;
}

The above small factory may look a little weird, but it makes it possible to offset initialization to a third-party service and becomes possible to mock when testing.

Techniques to modularize redis powered pub/sub

The need to modularize the pub/sub code has been discussed in previous segments.

The issue we still have at this time is at pub/sub handler level. As we may have noticed already, testing pub/sub handlers is challenging especially when not having an up and running redis instance.

Modularizing that two kinds of handlers provide an opportunity to test pub/sub handlers in isolation. It also makes it possible to share the handlers with other systems that may need exactly the same kind of behavior.

The need to lose coupling WebSocket with redis pub/sub system

One example of decoupling pub/subfrom redis and make its handlers re-usable, can be seen when the WebSocket server has to leverage socket server events.

For example, on a new message read on the socket, the socket server should notify other processes that there is in fact a new message on the socket.

The pub is the right place to post this kind of notification. On a new message posted in the store, the WebSocket server may need to respond to a particular user. and so forth.

How to modularize WebSocket redis communications

There is a use case where an infinite same message can be ping-pong-ed between pub and sub.

To make sure such a thing doesn't happen, a communication protocol should be initialized. For example, when a message is published to the store by a WebSocket and the message is destined to all participating processes, a corresponding listener should read from the store and forward the message to all participating sockets, In such a way a socket that receives the message simply publishes it but does not answer to the sender right away.

Subscribed sockets, can then read from the store, and forward the message to the right receiver.

There is an entire blog dedicated to modularizing nodejs WebSockets here

How modularize redis configuration

The need to configure a server comes not only for redis server but also for any other server or service.

In this particular instance, we will see how we can include redis configuration into an independent module that can then be used with the rest of the configurations.

//from the example above 
const redis = require("redis"); 
const port = process.ENV.REDIS_PORT || "6379";
const host = process.ENV.REDIS_HOST || "127.0.0.1";
module.exports = redis.createClient(port, host);

//abstracting configurations in lib/configs
module.exports = Object.freeze({ 
  redis: {
    port: process.ENV.REDIS_PORT || "6379",
    port: process.ENV.REDIS_HOST || "127.0.0.1"
  }
});

//using an abstracted configurations
const configs = require('./lib/configs');
module.exports = redis.createClient(
  configs.redis.port, 
  configs.redis.host
);

This strategy to rethink, application structure has been found here

Conclusion

Modularization is a key strategy in crafting re-usable composable software. Modularization brings not only elegance but makes copy/paste detectors happy, and at the same time improves both performance and testability.

In this article, we revisited how to aggregate WebSocket code into composable and testable modules. The need to group related tasks into modules involves the ability to add support of Pub/Sub on demand and using various solutions as project requirements evolve. There are additional complimentary materials in the “Testing nodejs applications” book.

References + Reading List

tags: #snippets #redis #nodejs #modularization

]]> https://getsimple.works/how-to-modularize-redis-clients Thu, 17 Jun 2021 05:32:50 +0000 Easy nodejs deployment https://getsimple.works/easy-nodejs-deployment?pk_campaign=rss-feed <![CDATA[This article is going to explore how to deploy a nodejs application on a traditional linux server -- in a non-cloud environment. Even though the use case is Ubuntu, any Linux distro or mac would work perfectly fine. For information on deploying on non-traditional servers, read: "Deploying nodejs applications". For zero-downtime knowledge, read "How to achieve zero downtime deployment with nodejs" In this article we will talk about: Preparing nodejs deployable releases Configuring nodejs deployment environment Deploying nodejs application on bare metal Ubuntu server Switching on nodejs application to be available to the world ~ adding nginx for the reverse proxy to make the application available to the world post-deployment support -- production support Even though this blog post was designed to offer complementary materials to those who bought my Testing nodejs Applications book book, the content can help any software developer to level up their knowledge. You may use this link to buy the book. Testing nodejs Applications Book Cover Preparing a deployable release There are several angles to look at release and deployment from. There are also several ways to release nodejs code, npm and tar for instance, and that depending on the environment in which the code is designed to run. Amongst environments, server-side, universal, or command line are classic examples. In addition, we have to take a look from a dependency management perspective. Managing dependencies at deployment time has two vectors to take into account: whether the deployment happens on online or offline. For the code to be prepared for release, it has to be packaged. Two methods of packaging nodejs software, amongst other things, are managed packaging and bundling. More on this is discussed here As a plus, versioning should be taken into consideration when preparing a deployable release. The versioning that is a little common circulation is SemVer. Configuring deployment environment Before we dive into deployment challenges, let's look at key software and configuration requirements. As usual, first-time work can be hard to do. But the yield should be then predictable, flexible to improvement, and capable of being built-upon for future deployments. For context, the deployment environment we are talking about in this section is the production environment. Two key configurations are an nginx reverse proxy server and nodejs. But, "Why coupling nodejs server to an nginx reverse proxy server"? The answer to this question is two folds. First, both nodejs and nginx are single-threaded non-blocking reactive systems. Second, the wide adoption of these two tools by the developer community makes it an easy choice, from both influence and availability of collective knowledge the developer community shares via forums/blogs and popular QA sites. How to install nginx server ~ there is an article dedicated to this. How to configure nginx as a nodejs application proxy server ~ there is an article dedicated to this. Additional tools to install and configure may include: mongod database server, redis server, monit for monitoring, upstart for enhancing the init system. There is a need to better understand the tools required to run nodejs application. It is also our responsibility as developers to have a basic understanding of each tool and the roles it plays in our project, in order to figure out how to configure each tool. Download source code Starting from the utility perspective, there is quite a collection of tools that are required to run on the server, alongside our nodejs application. Such software needs to be installed, and updated ~ for patch releases, and upgraded ~ to new major versions to keep the system secure and capable(bug-free/enhanced with new features). From the packaging perspective, both supporting tools and nodejs applications adhere to a packaging strategy that makes it easy to deploy. When packaging is indeed a bundle, wget/curl can be used to download binaries. When dealing with discoverable packages, npm/yarn/brew can also be used to download our application and its dependencies. Both operation yield same outcomes, which is un-packaging, configuration and installation. To deploy versioned nodejs application on bare metal Ubuntu server ~ understanding file system tweaks such as symlink-ing for faster deployments can save time for future deployments. first time on server side $ apt-get update $ apt-get install git updating|upgrading server side code $ apt-get update $ apt-get upgrade $ brew upgrade $ npm upgrade Package download and installs $ /bin/bash -c "$(curl -fsSL https://url.tld/version/install.sh)" $ wget -O - https://url.tld/version/install.sh | bash Discoverable packages $ npm install application@next $ yarn add application@next $ brew install application Example: The command above can be automated via a scheduled task. Both npm and yarn support the installation of applications bundled in a .tar file. See an example of a simple script source. We have to be mindful to clean up download directories, to save disk space. Switching on the application It sounds repetitive, but running npm start does not guarantee the application to be visible outside the metal server box the application is hosted on. This magic belongs to the nginx reverse proxy we were referring to in earlier paragraphs. A typical nodejs application needs to start one or more of the following services, each time to reboot the application. symlinking new version to default application path $ ln -sfn /var/www/new/version/appname /var/www/appname $ service nginx restart #nginx|apache server $ service redis restart #redis server $ service restart mongod #database server in some cases $ service appname restart #application itself Example: PS: Above services are managed with uptime Adding nginx reverse proxy makes the application available to the outside world. Switching on/off the application can be summarized in one command: service nginx stop. Likewise, to switch off and back on can be issued in one command: service nginx restart. Post-deployment support Asynchronous timely tasks can be used to resolve a wide range of issues. Background tasks such as fetching updates from third-party data providers, system health check, and notifications, automated software updates, database cleaning, cache busting, scheduled expensive/CPU intensive batch processing jobs just to name a few. It is possible to leverage existing asynchronous timely OS-provided tasks processing infrastructure to achieve any of the named use cases, as it is true for third-party tools to do exactly the same job. Rule of thumb The following is a mental model that can be applied to the common use cases of releases. It may be basic for DevOps professionals, but useful enough for developers doing some operations work as well. Prepare deployable releases Update and install binaries ~ using apt, brew etc. Download binaries ~ using git,wget, curl or brew symlink directories(/log, /config, /app) Restart servers and services ~ redis, nginx, mongodb and app When something goes bad ~ walk two steps back. That is our rollback strategy. This model can be refined, to make most of these tasks repeatable and automated, deployments included. Conclusion In this article, we revisited quick easy, and most basic nodejs deployment strategies. We also revisited how to expose the deployed applications to the world using nginx as a reverse proxy server. There are additional complimentary materials in the "Testing nodejs applications" book. References Testing nodejs Applications book How to set up CI/CD Pipeline for a nodejs app with Jenkins ~ Moshe Ezderman - Medium "Build your dependencies into your deployable packages" #snippets #code #annotations #question #discuss]]> This article is going to explore how to deploy a nodejs application on a traditional linux server — in a non-cloud environment. Even though the use case is Ubuntu, any Linux distro or mac would work perfectly fine.

For information on deploying on non-traditional servers, read: “Deploying nodejs applications”. For zero-downtime knowledge, read “How to achieve zero downtime deployment with nodejs“

In this article we will talk about:

  • Preparing nodejs deployable releases
  • Configuring nodejs deployment environment
  • Deploying nodejs application on bare metal Ubuntu server
  • Switching on nodejs application to be available to the world ~ adding nginx for the reverse proxy to make the application available to the world
  • post-deployment support — production support

Even though this blog post was designed to offer complementary materials to those who bought my Testing nodejs Applications book book, the content can help any software developer to level up their knowledge. You may use this link to buy the book. Testing nodejs Applications Book Cover

Preparing a deployable release

There are several angles to look at release and deployment from. There are also several ways to release nodejs code, npm and tar for instance, and that depending on the environment in which the code is designed to run. Amongst environments, server-side, universal, or command line are classic examples.

In addition, we have to take a look from a dependency management perspective. Managing dependencies at deployment time has two vectors to take into account: whether the deployment happens on online or offline.

For the code to be prepared for release, it has to be packaged. Two methods of packaging nodejs software, amongst other things, are managed packaging and bundling. More on this is discussed here

As a plus, versioning should be taken into consideration when preparing a deployable release. The versioning that is a little common circulation is SemVer.

Configuring deployment environment

Before we dive into deployment challenges, let's look at key software and configuration requirements.

As usual, first-time work can be hard to do. But the yield should be then predictable, flexible to improvement, and capable of being built-upon for future deployments. For context, the deployment environment we are talking about in this section is the production environment.

Two key configurations are an nginx reverse proxy server and nodejs. But, “Why coupling nodejs server to an nginx reverse proxy server”? The answer to this question is two folds. First, both nodejs and nginx are single-threaded non-blocking reactive systems. Second, the wide adoption of these two tools by the developer community makes it an easy choice, from both influence and availability of collective knowledge the developer community shares via forums/blogs and popular QA sites.

How to install nginx server ~ [there is an article dedicated to this](). How to configure nginx as a nodejs application proxy server ~ there is an article dedicated to this.

Additional tools to install and configure may include: mongod database server, redis server, monit for monitoring, upstart for enhancing the init system.

There is a need to better understand the tools required to run nodejs application. It is also our responsibility as developers to have a basic understanding of each tool and the roles it plays in our project, in order to figure out how to configure each tool.

Download source code

Starting from the utility perspective, there is quite a collection of tools that are required to run on the server, alongside our nodejs application. Such software needs to be installed, and updated ~ for patch releases, and upgraded ~ to new major versions to keep the system secure and capable(bug-free/enhanced with new features).

From the packaging perspective, both supporting tools and nodejs applications adhere to a packaging strategy that makes it easy to deploy. When packaging is indeed a bundle, wget/curl can be used to download binaries. When dealing with discoverable packages, npm/yarn/brew can also be used to download our application and its dependencies. Both operation yield same outcomes, which is un-packaging, configuration and installation.

To deploy versioned nodejs application on bare metal Ubuntu server ~ understanding file system tweaks such as symlink-ing for faster deployments can save time for future deployments.

#first time on server side  
$ apt-get update
$ apt-get install git

#updating|upgrading server side code
$ apt-get update
$ apt-get upgrade
$ brew upgrade 
$ npm upgrade 

# Package download and installs 
$ /bin/bash -c "$(curl -fsSL https://url.tld/version/install.sh)"
$ wget -O - https://url.tld/version/install.sh | bash

# Discoverable packages 
$ npm install application@next 
$ yarn add application@next 
$ brew install application

_Example: _

The command above can be automated via a scheduled task. Both npm and yarn support the installation of applications bundled in a .tar file. See an example of a simple script source. We have to be mindful to clean up download directories, to save disk space.

Switching on the application

It sounds repetitive, but running npm start does not guarantee the application to be visible outside the metal server box the application is hosted on. This magic belongs to the nginx reverse proxy we were referring to in earlier paragraphs.

A typical nodejs application needs to start one or more of the following services, each time to reboot the application.

# symlinking new version to default application path
$ ln -sfn /var/www/new/version/appname /var/www/appname 

$ service nginx restart #nginx|apache server
$ service redis restart #redis server
$ service restart mongod #database server in some cases
$ service appname restart #application itself

Example:

PS: Above services are managed with uptime

Adding nginx reverse proxy makes the application available to the outside world. Switching on/off the application can be summarized in one command: service nginx stop. Likewise, to switch off and back on can be issued in one command: service nginx restart.

Post-deployment support

Asynchronous timely tasks can be used to resolve a wide range of issues. Background tasks such as fetching updates from third-party data providers, system health check, and notifications, automated software updates, database cleaning, cache busting, scheduled expensive/CPU intensive batch processing jobs just to name a few.

It is possible to leverage existing asynchronous timely OS-provided tasks processing infrastructure to achieve any of the named use cases, as it is true for third-party tools to do exactly the same job.

Rule of thumb

The following is a mental model that can be applied to the common use cases of releases. It may be basic for DevOps professionals, but useful enough for developers doing some operations work as well.

  • Prepare deployable releases
  • Update and install binaries ~ using apt, brew etc.
  • Download binaries ~ using git,wget, curl or brew
  • symlink directories(/log, /config, /app)
  • Restart servers and services ~ redis, nginx, mongodb and app
  • When something goes bad ~ walk two steps back. That is our rollback strategy.

This model can be refined, to make most of these tasks repeatable and automated, deployments included.

Conclusion

In this article, we revisited quick easy, and most basic nodejs deployment strategies. We also revisited how to expose the deployed applications to the world using nginx as a reverse proxy server. There are additional complimentary materials in the “Testing nodejs applications” book.

References

#snippets #code #annotations #question #discuss

]]> https://getsimple.works/easy-nodejs-deployment Thu, 17 Jun 2021 05:08:19 +0000 Integration of redis in nodejs applications https://getsimple.works/integration-of-redis-in-nodejs-applications?pk_campaign=rss-feed <![CDATA[The reactive aspect of nodejs applications is synonymous with the nodejs runtime itself. Even though the real-time aspect may be attributed to WebSocket implementation, the realtime reactive aspect of nodejs applications heavily rely on pub/sub mechanisms -- most of the time backed by datastore engines like redis. This article explores how to integrate redis datastore into a nodejs application. In this article we will talk about: redis support with and without expressjs redis support with and without WebSocket push mechanism Alternatives to redis in nodejs world and beyond Even though this blog post was designed to offer complementary materials to those who bought my Testing nodejs Applications book, the content can help any software developer to tuneup working environment. You use this link to buy the book. Testing nodejs Applications Book Cover Show me the code The following code sample showcase how nodejs/redis integration can be achieved. It demonstrates that it is still possible to share sessions via a middleware between socket.io and expressjs. var app = express(); var server = Server(app); var sio = require("socket.io")(server), redis = require('redis'), rhost = process.env.REDISHOST, rport = process.env.REDISPORT, pub = redis.createClient(rport, rhost), sub = redis.createClient(rport, rhost); function middleware(req, res, next){ //session initialization thing next(); } //socket.io/expressjs session sharing middleware sio.use(function(socket, next){ middleware(socket.request, socket.request.res, next); }); //express uses middleware for session management app.use(middleware); //somewhere sio.sockets.on("connection", function(socket) { //socket.request.session //Now it's available from socket.io sockets too! Win! socket.on('message', (event) = { var payload = JSON.parse(event.payload || event), user = socket.handshake.user || false; //except when coming from pub pub.publish(payload.conversation, payload)); }); //redis listener sub.on('message', function(channel, event) { var payload = JSON.parse(event.payload || event), user = socket.handshake.user || false; sio.sockets.in(payload.conversation).emit('message', payload); }); Example: excerpt source: StackOverflow What can possibly go wrong? When trying to figure out how to approach redis datastore integration into nodejs application for inter-process communication and real-time feature, the following points may be a challenge: How to decouple the WebSocket events from the redis specific (pub/sub) events. We should be able to decouple, but still provide an environment where interoperability is possible at any time. How to make integration modular, testable, and overall friendly to the rest of the application ecosystem When testing this implementation, we should expect the additional challenge to emerge: The redis client instances (pub/sub) are created as soon as the library loads, and a redis server should be up and running by that time. The issue is when testing the application, there should be no server or any other system dependency hindering the application from being tested. getting rid of redis server with a drop-in-replacement, or stubs/mocks, is more of a dream than reality ~ hard but feasible. There is additional information in mocking and stubbing redis data store in "How to Mock redis datastore" article. Conclusion In this article, we revisited how to enhance nodejs application with redis based pub/sub mechanism, critical to having a reactive real-time experience. The use of WebSocket and Pub/Sub powered by a key/value data store was especially the main focus of this article. There are additional complimentary materials in the "Testing nodejs applications" book. References Testing nodejs Applications book Sharing session between expressjs and socket.io ~ StackOverflow Question examples using redis with socket.io ~ StackOverflow Question Using redis as PubSub over socket.io ~ StackOVerflow Question socket.io - redis redis pubsub library ~ Github Cloned Library Building a Chat Server with node and redis - tests ~ Matthew Daly Blog High Volume, low latency difficulties nodejs/pubsub/redis ~ StackOverflow Question #snippets #nodejs #integration #redis]]> The reactive aspect of nodejs applications is synonymous with the nodejs runtime itself. Even though the real-time aspect may be attributed to WebSocket implementation, the realtime reactive aspect of nodejs applications heavily rely on pub/sub mechanisms — most of the time backed by datastore engines like redis. This article explores how to integrate redis datastore into a nodejs application.

In this article we will talk about:

  • redis support with and without expressjs
  • redis support with and without WebSocket push mechanism
  • Alternatives to redis in nodejs world and beyond

Even though this blog post was designed to offer complementary materials to those who bought my Testing nodejs Applications book, the content can help any software developer to tuneup working environment. You use this link to buy the book. Testing nodejs Applications Book Cover

Show me the code

The following code sample showcase how nodejs/redis integration can be achieved. It demonstrates that it is still possible to share sessions via a middleware between socket.io and expressjs.

var app = express();
var server = Server(app);
var sio = require("socket.io")(server),
	redis = require('redis'), 
	rhost = process.env.REDIS_HOST,
	rport = process.env.REDIS_PORT,
	pub = redis.createClient(rport, rhost), 
	sub = redis.createClient(rport, rhost);


function middleware(req, res, next){
 //session initialization thing
 next();
}

//socket.io/expressjs session sharing middleware
sio.use(function(socket, next){
 	middleware(socket.request, socket.request.res, next);
});

//express uses middleware for session management
app.use(middleware);
    
//somewhere
sio.sockets.on("connection", function(socket) {
 
 //socket.request.session 
 //Now it's available from `socket.io` sockets too! Win!
 socket.on('message', (event) => {
	 var payload = JSON.parse(event.payload || event),
	 	user = socket.handshake.user || false;
	 
	 //except when coming from pub  			
	 pub.publish(payload.conversation, payload)); 
 });

 //redis listener
 sub.on('message', function(channel, event) {
	var payload = JSON.parse(event.payload || event),
		user = socket.handshake.user || false;
	sio.sockets.in(payload.conversation).emit('message', payload);
 });

Example: excerpt source: StackOverflow

What can possibly go wrong?

When trying to figure out how to approach redis datastore integration into nodejs application for inter-process communication and real-time feature, the following points may be a challenge:

  • How to decouple the WebSocket events from the redis specific (pub/sub) events. We should be able to decouple, but still provide an environment where interoperability is possible at any time.
  • How to make integration modular, testable, and overall friendly to the rest of the application ecosystem

When testing this implementation, we should expect the additional challenge to emerge:

  • The redis client instances (pub/sub) are created as soon as the library loads, and a redis server should be up and running by that time. The issue is when testing the application, there should be no server or any other system dependency hindering the application from being tested.
  • getting rid of redis server with a drop-in-replacement, or stubs/mocks, is more of a dream than reality ~ hard but feasible.

There is additional information in mocking and stubbing redis data store in “How to Mock redis datastore” article.

Conclusion

In this article, we revisited how to enhance nodejs application with redis based pub/sub mechanism, critical to having a reactive real-time experience. The use of WebSocket and Pub/Sub powered by a key/value data store was especially the main focus of this article. There are additional complimentary materials in the “Testing nodejs applications” book.

References

#snippets #nodejs #integration #redis

]]> https://getsimple.works/integration-of-redis-in-nodejs-applications Thu, 17 Jun 2021 04:46:06 +0000