Simple Engineering

configuration

The configuration is one of the software component layers, and as such, should be testable in isolation like any other component. Modularization of the configuration layer improves its reusability and testability. The question we should be asking is How do we get there, and that is the objective of this article.

The 12 App Factor, a collection of good practices, advocates for “strict separation of configuration from code” and “storing configuration in environment variables”, among other things.

The 12 App Factor challenges the status quo, when it comes to configuration management. The following paragraph taken verbatim from the documentation is a clear illustration of that fact.

“A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.” ~ verbatim text from 12 App Factor ~ config section

In this article we will talk about:

  • Differentiation of configuration layers
  • How to decouple code from configuration
  • How to modularize configuration for testability
  • How to prevent configurations key leaks in public space

Techniques and ideas discussed in this blog, are available in more detail in “Configurations” chapter of the “Testing nodejs Applications” book. You can grab a copy on this link.

Show me the code

const Twitter = require('twitter');

function TwitterClient() {

    this.client = new Twitter({
        consumer_key: `Plain Text Twitter Consumer Key`,
        consumer_secret: `Plain Text Twitter Consumer Secret`,
        access_token_key: `Plain Text Twitter Access Token Key`,
        access_token_secret: `Plain Text Twitter Access Token Secret`
    });

    //accounts such as : @TechCrunch, @Twitter, etc 
    this.track = Array.isArray(accounts) ? accounts.join(',') : accounts;
    //ids: corresponding Twitter Accounts IDs 816653, 783214, etc  
    this.follow = Array.isArray(ids) ? ids.join(',') : ids;
}

/**
 * <code>
 * let stream = new TwitterClient('@twitter', '783214').getStream();
 * stream.on('error', error => handleError(error));
 * stream.on('data', tweet => logTweet(tweet));
 * </code>
 * @name getStream - Returns Usable Stream
 * @returns {Object<TwitterStream>}
 */
TwitterClient.prototype.getStream = function(){
    return this.client.stream('statuses/filter', {track: this.track, follow: this.follow});
};

Example:

What can possibly go wrong?

When trying to figure out how to approach modularizing of configurations, the following points may be a challenge:

  • Being able to share the source code without leaking public keys to the world
  • Laying down a strategy to move configurations into configuration files
  • Making configuration settings as testable as any module.

The following sections will explore more on making points stated above work.

Layers of configuration of nodejs applications

Although this blog article provides basic understanding of configuration modularization, it defers configuration management to another blog post: “Configuring nodejs applications”.

From a production readiness perspective, at least in the context of this blog post, there are two distinct layers of application configurations.

The first layer consists of configurations that nodejs application needs to execute intrinsic business logic. They will be referred to as environment variables/settings. Third-party issued secret keys or server port number configurations, fall under this category. In most cases, you will find such configurations in static variables found in the application.

The second layer consists of configurations required by a system that is going to host the nodejs application. Database server settings, monitoring tools, SSH keys, and other third-party programs running on the hosting entity, are few examples that fall under this category. We will refer to these as system variables/settings.

This blog will be about working with the first layer: environment settings.

Decoupling code from configuration

The first step in decoupling configuration from code is to identify and normalize the way we store our environment variables.

module.exports = function hasSecrets(){
    const SOME_SECRET_KEY = 'xyz=';
    ...
};

Example: function with an encapsulated secret

The previous function encapsulates secret values that can be moved outside the application. If we apply this technique, SOME_SECRET_KEY will be moved outside the function, and imported whenever needed instead.

const SOME_SECRET_KEY = require("./config").SOME_SECRET_KEY;

module.exports = function hasSecrets(){
    ...
};

Example: function with a decoupled secret value

This process has to be repeated all over the application, till every single secret value is replaced with its constant equivalent. It doesn't have to be good on the first try, it has simply to work. We can make it better later on.

Configuration modularization

For curiosity's sake, how does the config.js looks like, after “decoupling configuration from code” step would look like, at the end of the exercise?

export const SOME_SECRET_KEY = 'xyz=';

Example: the first iteration of decoupling configuration from code

This step works but has essentially two key flaws:

  • In a team of multiple players, each player having its own environment variables, the config.js will become a liability. It doesn't scale that well.
  • This strategy will not prevent catastrophe of leaking the secret to the public, in case the code becomes open source.

To mitigate this, we are going to introduce After normalization of the way we store and retrieve environment variables, the next step is how to organize the results in a module. Modules are portable and easy to test.

Modularization makes it possible to test configuration in isolation. Yes, we will have to prove to ourselves it works, before we convince others that it does!

Measures to prevent private key leakage

The first line of defense when it comes to preventing secret keys from leaking to the public is to make sure not a single private value is stored in the codebase itself. The following example illustrates this statement.

module.exports = function leakySecret(){
    const SOME_SECRET_KEY = 'xyz=';
    ...
};

Example: function with a leak-able secret key

The second line of defense is to decouple secret values from an integral part of the application, and use an external service to provision secret values at runtime. nodejs makes it possible to read the process content.

A simple yet powerful tool is dotenv library. This library can be swapped, depending on taste or project requirements.

One of the alternatives to dotenv includes convict.js.

Last but not least, since we are using git, to add .gitignore prevents contributors to commit their .env files by accident to the shared repository.

dotenv-extended makes it possible to read *nix variables into a dotenv file.

require('dotenv').config();
const Twitter = require('twitter');

function TwitterClient(accounts, ids) {
    this.client = new Twitter({
        consumer_key: process.env.TWITTER_CONSUMER_KEY,
        consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
        access_token_key: process.env.TWITTER_ACCESS_TOKEN_KEY,
        access_token_secret: process.env.TWITTER_ACCESS_TOKEN_SECRET
    });
    ...
}

Example: preventing .env files from being checked into the central repository

Conclusion

Modularization is key to crafting re-usable composable software components. The configuration layer is not an exception to this rule. Modularization of configurations brings elegance, ease of management of critical information such as security keys.

In this article, we re-asserted that with a little bit of discipline, without breaking our piggy bank, it is still possible to better manage application configurations. Modularization of configuration makes it possible to reduce the risk of secret key leaks as well increasing testability readiness. There are additional complimentary materials in the “Testing nodejs applications” book.

References

tags: #snippets #modularization #nodejs #configuration