How to stub a stream function
The stream API provides a heavy-weight asynchronous computation model that keeps a small memory footprint. As exciting as it may sound, testing streams is somehow intimidating. This blog layout some key elements necessary to be successful when mocking stream API.
We keep in mind that there is a clear difference between mocking versus stub/spying/fakes even though we used mock interchangeably.
In this article we will talk about:
- Understanding the difference between Readable and Writable streams
- Stubbing Writable stream
- Stubbing Readable stream
- Stubbing Duplex or Transformer streams
Even though this blog post was designed to offer complementary materials to those who bought my Testing
nodejsApplications book, the content can help any software developer to tuneup working environment. You use this link to buy the book.
Show me the code
var gzip = require('zlib').createGzip();//quick example to show multiple pipings
var route = require('expressjs').Router();
//getter() reads a large file of songs metadata, transform and send back scaled down metadata
route.get('/songs' function getter(req, res, next){
let rstream = fs.createReadStream('./several-TB-of-songs.json');
rstream.
pipe(new MetadataStreamTransformer()).
pipe(gzip).
pipe(res);
// forwaring the error to next handler
rstream.on('error', (error) => next(error, null));
});
At a glance The code is supposed to read a very large JSON file of TB of metadata about songs, apply some transformations,
gzip, and send the response to the caller, by piping the results on the response object.
The next example demonstrates how a typical transformer such as MetadataStreamTransformer looks like
const inherit = require('util').inherits;
const Transform = require('stream').Tranform;
function MetadataStreamTransformer(options){
if(!(this instanceof MetadataStreamTransformer)){
return new MetadataStreamTransformer(options);
}
this.options = Object.assign({}, options, {objectMode: true});//<= re-enforces object mode chunks
Transform.call(this, this.options);
}
inherits(MetadataStreamTransformer, Transform);
MetadataStreamTransformer.prototype._transform = function(chunk, encoding, next){
//minimalistic implementation
//@todo process chunk + by adding/removing elements
let data = JSON.parse(typeof chunk === 'string' ? chunk : chunk.toString('utf8'));
this.push({id: (data || {}).id || random() });
if(typeof next === 'function') next();
};
MetadataStreamTransformer.prototype._flush = function(next) {
this.push(null);//tells that operation is over
if(typeof next === 'function') {next();}
};
Inheritance as explained in this program might be old, but illustrates good enough in a prototypal way that our
MetadataStreamTransformerinherits stuff fromStream#Transformer
What can possibly go wrong?
stubbing functions in stream processing scenario may yield the following challenges:
- How to deal with the asynchronous nature of streams
- Identify areas where it makes sense to a stub, for instance: expensive operations
- Identifying key areas needing drop-in replacements, for instance reading from a third party source over the network.
Primer
The keyword when stubbing streams is:
- To identify where the heavy lifting is happening. In pure terms of streams, functions that executes
_read()and_write()are our main focus. - To isolate some entities, to be able to test small parts in isolation. For instance, make sure we test
MetadataStreamTransformerin isolation, and mock any response fed into.pipe()operator in other places.
What is the difference between readable vs writable vs duplex streams? The long answer is available in
substack's Stream Handbook
Generally speaking, Readable streams produce data that can be feed into Writable streams. Readable streams can be .piped on, but not into. Readable streams have readable|data events, and implementation-wise, implement ._read() from Stream#Readable interface.
Writable streams can be .piped into, but not on. For example, res examples above are piped to an existing stream. The opposite is not always guaranteed. Writable streams also have writable|data events, and implementation-wise, implement _.write() from Stream#Writable interface.
Duplex streams go both ways. They have the ability to read from the previous stream and write to the next stream. Transformer streams are duplex, implement ._transform() Stream#Transformer interface.
Modus Operandi
How to test the above code by taking on smaller pieces?
fs.createReadStreamwon't be tested, but stubbed and returns a mocked readable stream.pipe()will be stubbed to return a chain of stream operatorsgzipandreswon't be tested, therefore stubbed to returns a writable+readable mocked stream objectsrstream.on('error', cb)stub readable stream with a read error, spy onnext()and check if it has been called uponMetadataStreamTransformerwill be tested in isolation andMetadataStreamTransformer._transform()will be treated as any other function, except it accepts streams and emits events
How to stub stream functions
describe('/songs', () => {
before(() => {
sinon.stub(fs, 'createReadStream').returns({
pipe: sinon.stub().returns({
pipe: sinon.stub().returns({
pipe: sinon.stub().returns(responseMock)
})
}),
on: sinon.spy(() => true)
})
});
});
This way of chained stubbing is available in our toolbox. Great power comes with great responsibilities, and wielding this sword may not always be a good idea.
There is an alternative at the very end of this discussion
The transformer stream class test in isolation may be broken down to
- stub the whole Transform instance
- Or stub the
.push()and simulate a write by feeding in the readable mocked stream of data
the stubbed
push()is a good place to add assertions
it('_transform()', function(){
var Readable = require('stream').Readable;
var rstream = new Readable();
var mockPush = sinon.stub(MetadataStreamTransformer, 'push', function(data){
assert.isNumber(data.id);//testing data sent to callers. etc
return true;
});
var tstream = new MetadataStreamTransformer();
rstream.push({id: 1});
rstream.push({id: 2});
rstream.pipe(tstream);
expect(tstream.push.called, '#push() has been called');
mockPush.restore();
});
How to Mock Stream Response Objects
The classic example of a readable stream is reading from a file. This example shows how mocking fs.createReadStream and returns a readable stream, capable of being asserted on.
//stubb can emit two or more streams + close the stream
var rstream = fs.createReadStream();
sinon.stub(fs, 'createReadStream', function(file){
//trick from @link https://stackoverflow.com/a/33154121/132610
assert(file, '#createReadStream received a file');
rstream.emit('data', "{id:1}");
rstream.emit('data', "{id:2}");
rstream.emit('end');
return false;
});
var pipeStub = sinon.spy(rstream, 'pipe');
//Once called this above structure will stream two elements: good enough to simulate reading a file.
//to stub `gzip` library: another transformer stream: producing
var next = sinon.stub();
//use this function| or call the whole route
getter(req, res, next);
//expectations follow:
expect(rstream.pipe.called, '#pipe() has been called');
Conclusion
In this article, we established the difference between Readable and Writable streams and how to stub each one of them when unit test.
Testing tends to be more of art, than a science, practice makes perfect. There are additional complimentary materials in the “Testing nodejs applications” book.
References
- Testing
nodejsApplications book - More on readable streams(Stream2) ~ Jimmy Chao ~ NeetHack Blog
- QA: Mock Streams ~ StackOverflow Question
- Mock System APIs ~ Gleb Bahmutov Blog
- Streaming to Mongo available for
shard-ed clusters ~mongodbDocs - Source code of glob stream to know more about using Glob Stream
- How to TDD Streams
- Testing with vinyl for writing to files