I have tried several times to reduce test time of Node-RED flow test and failed. For the flow test, I use node-red-node-test-helper module that actually loads flow.json file and run it on a http server with Node-RED runtime. Its test time is acceptable as long as the flow.json file is relatively small. However, the bigger project it grows, the slower the test is. It took 30 – 40 minutes in my project. Tests are not often executed if it takes long time. In addition to that, we have to wait for the build for a while even if the change is quite small.
I finally found a solution for that, so I will share my solution with you.
Looking for the bottleneck
To improve tha performance, we should know where the bottleneck is. I put code to measure the time. My code in my project has more codes than following one but we can check where the problem is.
const helper = require("../index.js");
helper.init(require.resolve('node-red'));
describe(("Test "), () => {
beforeEach((done) => {
console.time("startServer");
helper.startServer(() => {
console.timeEnd("startServer");
done();
});
});
afterEach((done) => {
console.time("unload");
helper.unload()
.then(() => {
console.timeEnd("unload");
console.time("stopServer");
helper.stopServer(() => {
console.timeEnd("stopServer");
done();
});
});
});
// Definitions of requiredNodes and flow are omitted
it(`should do something`, (done) => {
console.time("load");
helper.load(requiredNodes, flow, () => {
console.timeEnd("load");
const outNode = helper.getNode("out-node-id");
outNode.on("input", (msg) => {
try {
should(msg.payload).be.exactly(12345);
done();
} catch (e) {
done(e);
}
});
const inputNode = helper.getNode("input-node-id");
inputNode.send({ payload: 12345 });
});
});
});The test result is following.
stopServer: 1.074ms
startServer: 5.293ms
load: 356.088ms
unload: 13.543msload function is the bottleneck. load function loads flow.json file and does loop of loop of loop… The process is done inside of Node-RED runtime for the initialization. If we have 1000 test cases it takes more than 6 minutes! If we have 10 similar flows it takes 1 hour! It’s horrible. That’s why I tried to improve this several times.
Do you want to know how big the flow.json file is? It looks like following. It’s for this test.
Return cache for load function
We found where the problem is. We can reduce the time of we call load function only once. I modified the code of node-red-node-test-helper for it. You can find the module here.
I haven’t created a PR yet. As far as I remember the author wrote a comment that said this module is for node test but not for flow test. I will update this article according to the PR.
load function initializes the flow only once. If it has cache it returns it. We need to set callback function to load function. Its callback is of course different for each call because the cache is promise of startFlows function of Node-RED runtime. We can set different callback for each call.
How to apply the change to our flow test
This is most important point. How can we apply the change to our flow test? Go to following page first if you haven’t written tests for flow file.
The good point of my modification to node-red-node-test-helper is existing tests still work. We can apply the change one by one.
I added following 3 functions to helper. I will explain how to use them.
addListener(id, cb);
removeAllListeners();
restart(settings);First of all, we should make sure that the server starts/ends only once. Let’s define helper.startServer in before function, helper.unload and helper.stopServer in after function.
before((done) => {
helper.startServer(done);
});
after((done) => {
helper.unload()
.then(() => helper.stopServer(done));
});Next step is registering validation listener. We needed to register validation listener in this way.
const outNode = helper.getNode("out-node-id");
outNode.on("input", (msg) => {
try {
should(msg.payload).be.exactly(12345);
done();
} catch (e) {
done(e);
}
});But this way causes problems because we call helper.load only once which means that the flow keeps running. If we register a listener to outNode we have to remove it before executing next test. Otherwise, one test message fires two listeners and test start failing.
With modified node-red-node-test-helper, we need to register the listener via helper.addListener function.
helper.addListener("out-node-id", (msg) => {
try {
should(msg.payload).be.exactly(12345);
done();
} catch (e) {
done(e);
}
});It stores the node and listener function. How can we remove the listener then? Let’s define it in afterEach function.
afterEach(() => {
helper.removeAllListeners();
});This function removes all listeners registered by us. The final code looks following. It contains two ways for the comparison, without cache and with cache.
const path = require("path");
const helper = require("../index.js");
helper.init(require.resolve('node-red'));
describe("flow test", () => {
let flow;
let inputNodeIds = [];
let outputNodeIds = [];
before(() => {
flow = require("./flows/flows.json");
flow.forEach((node) => {
if (node.type === "inject") {
inputNodeIds.push(node.id);
node.type = "helper";
}
if (node.type === "debug") {
outputNodeIds.push(node.id);
node.type = "helper";
}
});
console.log(inputNodeIds.length)
console.log(outputNodeIds.length)
});
const requiredNodes = [
require("../node_modules/@node-red/nodes/core/common/20-inject.js"),
require("../node_modules/@node-red/nodes/core/common/21-debug.js"),
require("../node_modules/@node-red/nodes/core/function/10-function.js"),
require("../node_modules/@node-red/nodes/core/function/rbe.js"),
];
describe(("Without cache"), () => {
beforeEach((done) => {
helper.startServer(done);
});
afterEach((done) => {
helper.unload()
.then(() => helper.stopServer(done));
});
for (let i = 0; i < 100; i++) {
it(`should succeed ${i}`, (done) => {
helper.load(requiredNodes, flow, () => {
const listener = (msg) => {
try {
should(msg.payload).be.exactly(12345);
done();
} catch (e) {
done(e);
}
};
helper.addListener(outputNodeIds[i], listener);
const inputNode = helper.getNode(inputNodeIds[i]);
inputNode.send({ payload: 12345 });
});
});
}
});
describe(("With cache"), () => {
before((done) => {
helper.startServer(done);
});
afterEach(() => {
helper.removeAllListeners();
});
after((done) => {
helper.unload()
.then(() => helper.stopServer(done));
});
for (let i = 0; i < 160; i++) {
it(`should succeed ${i}`, (done) => {
helper.load(requiredNodes, flow, () => {
const listener = (msg) => {
try {
should(msg.payload).be.exactly(12345);
done();
} catch (e) {
done(e);
}
};
helper.addListener(outputNodeIds[i], listener);
const inputNode = helper.getNode(inputNodeIds[i]);
inputNode.send({ payload: 12345 });
});
});
}
it(`should block the same value (rbe node)`, (done) => {
helper.restart().then(() => {
helper.load(requiredNodes, flow, () => {
let count = 0;
const listener = (msg) => {
try {
count++;
if (count === 1 && count === 3) {
should(msg.payload).be.exactly(12345);
} else if (count === 2) {
should(msg.payload).be.exactly(123456);
}
if (count === 3) {
done();
}
} catch (e) {
done(e);
}
};
const outputNodeId = "1002a90a.f17117";
helper.addListener(outputNodeId, listener);
const inputNodeId = "f8321156.33691";
const inputNode = helper.getNode(inputNodeId);
inputNode.send({ payload: 12345 });
inputNode.send({ payload: 12345 });
inputNode.send({ payload: 123456 });
inputNode.send({ payload: 123456 });
inputNode.send({ payload: 12345 });
});
});
});
});
});Look at the difference between the two results!
flow test
160
160
Without cache
√ should succeed 0 (445ms)
√ should succeed 1 (370ms)
√ should succeed 2 (362ms)
√ should succeed 3 (333ms)
√ should succeed 4 (349ms)
√ should succeed 5 (315ms)
√ should succeed 6 (348ms)
√ should succeed 7 (321ms)
√ should succeed 8 (313ms)
√ should succeed 9 (318ms)
With cache
√ should succeed 0 (396ms)
√ should succeed 1
√ should succeed 2
... (omitted)
√ should succeed 158
√ should succeed 159
√ should block the same value (rbe node) (387ms)
171 passing (5s)Without cache, it takes more than 300 ms for all tests whereas it takes only first test 400 ms. It is really big difference.
Do you really need to restart the flow?
Do you recognize that I added additional test for rbe? We have to restart the flow in some cases. If the flow contains nodes that store received data we have to restart the flow. For example, rbe node send the received message if the value is different from previous one. In this case, we need to reset the state.
Test fails in following steps. Let’s assume that test 2 want to test if the out node always receives different value.
rbe node receives
- 11 in test 1
- 11 in test 2
- 11 in test 2
- 12 in test 2
- 11 in test 2
Step2 – Step5 are for test 2 but out node doesn’t receive first value 11 because rbe node has already received the same value in test 1. For this reason, we have to restart it.
It’s easy to restart the flow. Call helper.restart(). It returns Promise.
it(`test that requires restart`, (done) => {
helper.restart().then(() => {
helper.load(requiredNodes, flow, () => {
// your test code
});
});
});Pass settings if you specify it to helper.init or helper.settings functions.
it(`test that requires restart`, (done) => {
helper.restart(settings).then(() => {
helper.load(requiredNodes, flow, () => {
// your test code
});
});
});Call it in afterEach function if you need restart for the test group.
describe("flow test", () => {
afterEach(async () =>{
await helper.restart();
});
it(`test1`, (done) => {});
it(`test2`, (done) => {});
it(`test3`, (done) => {});
});End
If you write many tests for flow file you definitely face this problem. I hope this article saves your time!
Check following articles as well if you want to learn Node-RED.
Comments