Skip to main content

How to write and test Node.JS code that performs mutiple async tasks


This is currently WIP, I will fill in more details but it's a good starting version that is still hopefully useful to the reader...


Recently I worked on a project that involved scheduling a number of tasks in parallel in Node.js.

Running mutiple tasks


I used the await-the library to do this.

https://github.com/olono/await-the/blob/master/lib/result.js

There are multiple ways to solve this kind of problem. Mainly you have to understand the Node.js is a single threaded environment (See Event loop) and how callbacks operate with a library like async or await-the.

Basically, here idea was that you write a function that has a callback as its last argument

``` 
const doTask = async (arg1, callback) { 
  const results = await doStuffWith(arg1) ; 
  callback(results);
}
```

Then perform this action on multiple inputs in parallel such as by using the excellent async.mapLimit function if you want to limit how many tasks in parallel you want to to run

  async.mapLimit(items, 5, doTask)

The same result can be achieved using the await-the utility the.result

So here is how you would execute taskObject.doTask with an argument arg1 asynchronously

  const results = await the.result([taskObject, 'doTask'], arg)

Now to run this against mutiple values in parallel you can use the.each (or async.each)

const doParallelTasks = async function (taskObject, valuesList) {
  const results = the.each(valuesList, async value => await the.result([taskObject,'doTask'], value, { limit: 5})
  return results;
}

The limit arg limits how many tasks can be run in parallel, to avoid cpu overload.

Now each doTask instance will be promisified and awaited on, all results will be collected in results array.


Testing this code using Sinon stubs

Testing this code using a stub needs a trick since the callback function must be invoked for the.each to function correctly

Create a stub that calls the callback in some way, at least something like this

const stubFcn = sinon.stub().callsFake((arg, callback) => callback())

One way to to unit test the logic of doParallelTasks is to ensure that taskObject is called as many times as you expect

const stubFcn = sinon.stub().callsFake((arg, callback) => callback())
const results = doParallelTasks(taskObject, [value1, value2]);
assert(results.length).to.be(2);


Comments

Post a Comment

Popular posts from this blog

Authenticating Spring Boot based application against secure LDAP/AD server

Authenticating against an Active Directory setup is quite common in organizations using Spring Boot / Spring Security can be a pain if you don't know exactly the requirements. I needed to add auth in my web app and secure some but not all endpoints of the application. My story was, I needed Spring security to authenticate against my company LDAP server which uses Active Directory I started by using the standard LDAP guide such as this which are all over the Internet, https://spring.io/guides/gs/authenticating-ldap/ and was able to setup the basic framework However, only test level LDAP auth was working for me, when I tried to auth against the company LDAP secure server, I had to resolve a few issues After 1 week and working with several devs at the company, I finally found why it was not working and the fix was easy Since I spent a week or so resolving this, I wanted to write this up in case someone finds this useful. Here is what I did (it was easy until the fourth ...
Post a Comment
Read more

Using custom conditional logic to enable/disable Spring components

If you have a Spring component and you don't want it to load, you can use Spring's predefined conditionals as much as possible. For example, @Component   @ConditionalOnNotWebApplication   public class SchedulerEntryPoint implements ApplicationRunner { ...  } This will not load your component when running in non web application mode. Such as you may want to start the application but without any of the web framework using SpringApplicationBuilder. But sometimes you want to use custom conditions. It's pretty easy to do so, just use something like this @Component @Conditional (SchedulerCheck. class ) public class SchedulerEntryPoint implements ApplicationRunner { public static class SchedulerCheck implements Condition { @Override public boolean matches(ConditionContext conditionContext, AnnotatedTypeMetadata annotatedTypeMetadata) { return System. getProperty ( "scheduler" ) != ...
Post a Comment
Read more

Sending Form data to a backend REST API using Axios

This need is incredibly common and useful, and hopefully will save you a lot of time when doing server side calls from your UI application (or even non UI clients like NodeJS applications) Example here is to send a POST request to an endoint /api/item/new (which will create a new item in the database). We will just assume tbhe backend is already setup (it's not relevant to this article). All we need to know is that we can do a POST /api/item/new and send it form data with two pieces of info     name, filter So, if you have a node.js application (I was using Vue-cli generated project, but it does not matter), install 'axios' (a most popular tool to make server calls these days) npm i axios --save OR yarn add axios (my preferred method) Now, in your service JS file (which is generally when I keep all my api calls) do something like this createNew ( name , filter ) { let formData = new FormData (); formData . append ( "name" , ...
Post a Comment
Read more