Node.js Job Scheduling with Agenda

I have a freelance project I’ve been working on that needed to be setup to allow tasks to be retried automatically. The application depends upon an integration between two 3rd-party systems, and those systems only sync up at certain intervals. If one system didn’t have the information I needed to wait until it did. So I started looking at different scheduling and queueing systems to see if something already existed that would work for my situation. I came across Agenda.js, and it seemed to fit the bill well. I already had a MongoDB in place for my client and this wouldn’t require me using any new services.

I installed the npm package as usual and then went about setting everything up. I wanted to keep as much of the code for agenda and the tasks separate as possible so I created a new directory for all the code, in my case server/lib/processingQueue.

I added an index.js file where I would do most of the setup. I don’t expect to have many items using agenda for the moment so I turned the concurrency way down.

const Agenda = require('agenda');

const agenda = new Agenda({
    db: { address: process.env.MONGODB_URI },
    maxConcurrency: 5,
    defaultConcurrency: 1
});

agenda.on('ready', function () {
    agenda.define('myJobName', function(job, done) {
        // ...
        done();
    });

    agenda.start();
});

module.exports = {
    agenda,
}

This was a good start, but I didn’t want to clutter the ready event listener with function definitions when I added more tasks in the future. So I created another file in the server/lib/processingQueue directory and gave it the same name as the job. This is where I would define the job and have all the logic.

const { tryDoTheThing } = require('../myLibraryFile');
const { db } = require('../db');
const JOB_NAME = 'myJobName';

function defineJob(agenda) {
    console.log(`Defining ${JOB_NAME} job`);
    agenda.define(JOB_NAME, jobFunction);
}

async function jobFunction(job, done) {
    let { itemId } = job.attrs.data;
    if (!itemId) {
        return done();
    }
    let item = db.findById(itemId);

    let success = await tryDoTheThing(item);

    if (!success) {
        throw new Error(`Failed to do the thing in myJobName job, itemId ${itemId}`);
    }
    done(success);
}


function scheduleJob(agenda, itemId) {
    var job = agenda.create(JOB_NAME, { itemId });
    job.schedule('in 1 hour');
    job.unique({'data.itemId': 'itemId'});
    job.save(function (err) {
        if(err) {
            console.error(`Error saving job for itemId ${itemId}`, err);
        }
        console.log(`Job successfully saved for itemId ${itemId}`);
    });
}

module.exports = {
    JOB_NAME,
    defineJob,
    scheduleJob,
};

After that I went back to my main file for Agenda and updated it to use the new module.

const Agenda = require('agenda');
const myJobName = require('./myJobName');

const agenda = new Agenda({
    db: { address: process.env.MONGODB_URI },
    maxConcurrency: 5,
    defaultConcurrency: 1
});

agenda.on('ready', function () {
    myJobName.defineJob(agenda);

    agenda.start();
});

module.exports = {
    agenda,
    scheduleMyJobName: async (itemId) => myJobName.scheduleJob(agenda, itemId),
}

Note the exports at the bottom. I’m exporting an arrow function that calls myJobName.scheduleJob. I did this so that no matter where I was I’d only have to import one file to get the Agenda tasks. This works out very nicely because it also meant that if I forgot to import the application in my main JavaScript file it would still start Agenda anywhere. Importing Agenda in multiple places won’t be a problem like it might seem at first because Node only evaluates the file once, then it just passes the same reference after that first import.