#765·kue

Performance problem under high load

Author: scaryguyCreated Nov 26, 2015Updated Feb 9, 2020
Labelsneed more infounknown

Hi, firstly thanks for this cool library.

If the number of jobs being created is not too much, this library is perfect. I wanted to use this library in a heavy load scenario. Unfortunately results of my experiments were not so bright.

I'm working on > 100.000 keys of sample data. In Redis:

debug populate 100000 my:namespace

Here is my job creator:

javascript
var kue = require('kue');
var ioredis = require('ioredis');
var redis = ioredis();
var jobs = kue.createQueue();

var stream = redis.scanStream({
        match: 'my:namespace:*',
        count: 100
    });

    var keys = [];
    stream.on('data', function(resultKeys) {
        resultKeys.forEach(function(item, i) {
            console.log(i, item);
            keys.push(item);
        })

    });
    stream.on('end', function() {
        console.log("FINISHED!");
        keys.forEach(function(item) {
            jobs.create('ad_fetch', {
                item: item
            }).save();
        })
    });

    jobs.on('error', function(err) {
        console.log("There was an error!:");
        console.log(err);
    })

And here is my worker code:

javascript
var kue = require('kue');
var jobs = kue.createQueue();

jobs.process('ad_fetch', 100, function(job, done) {
  var data = job.data.item
  console.log(data + " is BEING PROCESSED!");
  done();

});

In this scenario, it takes 2 seconds to fetch 100.000 records from Redis using .scan stream. After the streaming ends and I have all keys inside an array and I iterate over it to create jobs for per key returned from Redis. Even though stream is complete, it takes almost 13 seconds for jobs to be created. I know that it's the time elapsed during necessary key/value creation in Redis. But, frankly, I think there should be some stuff to be optimized. For example why does the queue should wait for all jobs to be saved to start executing?

Am I doing something wrong? If there is a better way to do achieve what I'm trying to do, I'd be glad if you share it. Thank you!

Update: By the way, I thought maybe the reason could be limitations of a single Redis instance and I installed a second Redis instance. I seperated them in my sample app. I mean data is received from Redis1 and jobs data is saved to Redis2... Nothing changed. Same performance.