Performance problem under high load
Hi, firstly thanks for this cool library.
If the number of jobs being created is not too much, this library is perfect. I wanted to use this library in a heavy load scenario. Unfortunately results of my experiments were not so bright.
I'm working on > 100.000 keys of sample data. In Redis:
debug populate 100000 my:namespaceHere is my job creator:
var kue = require('kue');
var ioredis = require('ioredis');
var redis = ioredis();
var jobs = kue.createQueue();
var stream = redis.scanStream({
match: 'my:namespace:*',
count: 100
});
var keys = [];
stream.on('data', function(resultKeys) {
resultKeys.forEach(function(item, i) {
console.log(i, item);
keys.push(item);
})
});
stream.on('end', function() {
console.log("FINISHED!");
keys.forEach(function(item) {
jobs.create('ad_fetch', {
item: item
}).save();
})
});
jobs.on('error', function(err) {
console.log("There was an error!:");
console.log(err);
})And here is my worker code:
var kue = require('kue');
var jobs = kue.createQueue();
jobs.process('ad_fetch', 100, function(job, done) {
var data = job.data.item
console.log(data + " is BEING PROCESSED!");
done();
});In this scenario, it takes 2 seconds to fetch 100.000 records from Redis using .scan stream. After the streaming ends and I have all keys inside an array and I iterate over it to create jobs for per key returned from Redis. Even though stream is complete, it takes almost 13 seconds for jobs to be created. I know that it's the time elapsed during necessary key/value creation in Redis. But, frankly, I think there should be some stuff to be optimized. For example why does the queue should wait for all jobs to be saved to start executing?
Am I doing something wrong? If there is a better way to do achieve what I'm trying to do, I'd be glad if you share it. Thank you!
Update: By the way, I thought maybe the reason could be limitations of a single Redis instance and I installed a second Redis instance. I seperated them in my sample app. I mean data is received from Redis1 and jobs data is saved to Redis2... Nothing changed. Same performance.
Source: Automattic/kue