FILO queueing #84
Description
Feature suggestion:
Instead of the current FIFO queue job retrieval (lpop), i'd like some queues to work as FILO (rpop).
The queues in question (on my side) contain worker, that spawn more worker, which in turn spawn more workers, thus i can generate several hundred thousand jobs with one initial job (depending on parameters).
This is not a problem in itself. My Problem start where the workers use system resources to do their job.
Even if the last worker in a specific chain cleans up after himself, all the jobs in a chain won't get cleared, since the last job gets executed last in every chain. As a the system resources run out, before the chain gets finished, which in turn yielded failure for all jobs.
Another concern with such swarming jobs is:
- they are hard to keep track of (as in progress)
- with bigger final queues, one would see the results just after a long while
I'd consider making the queues plugable too for this, as this is (coding wise) a minor thing and should be more like a configuration of some sort.
Thoughts?
regards,
scones