We have a process in cirrus that is using the job queue that needs to evaluate the "pressure" on some jobqueue to throttle its processing. The process is explained in detail in https://phabricator.wikimedia.org/T149408#2762957.
Currently we use JobQueue::getSize() + JobQueue::getDelayedCount() to evaluate the queue pressure to eventually pause this process so that it does not interfere too much with production updates.
EventBus does not implement these methods.
Would there be a way to expose a number that reflect the pressure on a particular queue?
This number could be based on the time spent in the queue (receiveTime - sentTime). MediaWiki could then use this number to decide if it needs to stop/pause such process.
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
We have these metrics already exposed in Grafana:
However, these are per job type only, and are not separated by wiki.
Would using global numbers and pulling them out of Grafana work?
Comment Actions
I think that per queue is perfectly OK for my usecase.
Can we make these numbers accessible from MediaWiki?
Comment Actions
We can fetch the data from burrow as well, but the current version of burrow installed in our production is not stable enough yet, so we might need to wait for update to burrow 1.0 T188719
Comment Actions
With the new queue, the number of pending jobs in the queue will not be an issue, Kafka doesn't really care how many messages behind are we.
So the question is whether we need the process explained in T149408#2762957 or if we can pull it off with just having a moderate concurrency for actual job execution?
Comment Actions
Most of the current extra complexity of the Saneitizer process related to the queue is about throttling in response to backpressure from the indexing pipeline. If we can find a good way to do the same with moderate concurrency it should be workable. I imagine we still have to maintain the old code though as most systems outside wmf will still use the classic job queue (either redis or sql).
Comment Actions
EDIT: removed this comment it was too detailed and not very helpful sorry.
Here is a graph of how many times the threshold is hit.
I think we can try activating this process with the new job queue, it will basically run without any throttling. We can monitor and disable it if it goes out of control.
If a number can be exposed to MW that would allow us to keep this process mostly as is, if not I'll create a new task and would love some help to design it on top of wmf infrastructure.