Page MenuHomePhabricator

The mass-message queue reports 0 when there are still queued messages
Open, Needs TriagePublic

Description

The mass-message queue reports 0 when there are still queued messages

A large (50,000+) number of mass-messages were just sent on enwiki, in batches of 5000.

In reviewing status both Special:Statistics and the api view (https://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=statistics) show "queued-massmessages": 0

Manually following the messages shows that the system is still actively distributing messages (i.e. there are certainly not 0 to go)

Event Timeline

Tools like MassMessage add jobs to the job queue for later processing outside of the main MediaWiki process. MediaWiki used to track the job queue itself, but this is no longer the case (T157088). To increase stability and to allow multiple datacenters to handle the job queue, the job queue is now handled by Kafka: https://wikitech.wikimedia.org/wiki/Kafka_Job_Queue. Once MediaWiki sends the job to Kafka, it doesn't know about the job anymore, and the EventBus job queue isn't designed to send information back to MediaWiki about job execution. Job execution is tracked in Grafana, though. You can see the current state of the MassMessage queue in https://grafana.wikimedia.org/d/000000400/jobqueue-eventbus?orgId=1&var-site=eqiad&var-type=All&viewPanel=15 by filtering for "MassMessageJob_0".

We could add a method to JobQueue checking whether reporting statistic like this is supported and remove this statistic altogether from the site info response if that would be useful. Actually reporting the correct number is not technically possible.