The CirrusSearch extension makes extensive use of the JobQueue mechanism: it reacts to concrete MediaWiki events (revision creation, page (un)deletions, page moves, etc) and spawns various JobQueue jobs that update ElasticSearch. The exact process is described in T149408#2762756 .
We had a meeting where we realised these updates would actually profit from being moved to EventBus. This task lists the nest steps needed to be undertaken in order to make this a reality.
- CirrusSearch
- Set a flag controlling whether jobs should be posted directly to the JobQueue or should be sent to EventBus
- Separate the current existing jobs into smaller ones, concretely separate the ES document creation from request submission
- MediaWiki JobQueue - runJobs.php or a similar entity
- Make it accept HTTP requests comprising the job class to execute and the event to execute it against
- Make it return meaningful REST-y status codes based on the outcome of the job's execution
- EventBus / ChangeProp
- Resolve the storage issue: either have a place to stash the resulting documents and have events point to it, or limit the retention policy of the topics that are to deal with ES update documents
- Implement delaying normal jobs
- Partition the backlinks topic(s) based on wikis
- Assess if any other custom logic is needed to handle ES updates, and if so implement them as a module