Our job queue system has some unique functionality for job execution that we'll need to consider in the design and transition towards a more general change propagation service. Additionally, the job runner directly interacts with the existing job queue, which will need to be changed if we are to abstract job dispatching into a separate service.
The job queue provides several mechanisms to reduce duplicate work:
- a root job signature hash
- a root job timestamp
Both root and leaf jobs are typically de-duplicated using this information. Newer jobs abort both root & leaf jobs corresponding to an older job with the same job signature.
There are two levels of batching:
- Each leaf job typically processes $wgUpdateRowsPerQuery titles.
- The JobRunner pops & processes several jobs of a given type in a loop.
Each job runner run is triggered by a HTTP request to Special:RunJobs.