Page MenuHomePhabricator

Considerations for supporting job queue use cases with the unified event bus
Closed, ResolvedPublic

Description

Our job queue system has some unique functionality for job execution that we'll need to consider in the design and transition towards a more general change propagation service. Additionally, the job runner directly interacts with the existing job queue, which will need to be changed if we are to abstract job dispatching into a separate service.

Job de-duplication

The job queue provides several mechanisms to reduce duplicate work:

  • a root job signature hash
  • a root job timestamp

Both root and leaf jobs are typically de-duplicated using this information. Newer jobs abort both root & leaf jobs corresponding to an older job with the same job signature.

Batching

There are two levels of batching:

  • Each leaf job typically processes $wgUpdateRowsPerQuery titles.
  • The JobRunner pops & processes several jobs of a given type in a loop.

Each job runner run is triggered by a HTTP request to Special:RunJobs.

Delayed jobs

There is some support for running jobs after a delay.

Event Timeline

GWicke raised the priority of this task from to Needs Triage.
GWicke updated the task description. (Show Details)
GWicke added a project: Services.
GWicke added subscribers: GWicke, aaron.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 31 2015, 6:56 PM
Pchelolo closed this task as Resolved.Oct 12 2016, 6:53 PM
Pchelolo claimed this task.

The Event-Platform and ChangeProp now supports everything listed here, so I'm resolving this task

@Pchelolo, what is the equivalent for the delayed job functionality in ChangeProp?

@GWicke Hm.. Actually we don't have any, I've overlooked that point. But do we need it actually?