Page MenuHomePhabricator
Create Task
Maniphest T242726

Improve the Kafka job queue's handling of unknown new jobs
Open, MediumPublic
Actions

Description

Per T240518#5751630, newly defined jobs are processed as part of a catchall low_traffic_jobs group unless a separate queue is specifically defined for them via ChangeProp rule. This creates a risk of a misbehaving job blocking a large number of miscellaneous jobs, as seen recently with the misbehaving fetchGoogleCloudVisionAnnotation job.

Some possible ideas of how to improve the situation:

  • Do not add jobs with release timestamps to the catchall group (throw an exception instead)
  • Require explicitly registering jobs via ChangeProp rule, with review to ensure they are placed in the correct group
  • Do not assume unknown new jobs will be low-traffic

Related Objects

Event Timeline

• Mholloway created this task.Jan 14 2020, 10:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 14 2020, 10:16 AM
Krinkle moved this task from Untriaged to EventBus infra on the WMF-JobQueue board.Mar 6 2020, 11:07 PM
elukey subscribed.Mar 16 2020, 11:32 AM

Any progress on this? :)

• Pchelolo added a project: Platform Team Workboards (Clinic Duty Team).Mar 16 2020, 10:41 PM
• AMooney subscribed.Mar 27 2020, 12:05 PM

@Mholloway can you update the priority for this task?

• Mholloway triaged this task as Medium priority.Mar 27 2020, 1:06 PM
daniel edited projects, added: Platform Engineering; removed: Platform Team Workboards (Clinic Duty Team).Apr 14 2020, 8:39 PM
daniel moved this task from Inbox to Future Initiatives/Small Projects on the Platform Engineering board.
Krinkle edited projects, added: Sustainability (Incident Followup); removed: Wikimedia-Incident.Apr 28 2020, 9:50 PM
Naike added a project: Platform Engineering Roadmap Decision Making.Oct 9 2020, 12:50 PM
Naike moved this task from Untriaged to Icebox on the Platform Engineering Roadmap Decision Making board.Nov 16 2020, 3:25 PM
• CCicalese_WMF removed a project: Platform Engineering.Feb 24 2021, 11:37 PM
Krinkle removed a project: Platform Engineering Roadmap Decision Making.Jul 13 2026, 12:03 AM
Krinkle removed subscribers: • AMooney, • Mholloway.
Maintenance_bot added a project: MediaWiki-API-Platform-Team.Jul 13 2026, 12:31 AM
aaron subscribed.Jul 14 2026, 9:04 PM

Note that while renqueue_delay defaults to 5 in the change-propagation repo, we set it much higher for the job types known to use a certain delay. This is done in order to avoid the "renqueue+ack" logic to avoid flooding kafka with re-enqueued messages:

aaron@aaron-Aspire-S5-371T:~/PhpstormProjects/deployment-charts$ colorgrep -r -C5 reenqueue_delay .
./helmfile.d/services/changeprop-jobqueue/values.yaml-      fetchGoogleCloudVisionAnnotations:
./helmfile.d/services/changeprop-jobqueue/values.yaml-        enabled: true
./helmfile.d/services/changeprop-jobqueue/values.yaml-        concurrency: 10
./helmfile.d/services/changeprop-jobqueue/values.yaml-        # All the jobs of this kind are delayed exactly 48 hours, so we don't want
./helmfile.d/services/changeprop-jobqueue/values.yaml-        # the reenqueue feature to kick in.
./helmfile.d/services/changeprop-jobqueue/values.yaml:        reenqueue_delay: 259200
./helmfile.d/services/changeprop-jobqueue/values.yaml-      notificationGetStartedJob:
./helmfile.d/services/changeprop-jobqueue/values.yaml-        enabled: true
./helmfile.d/services/changeprop-jobqueue/values.yaml-        concurrency: 10
./helmfile.d/services/changeprop-jobqueue/values.yaml-        # All the jobs of this kind are delayed exactly 48 hours, avoid reenqueueing them
./helmfile.d/services/changeprop-jobqueue/values.yaml-        # by setting the reenqueue delay to 72 hours
./helmfile.d/services/changeprop-jobqueue/values.yaml:        reenqueue_delay: 259200
./helmfile.d/services/changeprop-jobqueue/values.yaml-      notificationKeepGoingJob:
./helmfile.d/services/changeprop-jobqueue/values.yaml-        enabled: true
./helmfile.d/services/changeprop-jobqueue/values.yaml-        concurrency: 10
./helmfile.d/services/changeprop-jobqueue/values.yaml-        # All the jobs of this kind are delayed exactly 48 hours, avoid reenqueueing them
./helmfile.d/services/changeprop-jobqueue/values.yaml-        # by setting the reenqueue delay to 72 hours
./helmfile.d/services/changeprop-jobqueue/values.yaml:        reenqueue_delay: 259200
./helmfile.d/services/changeprop-jobqueue/values.yaml-      notificationReEngageJob:
./helmfile.d/services/changeprop-jobqueue/values.yaml-        enabled: true
./helmfile.d/services/changeprop-jobqueue/values.yaml-        concurrency: 10
./helmfile.d/services/changeprop-jobqueue/values.yaml-        # All the jobs of this kind are delayed exactly 48 hours, avoid reenqueueing them
./helmfile.d/services/changeprop-jobqueue/values.yaml-        # by setting the reenqueue delay to 72 hours
./helmfile.d/services/changeprop-jobqueue/values.yaml:        reenqueue_delay: 259200
./helmfile.d/services/changeprop-jobqueue/values.yaml-      newcomerTasksCacheRefreshJob:
./helmfile.d/services/changeprop-jobqueue/values.yaml-        enabled: true
./helmfile.d/services/changeprop-jobqueue/values.yaml-        concurrency: 10
./helmfile.d/services/changeprop-jobqueue/values.yaml-        # All the jobs of this kind are delayed exactly 144 hours (6 days), avoid reenqueueing them
./helmfile.d/services/changeprop-jobqueue/values.yaml-        # by setting the reenqueue delay to 7 days
./helmfile.d/services/changeprop-jobqueue/values.yaml:        reenqueue_delay: 604800

A quick and dirty way to do this would be for JobQueueEvent bus config to take a "delayableTypes" parameter, that we'd configure to only have entries that have deployed changeprop-jobqueue/values.yaml config with reenqueue_delay. Other types would error out upon attempted insertion.

aaron moved this task from Incoming (Needs Triage) to Needs Further Discussion on the MediaWiki-API-Platform-Team board.Jul 15 2026, 6:20 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits