Page MenuHomePhabricator
Create Task
Maniphest T228705

Migrate JobQueue to eventgate
Closed, ResolvedPublic
Actions

Description

We need to convert MediaWiki-Core-JobQueue to eventgate.

I propose to take the same approach we've had been using for eventbus events:

  • Add support for per-job configuration of the event destination via wgEventServiceStreamConfig
  • Convert jobs one-by-one

Details

ProjectBranchLines +/-Subject
operations/mediawiki-configmaster+1 -111Switch all events to eventgate.
operations/mediawiki-configmaster+30 -0Switch all non-low-traffic jobs to eventgate.
operations/mediawiki-configmaster+27 -0Switch high-traffic jobs to eventgate.
operations/mediawiki-configmaster+0 -33Revert "Switch high-traffic jobs to eventgate."
operations/mediawiki-configmaster+33 -0Switch high-traffic jobs to eventgate.
operations/mediawiki-configmaster+5 -0Switch updateBetaFeaturesUserCounts job to eventgate.
mediawiki/extensions/EventBusmaster+44 -18Respect wgEventServiceStreamConfig in EventBus job queue.
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Pchelolo created this task.Jul 22 2019, 10:23 PM
gerritbot added a comment.Jul 22 2019, 10:40 PM

Change 524942 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@master] Respect wgEventServiceStreamConfig in EventBus.

https://gerrit.wikimedia.org/r/524942

gerritbot added a project: Patch-For-Review.Jul 22 2019, 10:40 PM
Ottomata moved this task from Backlog to In Progress Before Value Streams Kickoff (August 15th) on the Event-Platform Value Stream board.Jul 23 2019, 1:48 PM
Ottomata assigned this task to • Pchelolo.Jul 24 2019, 4:01 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot added a comment.Jul 26 2019, 5:42 PM

Change 524942 merged by Ppchelko:
[mediawiki/extensions/EventBus@master] Respect wgEventServiceStreamConfig in EventBus job queue.

https://gerrit.wikimedia.org/r/524942

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.16; 2019-07-30); removed MW-1.34-notes (1.34.0-wmf.15; 2019-07-23).Jul 26 2019, 6:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 26 2019, 6:10 PM
WDoranWMF moved this task from Modern Event Platform (TEC2) to mop on the Platform Engineering board.Jul 26 2019, 6:30 PM
WDoranWMF edited projects, added Platform Team Initiatives (Modern Event Platform (TEC2)); removed Platform Engineering (Modern Event Platform (TEC2)).
Ottomata mentioned this in T211248: Modern Event Platform: Stream Intake Service: Migrate eventlogging-service-eventbus events to eventgate-main.Jul 30 2019, 3:08 PM
• Pchelolo added a project: Platform Team Workboards (Clinic Duty Team).Aug 5 2019, 6:09 PM
• Pchelolo updated the task description. (Show Details)
• Pchelolo added a comment.EditedAug 5 2019, 6:12 PM

Given that we're quite confident in eventgate now as we've transitioned all the kafka-main mediawiki events to it with no issues, and a fairly limited difference between eventbus and eventgate, I propose to do a fairly quick transition. 3 steps should be enough.

  • Convert updateBetaFeatureUsersCount job - it's a tradition to start job-related migrations with this one.
  • Convert hight traffic jobs that have specific mentions in jobqueue changeprop config
  • Convert all jobs
• Pchelolo moved this task from Inbox to Doing(WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.Aug 5 2019, 6:13 PM
gerritbot added a comment.Aug 5 2019, 6:21 PM

Change 528209 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch updateBetaFeaturesUserCounts job to eventgate.

https://gerrit.wikimedia.org/r/528209

gerritbot added a project: Patch-For-Review.Aug 5 2019, 6:21 PM
gerritbot added a comment.Aug 6 2019, 11:07 PM

Change 528209 merged by jenkins-bot:
[operations/mediawiki-config@master] Switch updateBetaFeaturesUserCounts job to eventgate.

https://gerrit.wikimedia.org/r/528209

Maintenance_bot removed a project: Patch-For-Review.Aug 6 2019, 11:10 PM
Stashbot added a comment.Aug 6 2019, 11:19 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-06T23:19:39Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch updateBetaFeaturesUserCounts job to eventgate (T228705) (duration: 00m 57s)

gerritbot added a comment.Aug 7 2019, 4:20 PM

Change 528886 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch high-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/528886

gerritbot added a project: Patch-For-Review.Aug 7 2019, 4:20 PM
gerritbot added a comment.Aug 7 2019, 4:32 PM

Change 528886 merged by jenkins-bot:
[operations/mediawiki-config@master] Switch high-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/528886

gerritbot added a comment.Aug 7 2019, 4:37 PM

Change 528890 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/mediawiki-config@master] Revert "Switch high-traffic jobs to eventgate."

https://gerrit.wikimedia.org/r/528890

gerritbot added a comment.Aug 7 2019, 4:38 PM

Change 528890 merged by Mobrovac:
[operations/mediawiki-config@master] Revert "Switch high-traffic jobs to eventgate."

https://gerrit.wikimedia.org/r/528890

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2019, 5:10 PM
gerritbot added a comment.Aug 8 2019, 5:06 PM

Change 529124 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch high-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/529124

gerritbot added a project: Patch-For-Review.Aug 8 2019, 5:06 PM
• Pchelolo closed subtask T230048: Change-Prop partitioner fails with eventgate event as Resolved.Aug 9 2019, 7:35 PM
• Pchelolo moved this task from Doing(WIP:5) to Waiting for deployment on the Platform Team Workboards (Clinic Duty Team) board.
gerritbot added a comment.Sep 3 2019, 7:11 PM

Change 529124 merged by Ottomata:
[operations/mediawiki-config@master] Switch high-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/529124

Stashbot added a comment.Sep 3 2019, 7:12 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-03T19:12:42Z] <ottomata> switching jobqueue events to eventgate-main - T228705

Stashbot added a comment.Sep 3 2019, 7:14 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-03T19:14:00Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch high-traffic jobs to eventgate. Take 2 - T228705 (duration: 00m 56s)

gerritbot added a comment.Sep 3 2019, 7:54 PM

Change 534225 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch all non-low-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/534225

gerritbot added a comment.Sep 4 2019, 5:32 PM

Change 534225 merged by Ottomata:
[operations/mediawiki-config@master] Switch all non-low-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/534225

Stashbot added a comment.Sep 4 2019, 5:32 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-04T17:32:33Z] <ottomata> Switch all non-low-traffic jobs to eventgate - T228705

Stashbot added a comment.Sep 4 2019, 5:34 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-04T17:34:08Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch all non-low-traffic jobs to eventgate - T228705 (duration: 00m 56s)

Stashbot added a comment.Sep 4 2019, 5:43 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-04T17:43:22Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch all non-low-traffic jobs to eventgate - T228705 - take 2 (duration: 00m 55s)

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2019, 6:10 PM
gerritbot added a comment.Sep 4 2019, 6:31 PM

Change 534506 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch all events to eventgate.

https://gerrit.wikimedia.org/r/534506

gerritbot added a project: Patch-For-Review.Sep 4 2019, 6:31 PM
gerritbot added a comment.Sep 5 2019, 4:04 PM

Change 534506 merged by Ottomata:
[operations/mediawiki-config@master] Switch all events to eventgate.

https://gerrit.wikimedia.org/r/534506

Stashbot added a comment.Sep 5 2019, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-05T16:04:34Z] <ottomata> switching remaining job queue events (and all remaining events) to eventgate - T228705

Stashbot added a comment.Sep 5 2019, 4:06 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-05T16:05:45Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch all events to eventgate - T228705 (duration: 00m 48s)

Maintenance_bot removed a project: Patch-For-Review.Sep 5 2019, 4:10 PM
Stashbot added a comment.Sep 5 2019, 4:22 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-05T16:22:34Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch all events to eventgate - T228705 - take 2 (duration: 00m 49s)

• Pchelolo closed this task as Resolved.Sep 5 2019, 4:27 PM
• Pchelolo moved this task from Waiting for deployment to Done on the Platform Team Workboards (Clinic Duty Team) board.
• Pchelolo closed subtask T230049: Delayed jobs fail validation in eventgate as Resolved.Sep 11 2019, 8:10 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL