Page MenuHomePhabricator

Migrate JobQueue to eventgate
Closed, ResolvedPublic

Description

We need to convert MediaWiki-JobQueue to eventgate.

I propose to take the same approach we've had been using for eventbus events:

  • Add support for per-job configuration of the event destination via wgEventServiceStreamConfig
  • Convert jobs one-by-one

Event Timeline

Change 524942 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@master] Respect wgEventServiceStreamConfig in EventBus.

https://gerrit.wikimedia.org/r/524942

Ottomata moved this task from Backlog to In Progress on the Event-Platform board.Jul 23 2019, 1:48 PM
Ottomata assigned this task to Pchelolo.Jul 24 2019, 4:01 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 524942 merged by Ppchelko:
[mediawiki/extensions/EventBus@master] Respect wgEventServiceStreamConfig in EventBus job queue.

https://gerrit.wikimedia.org/r/524942

Pchelolo added a comment.EditedAug 5 2019, 6:12 PM

Given that we're quite confident in eventgate now as we've transitioned all the kafka-main mediawiki events to it with no issues, and a fairly limited difference between eventbus and eventgate, I propose to do a fairly quick transition. 3 steps should be enough.

  • Convert updateBetaFeatureUsersCount job - it's a tradition to start job-related migrations with this one.
  • Convert hight traffic jobs that have specific mentions in jobqueue changeprop config
  • Convert all jobs

Change 528209 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch updateBetaFeaturesUserCounts job to eventgate.

https://gerrit.wikimedia.org/r/528209

Change 528209 merged by jenkins-bot:
[operations/mediawiki-config@master] Switch updateBetaFeaturesUserCounts job to eventgate.

https://gerrit.wikimedia.org/r/528209

Mentioned in SAL (#wikimedia-operations) [2019-08-06T23:19:39Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch updateBetaFeaturesUserCounts job to eventgate (T228705) (duration: 00m 57s)

Change 528886 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch high-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/528886

Change 528886 merged by jenkins-bot:
[operations/mediawiki-config@master] Switch high-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/528886

Change 528890 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/mediawiki-config@master] Revert "Switch high-traffic jobs to eventgate."

https://gerrit.wikimedia.org/r/528890

Change 528890 merged by Mobrovac:
[operations/mediawiki-config@master] Revert "Switch high-traffic jobs to eventgate."

https://gerrit.wikimedia.org/r/528890

Change 529124 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch high-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/529124

Change 529124 merged by Ottomata:
[operations/mediawiki-config@master] Switch high-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/529124

Mentioned in SAL (#wikimedia-operations) [2019-09-03T19:12:42Z] <ottomata> switching jobqueue events to eventgate-main - T228705

Mentioned in SAL (#wikimedia-operations) [2019-09-03T19:14:00Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch high-traffic jobs to eventgate. Take 2 - T228705 (duration: 00m 56s)

Change 534225 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch all non-low-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/534225

Change 534225 merged by Ottomata:
[operations/mediawiki-config@master] Switch all non-low-traffic jobs to eventgate.

https://gerrit.wikimedia.org/r/534225

Mentioned in SAL (#wikimedia-operations) [2019-09-04T17:32:33Z] <ottomata> Switch all non-low-traffic jobs to eventgate - T228705

Mentioned in SAL (#wikimedia-operations) [2019-09-04T17:34:08Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch all non-low-traffic jobs to eventgate - T228705 (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2019-09-04T17:43:22Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch all non-low-traffic jobs to eventgate - T228705 - take 2 (duration: 00m 55s)

Change 534506 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Switch all events to eventgate.

https://gerrit.wikimedia.org/r/534506

Change 534506 merged by Ottomata:
[operations/mediawiki-config@master] Switch all events to eventgate.

https://gerrit.wikimedia.org/r/534506

Mentioned in SAL (#wikimedia-operations) [2019-09-05T16:04:34Z] <ottomata> switching remaining job queue events (and all remaining events) to eventgate - T228705

Mentioned in SAL (#wikimedia-operations) [2019-09-05T16:05:45Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch all events to eventgate - T228705 (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2019-09-05T16:22:34Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch all events to eventgate - T228705 - take 2 (duration: 00m 49s)