Page MenuHomePhabricator

Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main
Open, NormalPublic

Description

This includes making the minor schema changes needed, i.e. meta.schema_uri -> $schema and meta.topic -> meta.stream.

Event streams to migrate in order of lowest to highest risk:

  • user-blocks-change
  • revision-tags-change
  • page-properties-change
  • page-links-change
  • page-restrictions-change
  • campaign-{create,change,delete}
  • page-move
  • page-delete/undelete
  • page-create
  • recentchange - public
  • revision-visibility-change - security implications
  • revision-create - this is basically the starter event that is beginning most of the update chains for change-prop
  • resource-change

Event Timeline

Ottomata created this task.Dec 5 2018, 9:58 PM
Ottomata triaged this task as Normal priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2018, 9:58 PM
Ottomata moved this task from Backlog to Next Up on the EventBus board.Dec 5 2018, 10:02 PM
Ottomata removed Ottomata as the assignee of this task.Dec 5 2018, 10:10 PM
Nuria assigned this task to Pchelolo.Dec 5 2018, 10:17 PM
Nuria removed the point value for this task.

Change 506158 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] Add mediawiki/page/undelete/0.0.4 - draft-07 new schema style

https://gerrit.wikimedia.org/r/506158

Change 506158 merged by Ottomata:
[mediawiki/event-schemas@master] Add 1.0.0 versions of schemas using new draft-07 schema conventions

https://gerrit.wikimedia.org/r/506158

Change 508859 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] Add 1.0.0 draft-07 versions of revision/tags-change and user/blocks-change

https://gerrit.wikimedia.org/r/508859

Change 508859 merged by Ottomata:
[mediawiki/event-schemas@master] Add 1.0.0 draft-07 versions of revision/tags-change and user/blocks-change

https://gerrit.wikimedia.org/r/508859

@Pchelolo T218346: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main is pretty much done! We are ready to start porting events, I guess after T222822: Make EventBus extension support configurable per-event/stream EventServiceName is done too.

eventgate-main is running and configured in beta, so all that is needed there is for EventBus extension to send them to the eventgate-main service, instead of the default eventbus service.

Ottomata renamed this task from Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to EventGate to Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main.May 17 2019, 4:57 PM

Change 514301 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Use eventgate-main as default EventBus EventService in beta

https://gerrit.wikimedia.org/r/514301

Change 514301 merged by Ottomata:
[operations/mediawiki-config@master] Use eventgate-main as default EventBus EventService in beta

https://gerrit.wikimedia.org/r/514301

Mentioned in SAL (#wikimedia-operations) [2019-06-04T15:27:02Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: Use eventgate-main in beta. No-op in prod. T211248 (duration: 00m 49s)

Mentioned in SAL (#wikimedia-operations) [2019-06-04T15:50:34Z] <otto@deploy1001> Synchronized wmf-config/CommonSettings.php: Configure eventgate-main EventService. No-op in prod. T211248 (duration: 01m 19s)

Change 514523 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] [EventBus] use eventgate-main for 2 events on all wikis

https://gerrit.wikimedia.org/r/514523

Change 514523 merged by Ottomata:
[operations/mediawiki-config@master] [EventBus] use eventgate-main for 2 events on all wikis

https://gerrit.wikimedia.org/r/514523

Mentioned in SAL (#wikimedia-operations) [2019-06-05T16:22:38Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: use eventgate-main for 2 events on all wikis - T211248 (duration: 00m 55s)

Change 514525 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] [EventBus] Revert - Send user-blocks-change using eventbus

https://gerrit.wikimedia.org/r/514525

Change 514525 merged by Ottomata:
[operations/mediawiki-config@master] [EventBus] Revert - Send user-blocks-change using eventbus

https://gerrit.wikimedia.org/r/514525

Mentioned in SAL (#wikimedia-operations) [2019-06-05T16:29:14Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Revert user-blocks-change to use eventbus and old schema - T211248 (duration: 00m 54s)

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Thu, Jun 6, 5:04 PM

Change 516329 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventBus@master] Slight refactor to enable stream config use outside of EventBusHooks

https://gerrit.wikimedia.org/r/516329

Change 516332 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/vagrant@master] Use either EventGate or eventlogging-service-eventbus, but not both

https://gerrit.wikimedia.org/r/516332

Change 516329 merged by Ppchelko:
[mediawiki/extensions/EventBus@master] Slight refactor to enable stream config use outside of EventBusHooks

https://gerrit.wikimedia.org/r/516329

Ottomata moved this task from Next Up to In Progress on the EventBus board.Mon, Jun 17, 1:18 PM

Change 517421 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] Use additionalProperties for map type instead of patternProperties

https://gerrit.wikimedia.org/r/517421

Change 517421 merged by Ottomata:
[mediawiki/event-schemas@master] Use additionalProperties for map type instead of patternProperties

https://gerrit.wikimedia.org/r/517421

Ottomata added a comment.EditedMon, Jun 17, 2:03 PM

For my own reference, here are the steps needed to migrate an event from eventbus service to eventgate-main:

  • Make sure eventgate-main is deployed with most recent event-schemas repo, and has stream-config to allow schemas for the right streams.
  • Update wgEventServiceStreamConfig in mediawiki-config InitialiseSettings.php to change the event stream's EventServiceName.
  • Deploy InitialiseSettings.php to mwdebug1002, trigger an event, and verify that it works.
  • Deploy InitialiseSettings.php to rest of fleet.
  • Move stream from eventlogging_eventbus refine job to mediawiki_events using schema service in refine.pp
  • Verify that Refine continues to import events into Hive.

Change 517435 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] Fix error schema $id and version

https://gerrit.wikimedia.org/r/517435

Change 517435 merged by Ottomata:
[mediawiki/event-schemas@master] Fix error schema $id and version

https://gerrit.wikimedia.org/r/517435

Change 517452 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Produce mediawiki.user-blocks-change stream to eventgate-main

https://gerrit.wikimedia.org/r/517452

Change 517452 merged by Ottomata:
[operations/mediawiki-config@master] Produce mediawiki.user-blocks-change stream to eventgate-main

https://gerrit.wikimedia.org/r/517452

Mentioned in SAL (#wikimedia-operations) [2019-06-17T17:10:55Z] <ottomata> mw-config change to produce user-blocks-change event to eventgate-main - T211248

Mentioned in SAL (#wikimedia-operations) [2019-06-17T17:14:43Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Produce user-blocks-change to eventgate-main - T211248 (duration: 00m 48s)

Change 517465 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] Fix centralnotice/campaign event $id

https://gerrit.wikimedia.org/r/517465

Change 517465 merged by Ottomata:
[mediawiki/event-schemas@master] Fix centralnotice/campaign event $id

https://gerrit.wikimedia.org/r/517465

Status:

page-properties-change and user-blocks-change are now routed through eventgate-main. I'd like to proceed with more events this week, but I think @Pchelolo is at an offsite. I am not sure which events we should do next, so I'll wait for now.

Ok, I think this could be the list of remaining events from lowest to the highest risk, events that are coming from EventBus extension. The events coming from other sources (RB/ChangeProp) are not considered here as they are the highest risk.

  • revision-tags-change
  • page-links-change
  • page-restrictions-change
  • page-move
  • page-delete/undelete
  • page-create - interesting one since it shares schema with revision-create. Maybe move together with it
  • recentchange - public
  • revision-visibility-change - security implications
  • revision-create - this is basically the starter event that is beginning most of the update chains for the change-prop

Change 517643 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Produce page-properties-change stream to eventgate-main

https://gerrit.wikimedia.org/r/517643

Change 517643 merged by Ottomata:
[operations/mediawiki-config@master] Produce page-properties-change stream to eventgate-main

https://gerrit.wikimedia.org/r/517643

Mentioned in SAL (#wikimedia-operations) [2019-06-18T13:48:14Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Produce page-properties-change to eventgate-main - T211248 (duration: 00m 48s)

Change 517648 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Produce page-links-change stream to eventgate-main

https://gerrit.wikimedia.org/r/517648

Mentioned in SAL (#wikimedia-operations) [2019-06-18T14:04:05Z] <ottomata> deploying mediawiki-config to Produce page-linkT211248s-change stream to eventgate-main - T211248

Change 517648 merged by jenkins-bot:
[operations/mediawiki-config@master] Produce page-links-change stream to eventgate-main

https://gerrit.wikimedia.org/r/517648

Mentioned in SAL (#wikimedia-operations) [2019-06-18T14:05:45Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Produce page-links-change to eventgate-main - T211248 (duration: 00m 48s)

Change 517652 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Produce page-restrictions-change stream to eventgate-main

https://gerrit.wikimedia.org/r/517652

Change 517652 merged by Ottomata:
[operations/mediawiki-config@master] Produce page-restrictions-change stream to eventgate-main

https://gerrit.wikimedia.org/r/517652

Mentioned in SAL (#wikimedia-operations) [2019-06-18T14:12:50Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Produce page-restrictions-change to eventgate-main - T211248 (duration: 00m 47s)

Ottomata updated the task description. (Show Details)Tue, Jun 18, 2:17 PM
Ottomata updated the task description. (Show Details)Tue, Jun 18, 2:25 PM

Change 517664 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move newly migrated eventgate-main tables to new Refine job

https://gerrit.wikimedia.org/r/517664

Change 517664 merged by Ottomata:
[operations/puppet@production] Move newly migrated eventgate-main tables to new Refine job

https://gerrit.wikimedia.org/r/517664

Change 516332 merged by Ottomata:
[mediawiki/vagrant@master] Use either EventGate or eventlogging-service-eventbus, but not both

https://gerrit.wikimedia.org/r/516332

Apparently, change-prop is not prepared for the new event schemas.

Before we update it, we should hold switching topics that are used in change-prop:

page-properties-change
page-move
page-delete/undelete
page-create
revision-visibility-change
revision-create
resource-change

The others can still be switched.