Migrate these schemas following the process described in T259163:
(*) Migration for NewcomerTask had already started on T259163 before this task was created. Some patches for it are assigned to that task.
See also: https://wikitech.wikimedia.org/wiki/Event_Platform/EventLogging_legacy
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T185233 Modern Event Platform | |||
| Open | Ottomata | T238230 Decommission EventLogging backend components by migrating to MEP | |||
| Open | jlinehan | T259734 BUOD-KR1-Q4+: Certify that analytics schema and instruments have been upgraded to use the MEP system (clearing the legacy system for sunsetting) | |||
| Open | Ottomata | T259163 Migrate legacy metawiki schemas to Event Platform | |||
| Resolved | mforns | T267333 Migrate Growth EventLogging schemas to Event Platform | |||
| Resolved | Mholloway | T253121 MEP Client MediaWiki PHP | |||
| Resolved | Ottomata | T268517 Migrate Anti-Harassment EventLogging schemas to Event Platform | |||
| Duplicate | None | T267350 SpecialMuteSubmit Event Platform Migration | |||
| Duplicate | None | T267349 SpecialInvestigate Event Platform Migration | |||
| Duplicate | None | T267341 CookieBlock Event Platform Migration | |||
| Duplicate | None | T267340 AutoblockIpBlock Event Platform Migration | |||
| Resolved | Mholloway | T272863 EventLogging PHP EventServiceClient should use EventBus->send(). |
Also, @nettrom_WMF, can you confirm whether we should migrate all those at the exact same time, or just migrate them close enough? This is in reference to @Ottomata's comment in the email thread.
The team has three other schemas in use that are related to NewcomerTask (HomepageVisit, HomepageModule, and HelpPanel). I think all of these should migrate together since they're frequently used together in reporting. It would be convenient to track all of them in a single phab task. I can create that as a subtask unless there are good reasons not to.
We can do that! I hadn't created any Phab tasks because 1. The data will not change for users and 2. I didn't want to make so many tasks! But we can indeed for these, and we can indeed switch them together. Do you mean you would like us to flip the switch for these all at the exact same time, or just that they should be migrated as a group close together. We can do either!
Thanks!
Change 639611 had a related patch set uploaded (by Mforns; owner: Mforns):
[schemas/event/secondary@master] Add HomepageVisit schema to analytics legacy
Thanks for looping in that part of the conversation, @mforns! The main usage of these schemas is @MMiller_WMF's weekly reporting, which if I remember correctly he does some time on Mondays (Pacific time). So I don't think it's necessary to migrate them all at the same time, but within the same week. That way, we can make one set of updates to Marshall's notebook (reusing T255517 for it) and get the next week's reports going.
Thanks for the clarification @nettrom_WMF, will work on that once we agree on T240460.
Hiya @nettrom_WMF, we'd like to migrate these schemas during the week of Dec 7 - Dec 11. Let us know if you have questions or objections! :)
EditorJourney is currently inactive
Let's not migrate this then. We can always do so if you want to turn it back on in the future. I'm marking to Deprecate in the EventLogging Schema Audit spreadsheet, FYI @kostajh.
@nettrom_WMF I may have already asked you this elsewhere, but I'll ask again here so we have an officially documented answer.
Do any of these event streams need client IP and/or geocoded data? If not, it will be removed as part of this migration.
It was mentioned on Slack, but it's good to have it documented here and so I appreciate you bringing it up again! Neither of these streams need client IP or geocoded data, removing that is good.
I'm following up on the proposed migration window with the Growth team. I don't expect any issues with it, but will return with concerns if there are any.
I've run that plan by the team and there has been no objections, you can go ahead with that scheduled migration!
Change 647374 had a related patch set uploaded (by Mforns; owner: Mforns):
[schemas/event/secondary@master] Add HelpPanel schema to analytics legacy
Change 647377 had a related patch set uploaded (by Mforns; owner: Mforns):
[schemas/event/secondary@master] Add ServerSideAccountCreation to analytics legacy
Change 647382 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/mediawiki-config@master] Migrate HelpPanel schema from EventLogging to EventGate
Change 647383 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/mediawiki-config@master] Migrate HomepageModule schema from EventLogging to EventGate
Change 647384 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/mediawiki-config@master] Migrate HomepageVisit schema from EventLogging to EventGate
Change 647386 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/mediawiki-config@master] Migrate ServerSideAccountCreation schema from EventLogging to EventGate
Change 639611 merged by Mforns:
[schemas/event/secondary@master] Add HomepageVisit schema to analytics legacy
Change 647374 merged by Mforns:
[schemas/event/secondary@master] Add HelpPanel schema to analytics legacy
Change 647377 abandoned by Mforns:
[schemas/event/secondary@master] Add ServerSideAccountCreation to analytics legacy
Reason:
Given we have to wait for changes in the PHP client, which will be paused for a bit, I'm abandoning this change to not cause any confusion. Let's recreate later on.
Change 647386 abandoned by Mforns:
[operations/mediawiki-config@master] Migrate ServerSideAccountCreation schema from EventLogging to EventGate
Reason:
Given that this schema depends on the PHP client, and its development will be paused for a bit, I will abandon to avoid confusion and open patches. Let's recreate later on.
Change 647382 abandoned by Mforns:
[operations/mediawiki-config@master] Migrate HelpPanel schema from EventLogging to EventGate
Reason:
Abandoning to merge this and other two patches into the same change.
Change 647383 abandoned by Mforns:
[operations/mediawiki-config@master] Migrate HomepageModule schema from EventLogging to EventGate
Reason:
Abandoning to merge this and other two patches into the same change.
Change 647384 abandoned by Mforns:
[operations/mediawiki-config@master] Migrate HomepageVisit schema from EventLogging to EventGate
Reason:
Abandoning to merge this and other two patches into the same change.
Change 647782 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/mediawiki-config@master] Migrate Growth schemas from EventLogging to EventGate on testwiki
Change 647782 merged by jenkins-bot:
[operations/mediawiki-config@master] Migrate Growth schemas from EventLogging to EventGate on testwiki
Mentioned in SAL (#wikimedia-operations) [2020-12-10T20:54:35Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Migrate Growth EventLogging schemas to Event Platform on testwiki - T267333 (duration: 01m 03s)
Change 647811 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/mediawiki-config@master] Migrate EventLogging Growth schemas to EventGate on all wikis
Change 639539 abandoned by Mforns:
[operations/mediawiki-config@master] Migrate EventLogging NewcomerTask to EventGate on all wikis
Reason:
Abandoning in favor of https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /647811
Change 647811 merged by Ottomata:
[operations/mediawiki-config@master] Migrate EventLogging Growth schemas to EventGate on all wikis
Mentioned in SAL (#wikimedia-operations) [2020-12-10T23:01:25Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Migrate Growth EventLogging schemas to Event Platform on all wikis - T267333 (duration: 01m 09s)
Change 647817 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/puppet@production] Refine Growth schemas using eventlogging_legacy job
Change 647817 merged by Ottomata:
[operations/puppet@production] Refine Growth schemas using eventlogging_legacy job
Mentioned in SAL (#wikimedia-analytics) [2020-12-11T19:30:27Z] <ottomata> now ingesting Growth EventLogging schemas using event platform refine job; they are exclude-listed from eventlogging-processor. - T267333
Change 648334 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/puppet@production] Do not refine HomepageVisit using eventlogging_legacy job
Change 648334 merged by Ottomata:
[operations/puppet@production] Do not refine HomepageVisit using eventlogging_legacy job
Change 648336 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/mediawiki-config@master] Remove HomepageVisit from wgEventLoggingSchemas
Change 648336 merged by Ottomata:
[operations/mediawiki-config@master] Remove HomepageVisit from wgEventLoggingSchemas
Mentioned in SAL (#wikimedia-operations) [2020-12-11T20:11:05Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Un-migrtate Growth EventLogging schema HomepageVisit back to EventLogging-backend on all wikis (this is a server side event which is not yet ready to migrate) - T267333 (duration: 00m 58s)
Change 648337 had a related patch set uploaded (by Mforns; owner: Mforns):
[mediawiki/extensions/GrowthExperiments@master] Set EventLogging schemas to Event Platform schema URI
Hey all!
The following schemas have been migrated successfully.
The other schemas have not been migrated, because they depend on server-side client changes that are paused. We'll work on these whenever possible.
The following schemas have been migrated successfully. [...]
Thank you!
The other schemas have not been migrated, because they depend on server-side client changes that are paused. We'll work on these whenever possible.
Is there a task or more information about the pause on the server-side client changes?
Hey all!
Given that T253121: MEP Client MediaWiki PHP is now resolved, I will go ahead and finish the migration of the 2 remaining schemas that depended on the last changes of the PHP client.
Will comment here once the migration is done.
Cheers!
Change 655715 had a related patch set uploaded (by Mforns; owner: Mforns):
[schemas/event/secondary@master] Modify analytics/legacy/homepagevisit to account for changes on metawiki
Change 655720 had a related patch set uploaded (by Mforns; owner: Mforns):
[schemas/event/secondary@master] Add ServerSideAccountCreation to analytics/legacy
Change 655720 merged by Mforns:
[schemas/event/secondary@master] Add ServerSideAccountCreation to analytics/legacy
Change 655715 merged by Mforns:
[schemas/event/secondary@master] Modify analytics/legacy/homepagevisit to account for changes on metawiki
Change 655723 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/mediawiki-config@master] Migrate HomepageVisit and ServerSideAccountCreation to Event Platform on testwiki
Mentioned in SAL (#wikimedia-operations) [2021-01-12T19:25:03Z] <ottomata> rolling restart of eventgate-analytics-external pods to clear schema caches - T267333
Change 655723 merged by jenkins-bot:
[operations/mediawiki-config@master] Migrate HomepageVisit and ServerSideAccountCreation to Event Platform on testwiki
Mentioned in SAL (#wikimedia-operations) [2021-01-12T19:48:59Z] <tgr_> synced Config: [[gerrit:655723|Migrate HomepageVisit and ServerSideAccountCreation to Event Platform on testwiki (T267333)]]
Change 655753 had a related patch set uploaded (by Mforns; owner: Mforns):
[schemas/event/secondary@master] Add maxLength to formatted field in analytics/legacy/serversideaccountcreation
Change 655753 abandoned by Mforns:
[schemas/event/secondary@master] Add maxLength to formatted field in analytics/legacy/serversideaccountcreation
Reason:
This idea did not work, trying something else.
Change 655756 had a related patch set uploaded (by Mforns; owner: Mforns):
[schemas/event/secondary@master] Add maxLength to formatted field in analytics/legacy/serversideaccountcreation
Change 655756 merged by Mforns:
[schemas/event/secondary@master] Add maxLength to formatted field in analytics/legacy/serversideaccountcreation
FYI, I don't know if these are new, but there are a fair number of validation errors for HomepageModule:
'.event.module' should be equal to one of the allowed values, '.event.state' should be equal to one of the allowed values
hi @mforns, is there more info about why this is paused? I need to update HomepageVisit schema but the page on metawiki is locked https://meta.wikimedia.org/w/index.php?title=Schema:HomepageVisit, and according to operations/mediawiki-config only testwiki is using the schemas from https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/legacy
Change 666646 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Migrate HomepageVisit and ServerSideAccountCreation EL streams to all wikis
Change 666646 merged by Ottomata:
[operations/mediawiki-config@master] Migrate HomepageVisit and ServerSideAccountCreation EL streams to all wikis
Mentioned in SAL (#wikimedia-operations) [2021-02-24T15:06:23Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Migrate HomepageVisit and ServerSideAccountCreation EL streams to all wikis - T267333 (duration: 01m 05s)
@kostajh it looks good from here, I'm going to make a patch to the relevant extension.json files to finalize the change. When that is done and deployed we can remove the mediawiki-config override in wgEventLoggingSchemas. In the meantime, if you want to bump schema versions, you should make the config change in both places. Lemme know if that doesn't make sense.
Change 666651 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/GrowthExperiments@master] Finalize event platform migration
Change 666655 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/Campaigns@master] ServerSideAccountCreation - Finalize event platform migration
Change 666730 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Finalize Growth schema ingestion migration to Event Platform
Change 666651 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Finalize event platform migration
Change 666655 merged by jenkins-bot:
[mediawiki/extensions/Campaigns@master] ServerSideAccountCreation - Finalize event platform migration
Change 666730 merged by Ottomata:
[operations/puppet@production] Finalize Growth schema ingestion migration to Event Platform
Change 668764 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Remove overrides for EL migration for Growth team schemas
Change 668764 merged by Ottomata:
[operations/mediawiki-config@master] Remove overrides for EL migration for Growth team schemas
Mentioned in SAL (#wikimedia-operations) [2021-03-08T15:02:50Z] <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Remove wgEventLoggingSchemas overrides for Growth and WMDE Tech wishes schemas - T267333, etc. (duration: 00m 59s)
Change 648337 abandoned by Kosta Harlan:
[mediawiki/extensions/GrowthExperiments@master] Set EventLogging schemas to Event Platform schema URI
Reason:
Implemented in another patch