Let's migrate these schemas (as confirmed in T281300):
See: https://wikitech.wikimedia.org/wiki/Event_Platform/EventLogging_legacy
STATUS:
Done.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Ottomata | T185233 Modern Event Platform | |||
| Open | Ottomata | T238230 Decommission EventLogging backend components by migrating to MEP | |||
| Declined | None | T259734 BUOD-KR1-Q4+: Certify that analytics schema and instruments have been upgraded to use the MEP system (clearing the legacy system for sunsetting) | |||
| Open | Ottomata | T259163 Migrate legacy metawiki schemas to Event Platform | |||
| Resolved | mforns | T282562 WMDEBanner* Event Platform Migration |
Hi WMDE folks!
I have 1 question regarding the migration of WMDEBanner* schemas:
With the old EventLogging system, the fields client_ip and geocoded_data were collected/prepared by default.
With the new Event Platform system, we only collect/prepare those if needed (privacy by design principle).
Do you need client_ip and/or geocoded_data for WMDEBanner* schemas?
If so, I will mark them to be collected/prepared when I perform the migration, and nothing is going to change.
Otherwise, I will perform a basic migration, and they won't be collected from then on.
Please, let me know what you prefer!
Thanks a lot
@mforns For the New Editors campaign analytics we do not need those two fields. Let's hear from the WMDE FUN team if they do.
Sorry, long wait for a short answer: No, the Fundraising team does not need the two fields, either.
Change 698798 had a related patch set uploaded (by Mforns; author: Mforns):
[schemas/event/secondary@master] Add wmdebannerevents schema to analytics/legacy
Change 698798 merged by jenkins-bot:
[schemas/event/secondary@master] Add wmdebannerevents schema to analytics/legacy
Change 698802 had a related patch set uploaded (by Mforns; author: Mforns):
[schemas/event/secondary@master] Add wmdebannerimpressions schema to analytics/legacy
Change 698802 merged by jenkins-bot:
[schemas/event/secondary@master] Add wmdebannerimpressions schema to analytics/legacy
Change 698804 had a related patch set uploaded (by Mforns; author: Mforns):
[schemas/event/secondary@master] Add wmdebannersizeissue schema to analytics/legacy
Change 698804 merged by jenkins-bot:
[schemas/event/secondary@master] Add wmdebannersizeissue schema to analytics/legacy
Change 698811 had a related patch set uploaded (by Mforns; author: Mforns):
[operations/mediawiki-config@master] Migrate WMDEBanner* schemas to EventPlatform on testwiki
Change 698811 merged by Ottomata:
[operations/mediawiki-config@master] Migrate WMDEBanner* schemas to EventPlatform on testwiki
Mentioned in SAL (#wikimedia-operations) [2021-06-09T13:59:33Z] <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Migrate WMDEBanner* schemas to EventPlatform on testwiki - T282562 (duration: 01m 08s)
Change 698996 had a related patch set uploaded (by Mforns; author: Mforns):
[operations/mediawiki-config@master] Migrate WMDEBanner* schemas to EventPlatform on all wikis
Change 698996 merged by Ottomata:
[operations/mediawiki-config@master] Migrate WMDEBanner* schemas to EventPlatform on all wikis
Mentioned in SAL (#wikimedia-operations) [2021-06-09T14:37:00Z] <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Migrate WMDEBanner* schemas to EventPlatform on all wikis - T282562 (duration: 01m 06s)
Change 699002 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] Finalize backend EP migration of 4 EL schemas
Mentioned in SAL (#wikimedia-analytics) [2021-06-10T14:07:20Z] <ottomata> altered event.wmdebannerevent event.eventRate field to change type from BIGINT to DOUBLE - T282562
FYI, WMDEBannerEvent had a badly interpreted field type in Hive from years ago. Its event.eventRate had been interpreted as an integer instead of a double (from a time LONG ago when we weren't able to use the JSONSchemas to import this data into Hive). As such, any data for this field that was coming in were being cast to an integer.
We just noticed this as we migrated this schema to Event Platform and started emitting example canary events with decimal values for this field, which caused an alert to fire when ingesting the data into Hive.
I fixed this by manually altering the Hive table to change event.eventRate to a double.
alter table `event.wmdebannerevents` change `event` `event` STRUCT<`bannerAction`: STRING, `bannerName`: STRING, `eventRate`: DOUBLE, `finalSlide`: BIGINT, `slidesShown`: BIGINT>;
Ingesting is succeeding now.
Huh, I'm pretty sure that we saw non-integer values in there. Anyway, probably that also applies to the two other events.
Change 699002 merged by Ottomata:
[operations/puppet@production] Finalize backend EP migration of 4 EL schemas
Change 701101 had a related patch set uploaded (by Ottomata; author: Ottomata):
[mediawiki/extensions/WikimediaEvents@master] Finalize WMDEBanner* schema migration to Event Platform
Change 701101 merged by Ottomata:
[mediawiki/extensions/WikimediaEvents@master] Finalize WMDEBanner* schema migration to Event Platform
Change 701103 had a related patch set uploaded (by Ottomata; author: Ottomata):
[mediawiki/extensions/WikimediaEvents@wmf/1.37.0-wmf.11] Finalize WMDEBanner* schema migration to Event Platform
Change 701103 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.37.0-wmf.11] Finalize WMDEBanner* schema migration to Event Platform
Mentioned in SAL (#wikimedia-operations) [2021-06-23T18:39:17Z] <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/WikimediaEvents/extension.json: 01f034b466ff7bdd274e18c9ad7cefe88245548d: Finalize WMDEBanner* schema migration to Event Platform (T282562) (duration: 01m 05s)
Change 706689 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/mediawiki-config@master] Finalize several EventLogging -> Event Platfom migrations
Change 706689 merged by jenkins-bot:
[operations/mediawiki-config@master] Finalize several EventLogging -> Event Platfom migrations
Mentioned in SAL (#wikimedia-operations) [2021-07-22T19:26:12Z] <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Finalize several EventLogging -> Event Platfom migrations - T282855 T238138 T282562 T271168 (duration: 00m 55s)