Description

See: https://wikitech.wikimedia.org/wiki/Event_Platform/EventLogging_legacy

We need to keep client_ip and geocoded data for this schema.

This migration will be different than others, as VirtualPageView does not use the EventLogging extension Javascript client to send events. This was done years ago to work around a Do Not Track restriction in EventLogging. This restriction no longer exists, so it should be possible to modify the code in the Popups extension to call mw.eventLog.logEvent instead of sendBeacon directly.

STATUS:
Done.

Details

Subject	Repo	Branch	Lines +/-
Finalize several EventLogging -> Event Platfom migrations	operations/mediawiki-config	master	+0 -7
Set default VirtualPageView version to Event Platform schema URI	mediawiki/extensions/Popups	master	+1 -1
Refine VirtualPageView schema using eventlogging_legacy job	operations/puppet	production	+2 -0
Migrate VirtualPageView to EventPlatform on all wikis	operations/mediawiki-config	master	+0 -5
Migrate VirtualPageView to EventPlatform on group 0 and 1 wikis	operations/mediawiki-config	master	+6 -1
Migrate VirtualPageView to EventPlatform on testwiki	operations/mediawiki-config	master	+11 -0
Add VirtualPageView schema to analytics/legacy	schemas/event/secondary	master	+499 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Ottomata	T185233 Modern Event Platform
Open	Ottomata	T238230 Decommission EventLogging backend components by migrating to MEP
Declined	None	T259734 BUOD-KR1-Q4+: Certify that analytics schema and instruments have been upgraded to use the MEP system (clearing the legacy system for sunsetting)
Open	Ottomata	T259163 Migrate legacy metawiki schemas to Event Platform
Open	None	T228175 Event Platform Client Libraries
Resolved	mforns	T238138 VirtualPageView Event Platform Migration
Resolved	ovasileva	T279382 VirtualPageView should use EventLogging api to send virtual page view events

Event Timeline

Helpful notes on etherpad: https://etherpad.wikimedia.org/p/event-platform

Ottomata mentioned this in T238230: Decommission EventLogging backend components by migrating to MEP.Nov 13 2019, 4:37 PM

Ottomata added a parent task: T238230: Decommission EventLogging backend components by migrating to MEP.

Ottomata mentioned this in T185233: Modern Event Platform.Nov 13 2019, 4:40 PM

Ottomata triaged this task as Medium priority.Nov 14 2019, 6:26 PM

Ottomata moved this task from Incoming to Event Platform on the Analytics board.

• jlinehan moved this task from Inbox to Sign-off on the Better Use Of Data board.Nov 19 2019, 5:26 PM

• jlinehan moved this task from Sign-off to Done! on the Better Use Of Data board.Nov 19 2019, 5:34 PM

LGoto moved this task from Needs triage to Epics on the Product-Infrastructure-Team-Backlog-Deprecated board.Nov 20 2019, 4:50 PM

Ottomata moved this task from Backlog to Estimated/ Discussed on the Event-Platform board.Jan 23 2020, 3:19 PM

Ottomata mentioned this in T223643: Create EventLogging service.Mar 26 2020, 8:06 PM

Oof. I think VirtualPageView is a bad choice for our first port.
https://phabricator.wikimedia.org/T187277

The Popups extension doesn't use the eventLog.logEvent interface; they built their own and call sendBeacond directly to get around EventLogging's DNT check.

@Ottomata ae you thinking next quarter we will do teh instraumentation of session length but also an additional port of another event?

Not to necessarily port another event, but at least prepare to do so. If things turn out to be easy, then sure, we can port.

We can't do VirtualPageView because they totally bypass EventLogging. Instead, I'll work on T249261: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform.

Sounds good, added the searchsatisfaction ticket to goals.

Ottomata moved this task from Estimated/ Discussed to Backlog on the Event-Platform board.Apr 13 2020, 8:36 PM

Ottomata edited parent tasks, added: T259163: Migrate legacy metawiki schemas to Event Platform; removed: T238230: Decommission EventLogging backend components by migrating to MEP.Aug 17 2020, 5:36 PM

• sdkim moved this task from Done! to Inbox on the Better Use Of Data board.Oct 29 2020, 4:39 PM

@sdkim @Ottomata @jlinehan When we migrate this, we need to make sure we include geocoded data. It's important for our high-level metrics (this is included in content interactions, and we look at this by geography).

Also looping in @cchen since she's using this data

@kzimmerman acknowledged. Last speaking to @Ottomata, this would require it's own migration plan and has been slotted at the end of our migration schedule

When we migrate this, we need to make sure we include geocoded data

Great, thanks for the heads up. Can do. :)

Ottomata renamed this task from Vertical: Virtualpageview datastream on MEP to VirtualPageView Event Platform Migration.Jan 28 2021, 4:26 PM

Ottomata reassigned this task from • jlinehan to mforns.

Ottomata raised the priority of this task from Medium to High.

Ottomata added a project: Analytics-Kanban.

Ottomata updated the task description. (Show Details)

Ottomata mentioned this in T259163: Migrate legacy metawiki schemas to Event Platform.

Ottomata added a subscriber: Jdlrobson.

• Mholloway subscribed.Feb 8 2021, 7:35 PM

LGoto moved this task from Inbox to To Do on the Better Use Of Data board.Feb 8 2021, 7:35 PM

FYI: the KaiOS-Wikipedia-app is also logging virtual page views so let us know when we should start sending those to event gate.

Oh! Cool. Q: are there any upcoming schema modificiations? If not, we can migrate the schema ASAP, which would enable KaiOS to produce VirtualPageVieww events to EventGate now. If we do that, we'd like to edit protect the metawiki schema, so we don't have to synchronize changes between the systems.

Hi all, FYI, I'd like to migrate this schema in Q4.

Ottomata mentioned this in T279382: VirtualPageView should use EventLogging api to send virtual page view events.Apr 5 2021, 8:06 PM

mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.Apr 14 2021, 3:24 PM

mforns moved this task from To Do to Doing on the Better Use Of Data board.Apr 21 2021, 6:42 PM

kzimmerman added a project: Product-Analytics.Apr 21 2021, 6:43 PM

ldelench_wmf moved this task from Triage to Tracking on the Product-Analytics board.Apr 27 2021, 5:18 PM

Ottomata moved this task from Backlog to In Progress Before Value Streams Kickoff (August 15th) on the Event-Platform board.Apr 30 2021, 8:02 PM

Change 685919 had a related patch set uploaded (by Mforns; author: Mforns):

[schemas/event/secondary@master] Add VirtualPageView schema to analytics/legacy

https://gerrit.wikimedia.org/r/685919

Change 685919 merged by jenkins-bot:

[schemas/event/secondary@master] Add VirtualPageView schema to analytics/legacy

https://gerrit.wikimedia.org/r/685919

Change 685928 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/mediawiki-config@master] Migrate VirtualPageView to EventPlatform on testwiki

https://gerrit.wikimedia.org/r/685928

mforns updated the task description. (Show Details)May 6 2021, 9:07 PM

Change 685928 merged by jenkins-bot:

[operations/mediawiki-config@master] Migrate VirtualPageView to EventPlatform on testwiki

https://gerrit.wikimedia.org/r/685928

Mentioned in SAL (#wikimedia-operations) [2021-05-11T18:53:14Z] <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Migrate VirtualPageView to EventPlatform on testwiki - T238138 (duration: 01m 09s)

Change 689205 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/mediawiki-config@master] Migrate VirtualPageView to EventPlatform on all wikis

https://gerrit.wikimedia.org/r/689205

cchen mentioned this in T282710: Missing data in virtualpageview_hourly table since April 15, 2021.May 12 2021, 6:04 PM

Change 689205 merged by Ottomata:

[operations/mediawiki-config@master] Migrate VirtualPageView to EventPlatform on group 0 and 1 wikis

https://gerrit.wikimedia.org/r/689205

Mentioned in SAL (#wikimedia-operations) [2021-05-17T19:13:34Z] <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Migrate VirtualPageView to Event Platform on group 0 and group 1 - T238138 (duration: 00m 59s)

Maintenance_bot removed a project: Patch-For-Review.May 17 2021, 8:10 PM

Change 692611 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/mediawiki-config@master] Migrate VirtualPageView to EventPlatform on all wikis

https://gerrit.wikimedia.org/r/692611

gerritbot added a project: Patch-For-Review.May 18 2021, 2:15 PM

Change 692611 merged by Ottomata:

[operations/mediawiki-config@master] Migrate VirtualPageView to EventPlatform on all wikis

https://gerrit.wikimedia.org/r/692611

Mentioned in SAL (#wikimedia-operations) [2021-05-18T14:38:55Z] <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Migrate VirtualPageView to EventPlatform on all wikis - T238138 (duration: 01m 06s)

Maintenance_bot removed a project: Patch-For-Review.May 18 2021, 3:10 PM

phuedx closed subtask T279382: VirtualPageView should use EventLogging api to send virtual page view events as Resolved.May 24 2021, 12:14 PM

Change 693931 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/puppet@production] Refine VirtualPageView schema using eventlogging_legacy job

https://gerrit.wikimedia.org/r/693931

gerritbot added a project: Patch-For-Review.May 24 2021, 6:11 PM

Change 693931 merged by Ottomata:

[operations/puppet@production] Refine VirtualPageView schema using eventlogging_legacy job

https://gerrit.wikimedia.org/r/693931

Change 693934 had a related patch set uploaded (by Mforns; author: Mforns):

[mediawiki/extensions/Popups@master] Set default VirtualPageView version to Event Platform schema URI

https://gerrit.wikimedia.org/r/693934

mforns updated the task description. (Show Details)May 24 2021, 6:36 PM

Change 693934 merged by jenkins-bot:

[mediawiki/extensions/Popups@master] Set default VirtualPageView version to Event Platform schema URI

https://gerrit.wikimedia.org/r/693934

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.11; 2021-06-21).Jun 8 2021, 10:00 AM

Ottomata moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.Jun 9 2021, 4:15 PM

Change 706689 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] Finalize several EventLogging -> Event Platfom migrations

https://gerrit.wikimedia.org/r/706689

Change 706689 merged by jenkins-bot:

[operations/mediawiki-config@master] Finalize several EventLogging -> Event Platfom migrations

https://gerrit.wikimedia.org/r/706689

Mentioned in SAL (#wikimedia-operations) [2021-07-22T19:26:12Z] <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Finalize several EventLogging -> Event Platfom migrations - T282855 T238138 T282562 T271168 (duration: 00m 55s)