Page MenuHomePhabricator

Set up preview counting for KaiOS app
Closed, ResolvedPublic

Description

The Inuka team is getting ready to deploy a Wikipedia app for KaiOS, and we need to make sure that its page previews are included in the virtualpageview data stream.

As far as I can tell, this is what needs to happen:

Event Timeline

nshahquinn-wmf moved this task from Triage to Tracking on the Product-Analytics board.

@Nuria is the list of steps in the description complete?

fdans added a subscriber: fdans.Feb 10 2020, 5:22 PM

@nshahquinn-wmf Speaking for the team: that approach sounds good!

@nshahquinn-wmf Speaking for the team: that approach sounds good!

Okay, thank you!

AMuigai moved this task from Backlog to Analyst on the Inuka-Team board.Feb 13 2020, 1:52 PM
nshahquinn-wmf moved this task from Analyst to Future on the Inuka-Team board.Mar 13 2020, 7:12 PM

Because the KaiOS app is an experimental project and the team is pushing to get the initial release ready as soon as possible, they have decided to defer the VirtualPageView instrumentation until later. The Android app has page previews but hasn't set up this instrumentation either, so this decision isn't unprecedented.

@fdans, @Nuria, I'm sorry that I made you do T246309 prematurely!

AMuigai moved this task from Future to Q2-2020 on the Inuka-Team board.Nov 9 2020, 12:13 PM
AMuigai moved this task from Q2-2020 to Needs design on the Inuka-Team board.Nov 18 2020, 11:25 AM
AMuigai moved this task from Needs design to Kanban on the Inuka-Team board.
AMuigai edited projects, added Inuka-Team (Kanban); removed Inuka-Team.
SBisson claimed this task.Nov 24 2020, 1:59 PM
SBisson triaged this task as Medium priority.
SBisson moved this task from Ready for Dev to Dev on the Inuka-Team (Kanban) board.
SBisson removed SBisson as the assignee of this task.Nov 26 2020, 6:29 PM
SBisson moved this task from Dev to Ready for Dev on the Inuka-Team (Kanban) board.
hueitan claimed this task.Nov 30 2020, 3:09 PM
hueitan moved this task from Ready for Dev to Dev on the Inuka-Team (Kanban) board.
SBisson moved this task from Code Review to QA on the Inuka-Team (Kanban) board.Dec 9 2020, 4:03 PM

The PR is merged

Jpita added a subscriber: Jpita.Dec 9 2020, 6:10 PM

can someone check that the metrics are being sent?
also, I can't move this to any other column .

can someone check that the metrics are being sent?
also, I can't move this to any other column .

@nshahquinn-wmf did a spot check a few weeks ago and the data was showing up. He should probably check again now that the final version has been merged.

Let's keep it in QA until this is successfully tested by Neil.

@nshahquinn-wmf did a spot check a few weeks ago and the data was showing up. He should probably check again now that the final version has been merged.

Let's keep it in QA until this is successfully tested by Neil.

Sounds good to me! I've created a subtask for me to do that.

As I saw earlier, the database has a small number of recent virtualpageview events with the access method as "mobile web", exactly as I'd expect from sporadic dev work:

year month     day  events
---- ----- ------- -------
2020	12	1	20
2020	12	2	1
2020	12	8	6
2020	12	9	4
2020	12	10	2
2020	12	11	10

However, I don't see any of these events showing up in the virtualpageviews_hourly data set in Turnilo. Since the events seem to be making it from the app to the database without a problem, I assume it's an issue in the ETL step; I'll follow up with Analytics on that.

I can't whether the user agent parsing is working correctly since all the events so far have come from computers rather than actual KaiOS devices. However, I expect that it will work fine since it shares logic with the pageviews stream, where we know the parsing is working correctly.

So, overall, this seems to be working correctly.

Jpita added a comment.Dec 13 2020, 9:11 PM

As I saw earlier, the database has a small number of recent virtualpageview events with the access method as "mobile web", exactly as I'd expect from sporadic dev work:

year month     day  events
---- ----- ------- -------
2020	12	1	20
2020	12	2	1
2020	12	8	6
2020	12	9	4
2020	12	10	2
2020	12	11	10

However, I don't see any of these events showing up in the virtualpageviews_hourly data set in Turnilo. Since the events seem to be making it from the app to the database without a problem, I assume it's an issue in the ETL step; I'll follow up with Analytics on that.

I can't whether the user agent parsing is working correctly since all the events so far have come from computers rather than actual KaiOS devices. However, I expect that it will work fine since it shares logic with the pageviews stream, where we know the parsing is working correctly.

So, overall, this seems to be working correctly.

If you need me to do some specific actions on a device please let me know.
I've been using the app a bit Thursday and Friday so there should be some events.

If you need me to do some specific actions on a device please let me know.
I've been using the app a bit Thursday and Friday so there should be some events.

Hmm, I checked again and I only see a single event that's actually from a KaiOS device (on last Thursday at 16:21 your time). Maybe there's something wrong with the the app after all...

I also checked again for events that failed validation and this time found two from last Tuesday that were missing the required source_title property, but those were both from a Windows computer, so I don't think it's related to this.

I also checked again for events that failed validation and this time found two from last Tuesday that were missing the required source_title property, but those were both from a Windows computer, so I don't think it's related to this.

I can confirm that, not related, this is from me when I'm doing some testing from my WIndows computer.

@Jpita you need to look at the preview for a full second for us to log the event. Make sure you do that.

Jpita added a comment.Dec 14 2020, 1:47 PM

@Jpita you need to look at the preview for a full second for us to log the event. Make sure you do that.

yeah I never do that, I open it, read the title and go to the article.
that's my MO on all similar things in the internet, not just for testing.
I'm not assuming everyone does it though.

yeah I never do that, I open it, read the title and go to the article.
that's my MO on all similar things in the internet, not just for testing.
I'm not assuming everyone does it though.

The 1 second rule is a cross-platform standard that we're adhering to, because this preview counting rolls up into our movement metric of content interactions.

Neil was able to see the events

AMuigai closed this task as Resolved.Dec 15 2020, 7:28 PM

One minor issue I noticed: because we can't alter the user agent for sendBeacon calls, we aren't able to set the wmf_app_version field. We are still able to identify previews from the app using access_method = "mobile app" and browser_family = "KaiOS", but we won't be able to drill down on specific versions.

If you use Event Platform, your client can explicitly set http.request_headers['user-agent'].

Oh, I see this is for VirtualPageView, which makes what I said more complicated. :)

Oh, I see this is for VirtualPageView, which makes what I said more complicated. :)

Because VirtualPageView hasn't been migrated to the Event Platform yet? Or because it has some special-case behavior?

It has special case behavior (it doesn't use the EventLogging extension to send the events), so it will be one of the last schemas migrated.

It may be possible to begin the migration and KaiOS to send the migrated events to EventGate, while maintaining the schema on metawiki and the old eventlogging backend. We'd have to lock edits to the metawiki schema, which means that the destkop usage could not be modified until we finish the migration. Probably not the best, but could be done.

It has special case behavior (it doesn't use the EventLogging extension to send the events), so it will be one of the last schemas migrated.

It may be possible to begin the migration and KaiOS to send the migrated events to EventGate, while maintaining the schema on metawiki and the old eventlogging backend. We'd have to lock edits to the metawiki schema, which means that the destkop usage could not be modified until we finish the migration. Probably not the best, but could be done.

Thanks for the details! There's definitely no need for complex work like that; as I said, this is a very minor issue 😊