Eventlogging: Add infrastructure for measuring readers reading habits
Closed, ResolvedPublic3 Story Points

Description

We want to measure a few things related to our future perf improvements/impact. The purpose is

  1. Identify what % of our users have service worker support
  2. Identify what % of ours users read beyond the lead section
  3. Which % of users read beyond the lead section and has service worker support

Setup an EventLogging schema in stable on a 1% sample of our users in stable that:

  • Logs Service Worker availability on each event
  • Each event records the number of top level sections available to read/expand
  • Logs an event for entering experiment
  • Logs an event on opening a section
  • Do not impact first paint in this change
Jdlrobson created this task.Oct 5 2015, 3:50 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson raised the priority of this task from to Needs Triage.
Jdlrobson moved this task to Sprint 58: 6 on the Reading-Web-Planning board.
Jdlrobson added a subscriber: Jdlrobson.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2015, 3:50 PM

@JKatzWMF may have some additional questions to add here.

@Jdlrobson wouldn't getting the user-agent data from Analytics give us info about Service Worker && more?

How do we know how to bucket the EL to not break EL servers?

Possibly could use user agent but I'd rather make this easier to filter by given its importance in answering the question of do people that read post lead have service worker. Going forward we can use it to identify when we have critical mass.

I picked 2% sample arbitrarily. I'll sync up with analytics to determine what's doable we can always go lower.

Jdlrobson updated the task description. (Show Details)Oct 5 2015, 6:13 PM
Jdlrobson set Security to None.
Tbayer added a subscriber: Tbayer.Oct 5 2015, 9:29 PM
Jdlrobson renamed this task from Measure readers reading habits to Eventlogging: Add infrastructure for measuring readers reading habits.Oct 9 2015, 7:10 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson edited a custom field.
phuedx added a subscriber: phuedx.Oct 13 2015, 5:26 PM

How reliable would the request logs be for gathering #1?

Tbayer added a comment.EditedOct 14 2015, 12:55 AM

...

How do we know how to bucket the EL to not break EL servers?

Is this still a concern now that Eventlogging has moved to Kafka (T102225)?

I know this is coming late (@Tbayer and I met yesterday evening to discuss), but @phuedx @Jdlrobson if you could measure the following, it would add a lot of extra value as @Tbayer and I evaluate the collapsed section paradigm as a whole:

Total number of sections on the page
"title" of section
section impression (how many times did a section appear) - if we have 500M section opens, is that good or bad? we need to know what % were opened or closed.

Change 246437 had a related patch set uploaded (by Jdlrobson):
Introduce SchemaMobileWebSectionUsage

https://gerrit.wikimedia.org/r/246437

So this requires a bunch of changes already. @JKatzWMF I will be finishing up the changes we committed to in kick off. Please raise these new requirements in another phabricator card outside the sprint. They should be trivial but I can't commit to getting those done during the next two weeks.

@phuedx @Jhernandez @bmansurov I took a first pass at this and it's going to require a few changes

I will look at this!

244196 is C:-1. I'll look at 246437 tomorrow morning.

@Jdlrobson, a couple of minor suggestions:

  1. sectionCount should make it clear whether the lead section is included in counting sections. Or is just just collapsible sections. Are the indexes 1-based or 0-based?
  2. eventName has a wrong description.

@Jdlrobson, a couple of minor suggestions:

  1. sectionCount should make it clear whether the lead section is included in counting sections. Or is just just collapsible sections. Are the indexes 1-based or 0-based?
  2. eventName has a wrong description.

Good advice! Fixed! Thanks for checking.

Jdlrobson triaged this task as High priority.

Bumping priority as we need this asap. Without it we cannot justify our work!

Change 246437 merged by jenkins-bot:
Introduce SchemaMobileWebSectionUsage

https://gerrit.wikimedia.org/r/246437

Jdlrobson closed this task as Resolved.Oct 28 2015, 9:03 PM

Data is coming in. Boom! Created T118041 to continue this work