Eventlogging: Add infrastructure for measuring readers reading habits
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

We want to measure a few things related to our future perf improvements/impact. The purpose is

Identify what % of our users have service worker support
Identify what % of ours users read beyond the lead section
Which % of users read beyond the lead section and has service worker support

Setup an EventLogging schema in stable on a 1% sample of our users in stable that:

Logs Service Worker availability on each event
Each event records the number of top level sections available to read/expand
Logs an event for entering experiment
Logs an event on opening a section
Do not impact first paint in this change

Details

	Subject	Repo	Branch	Lines +/-
	Introduce SchemaMobileWebSectionUsage	mediawiki/extensions/MobileFrontend	master	+93 -3

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	Release	None	T84936 Release VisualEditor-MediaWiki as "1.0"
Open		None	T50429 [Epic] Support editing parts of a page in VisualEditor-MediaWiki
Open		None	T54365 Explore performance gains from progressive (JIT?) de-alienation in VisualEditor
Open		None	T174303 Copy-pasting linked ISBN numbers from view mode HTML into VisualEditor inserts wikitext links to Special:BookSources (it should turn them into magic links?)
Open	Feature	None	T54091 The read HTML should have hinting to allow full DOM copying (as opposed to just rich copying) from read mode into VE surfaces
Open		None	T55784 [EPIC] Use Parsoid HTML for all page views
Resolved		dr0ptp4kt	T114542 Next Generation Content Loading and Routing, in Practice
Duplicate		• Jhernandez	T104432 [EPIC]: Improve mobile site performance
Duplicate		dr0ptp4kt	T120341 [GOAL] Make Wikipedia more accessible to all connections with new fast API-driven web experience in mobile web beta
Declined		None	T125920 [EPIC] Future exciting reading web performance endeavours
Resolved		Jdlrobson	T113066 [GOAL] Make Wikipedia more accessible to 2G connections
Resolved		Jdlrobson	T114655 Eventlogging: Add infrastructure for measuring readers reading habits

Event Timeline

Jdlrobson created this task.Oct 5 2015, 3:50 PM

Jdlrobson raised the priority of this task from to Needs Triage.

Jdlrobson updated the task description. (Show Details)

Jdlrobson added a project: Reading-Web-Planning.

Jdlrobson moved this task to Sprint 58: 6 on the Reading-Web-Planning board.

Jdlrobson subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2015, 3:50 PM

Jdlrobson added a parent task: T113066: [GOAL] Make Wikipedia more accessible to 2G connections.Oct 5 2015, 3:50 PM

@JKatzWMF may have some additional questions to add here.

@Jdlrobson wouldn't getting the user-agent data from Analytics give us info about Service Worker && more?

How do we know how to bucket the EL to not break EL servers?

Possibly could use user agent but I'd rather make this easier to filter by given its importance in answering the question of do people that read post lead have service worker. Going forward we can use it to identify when we have critical mass.

I picked 2% sample arbitrarily. I'll sync up with analytics to determine what's doable we can always go lower.

Jdlrobson updated the task description. (Show Details)Oct 5 2015, 6:13 PM

Jdlrobson set Security to None.

• Tbayer subscribed.Oct 5 2015, 9:29 PM

• Moushira added a project: Readers-Community-Engagement.Oct 5 2015, 10:01 PM

• Moushira moved this task from ToDo to On The Radar on the Readers-Community-Engagement board.

Jdlrobson renamed this task from Measure readers reading habits to Eventlogging: Add infrastructure for measuring readers reading habits.Oct 9 2015, 7:10 PM

Jdlrobson edited projects, added reading-web-sprint-58-The-Sixth-Sense; removed Reading-Web-Planning.

Jdlrobson updated the task description. (Show Details)

Jdlrobson edited a custom field.

• KLans_WMF moved this task from Needs Analysis to To Do on the reading-web-sprint-58-The-Sixth-Sense board.Oct 13 2015, 5:00 PM

How reliable would the request logs be for gathering #1?

In T114655#1702472, @Jhernandez wrote:

...

How do we know how to bucket the EL to not break EL servers?

Is this still a concern now that Eventlogging has moved to Kafka (T102225)?

Jdlrobson moved this task from To Do to Doing on the reading-web-sprint-58-The-Sixth-Sense board.Oct 14 2015, 5:02 PM

Jdlrobson claimed this task.Oct 14 2015, 6:18 PM

Schema setup = https://meta.wikimedia.org/wiki/Schema:MobileWebSectionUsage

I know this is coming late (@Tbayer and I met yesterday evening to discuss), but @phuedx @Jdlrobson if you could measure the following, it would add a lot of extra value as @Tbayer and I evaluate the collapsed section paradigm as a whole:

Total number of sections on the page
"title" of section
section impression (how many times did a section appear) - if we have 500M section opens, is that good or bad? we need to know what % were opened or closed.

Change 246437 had a related patch set uploaded (by Jdlrobson):
Introduce SchemaMobileWebSectionUsage

https://gerrit.wikimedia.org/r/246437

gerritbot added a project: Patch-For-Review.Oct 14 2015, 11:22 PM

So this requires a bunch of changes already. @JKatzWMF I will be finishing up the changes we committed to in kick off. Please raise these new requirements in another phabricator card outside the sprint. They should be trivial but I can't commit to getting those done during the next two weeks.

@phuedx @Jhernandez @bmansurov I took a first pass at this and it's going to require a few changes

An easy way to override sampling (already done as a result of the second time search bug we hit recently - but someone needs to merge) - https://gerrit.wikimedia.org/r/244196
Turning toggle into a OOjs class. https://gerrit.wikimedia.org/r/246432 < @bmansurov I know you have a lot of experience here.
The change itself - which I seem to be having problems to get working. I can tell if this is my EventLogging setup or if I'm doing something wrong. @phuedx could you take a look you seem to know this stuff well... https://gerrit.wikimedia.org/r/246437

Jdlrobson moved this task from Doing to Code Review on the reading-web-sprint-58-The-Sixth-Sense board.Oct 14 2015, 11:54 PM

I will look at this!

244196 is C:-1. I'll look at 246437 tomorrow morning.

phuedx moved this task from Code Review to -1 (Needs More Work) on the reading-web-sprint-58-The-Sixth-Sense board.Oct 15 2015, 7:54 PM

Jdlrobson moved this task from -1 (Needs More Work) to Code Review on the reading-web-sprint-58-The-Sixth-Sense board.Oct 15 2015, 8:11 PM

Jdlrobson moved this task from Code Review to -1 (Needs More Work) on the reading-web-sprint-58-The-Sixth-Sense board.

Jdlrobson moved this task from -1 (Needs More Work) to Code Review on the reading-web-sprint-58-The-Sixth-Sense board.Oct 16 2015, 5:51 PM

In T114655#1726012, @Jdlrobson wrote:

Schema setup = https://meta.wikimedia.org/wiki/Schema:MobileWebSectionUsage

@Jdlrobson, a couple of minor suggestions:

sectionCount should make it clear whether the lead section is included in counting sections. Or is just just collapsible sections. Are the indexes 1-based or 0-based?
eventName has a wrong description.

• bmansurov moved this task from Code Review to -1 (Needs More Work) on the reading-web-sprint-58-The-Sixth-Sense board.Oct 26 2015, 8:07 AM

Jdlrobson moved this task from -1 (Needs More Work) to Code Review on the reading-web-sprint-58-The-Sixth-Sense board.Oct 26 2015, 4:38 PM

In T114655#1752397, @bmansurov wrote:

In T114655#1726012, @Jdlrobson wrote:

Schema setup = https://meta.wikimedia.org/wiki/Schema:MobileWebSectionUsage

@Jdlrobson, a couple of minor suggestions:

sectionCount should make it clear whether the lead section is included in counting sections. Or is just just collapsible sections. Are the indexes 1-based or 0-based?

eventName has a wrong description.