Page MenuHomePhabricator

Mobile Homepage: instrumentation
Closed, ResolvedPublic

Description

This task is about updating the homepage schemas and/or instrumentation code for mobile. Below is a list of places in which the mobile homepage differs from the desktop version such that we would want to measure differently. Apart from this list, we will want to measure everything in the same way as desktop.

  • Hovers: hovering with a mouse is not a concept that exists on mobile, and so we don't expect any "hover-in" or "hover-out" events.
  • Previews: the mobile version has previews for each module that the user taps to open the full version. We want to record that tap as an event along with the module that was tapped. Each preview has a "back" button, which we similarly need to record when clicked.
  • Tooltip in impact module: in T216220 we designed an icon that the user taps to open up a tooltip. We want to know if the user taps that icon and then again when they dismiss or close the tooltip.

Event Timeline

SBisson created this task.May 8 2019, 6:54 PM
MMiller_WMF updated the task description. (Show Details)May 30 2019, 12:40 AM

@nettrom_WMF and @Cntlsn -- could you please look at the list I've put in the description of where we would like measure differently for mobile? Please go ahead and edit it. Once we feel good about it, I'll assign to @nettrom_WMF for him to modify the measurement plan document.

@MMiller_WMF : The list in the description seems fine to me. I'm wondering about the need for capturing the user browsing behavior to their user and/or user talk page, though. We don't do that for desktop, so why capture that specifically for mobile? Is there a particular question we're interested in answering here? (In the measurement plan, we're mainly interested in how users get to the homepage, not how they leave it)

One of the questions I have for @SBisson and/or @kostajh is whether the referer_route, referer_namespace, and referer_action fields in the HomepageVisit schema will work on mobile? I'm not sure if there are any technical difficulties there, but I wanted to ask early since that came to my mind when I was looking through the schemas and questions we're trying to answer.

One of the questions I have for @SBisson and/or @kostajh is whether the referer_route, referer_namespace, and referer_action fields in the HomepageVisit schema will work on mobile? I'm not sure if there are any technical difficulties there, but I wanted to ask early since that came to my mind when I was looking through the schemas and questions we're trying to answer.

I think it can work just the same as on desktop but the referer_* fields are partially based on adding parameters (source, namespace) to some links. We would have to do that on mobile too.

@nettrom_WMF -- the reason I included the instrumentation for the tabs (even though we're not doing it on desktop) is because we're making this special effort to implement the tabs on mobile, and because Readers Web might be interested in it. But now that I'm thinking about it, we better skip it. We're not looking at it on desktop, and it will be confusing to have it for one platform and not the other. I'll change it in the description.

@nettrom_WMF -- could you make an amendment to the measurement plan to add the things we're instrumenting specifically on mobile?

This is now ready for development.

MMiller_WMF updated the task description. (Show Details)Jun 1 2019, 12:16 AM

@MMiller_WMF @nettrom_WMF
All sounds good to me. Apart from hovers, I think the biggest difference for the mobile version of the homepage would be to measure the interactions with the modules preview, so thanks for adding that!

Thinking out loud here, are we considering (and would it possible) to measure other kinds of gestures? I'm thinking of scrolls and swipes mostly.
Talking about scrolls, it would be interesting for instance to see if users actually scroll down the homepage to see the modules that sits below the fold, in case we don't see many interactions with them.
Same for the start module, where the userpage submodule (a successful one according to desktop measurements) actually sits below the fold in the mobile version.

Talking about swipes, I think it could be nice to have some data to measure our assumption that swiping between the homepage and module overlays is actually naturally embraced by users.

@nettrom_WMF -- the reason I included the instrumentation for the tabs (even though we're not doing it on desktop) is because we're making this special effort to implement the tabs on mobile, and because Readers Web might be interested in it. But now that I'm thinking about it, we better skip it. We're not looking at it on desktop, and it will be confusing to have it for one platform and not the other. I'll change it in the description.

I can see that data on usage of those tabs can be useful to both us and Readers Web, as it helps us understand to what extent those are being used and thereby informs future design decisions. Not sure if it's our job to implement it, and if it happens I would probably want to see it done in its own schema as I don't see it tightly connected to the experiments we're doing with the homepage. And I agree that it's confusing to have this available just for mobile, so taking it off the table made sense. Thanks!

@nettrom_WMF -- could you make an amendment to the measurement plan to add the things we're instrumenting specifically on mobile?

Working on that, will hand a draft off to you when I have one available.

All sounds good to me. Apart from hovers, I think the biggest difference for the mobile version of the homepage would be to measure the interactions with the modules preview, so thanks for adding that!

Sure thing! :)

Thinking out loud here, are we considering (and would it possible) to measure other kinds of gestures? I'm thinking of scrolls and swipes mostly.
Talking about scrolls, it would be interesting for instance to see if users actually scroll down the homepage to see the modules that sits below the fold, in case we don't see many interactions with them.
Same for the start module, where the userpage submodule (a successful one according to desktop measurements) actually sits below the fold in the mobile version.

I'm not sure what the possibilities are when it comes to what we can measure, so I'll leave that to the engineers to comment on. My approach to adding or modifying instrumentation is largely focused on the team's goals (increasing activation and retention), the health of the product, and limiting the data we gather to what we need. In other words, I'd be concerned if we see low interactions with modules below the fold, meaning that sounds like a good candidate for a leading indicator of performance of the mobile homepage, but I wouldn't necessarily instrument that specifically to begin with.

nettrom_WMF updated the task description. (Show Details)Jun 3 2019, 10:30 PM

@MMiller_WMF : I've drafted some suggested addition to the measurement specification, could you take a look at those and let me know if further changes are needed? Also, I updated this task's description so it mentions that we'd also like to capture clicks on the "back" buttons in the modules on the mobile site, because that kind of interaction isn't available on the desktop site.

MMiller_WMF removed nettrom_WMF as the assignee of this task.Jun 5 2019, 4:14 PM

This is ready for development.

@nettrom_WMF -- thanks for noticing that we also want to capture "back" buttons. I think your additions to the measurement plan look good, and you can incorporate them and update the change log.

Hovers: hovering with a mouse is not a concept that exists on mobile, and so we don't expect any "hover-in" or "hover-out" events.

Right

Previews: the mobile version has previews for each module that the user taps to open the full version. We want to record that tap as an event along with the module that was tapped. Each preview has a "back" button, which we similarly need to record when clicked.

I think we can add information to the "impression" action of HomepageModule to specify whether the impression is

  • "desktop" (like it is now)
  • "mobile summary" (the shorter preview shown on the mobile homepage)
  • "mobile details" (the full mobile version, shown in a subpage currently but will be in a mobile overlay when T222833 is done)

Do you want to see the back button as a new action type in the same schema, or maybe a 'link-click' with link-id=back-button?

Tooltip in impact module: in T216220 we designed an icon that the user taps to open up a tooltip. We want to know if the user taps that icon and then again when they dismiss or close the tooltip.

I think what I'm proposing here T216220#5238107 would cover this point.

I think we can add information to the "impression" action of HomepageModule to specify whether the impression is

  • "desktop" (like it is now)
  • "mobile summary" (the shorter preview shown on the mobile homepage)
  • "mobile details" (the full mobile version, shown in a subpage currently but will be in a mobile overlay when T222833 is done)

I think having these as impression events makes sense. While the user has to take some action to see the detailed view on mobile, they're still getting an "impression" of that module, they're shown the same thing as they would see on desktop (if my understanding of the design is correct).

Because the detailed view on mobile is so similar to the desktop one, should we just keep those the same and let the is_mobile field distinguish between them? Then we could add module names for the summary versions. Maybe just add "-summary" to the module names, and create a new "start-summary" for that one because there's not an equivalent on desktop?

I don't think I'd like to have too many module names to work with, so that's motivating me to try to reuse them where possible.

Let me know if that's a good suggestion or not, happy to iterate on it or switch to a different approach!

Do you want to see the back button as a new action type in the same schema, or maybe a 'link-click' with link-id=back-button?

I think having it as a new action type would be good (e.g. "close") because it's not really a link that's clicked. That would also keep it similar to the Help Panel, which makes it less confusing to work with.

I think we can add information to the "impression" action of HomepageModule to specify whether the impression is

  • "desktop" (like it is now)
  • "mobile summary" (the shorter preview shown on the mobile homepage)
  • "mobile details" (the full mobile version, shown in a subpage currently but will be in a mobile overlay when T222833 is done)

I think having these as impression events makes sense. While the user has to take some action to see the detailed view on mobile, they're still getting an "impression" of that module, they're shown the same thing as they would see on desktop (if my understanding of the design is correct).
Because the detailed view on mobile is so similar to the desktop one, should we just keep those the same and let the is_mobile field distinguish between them? Then we could add module names for the summary versions. Maybe just add "-summary" to the module names, and create a new "start-summary" for that one because there's not an equivalent on desktop?
I don't think I'd like to have too many module names to work with, so that's motivating me to try to reuse them where possible.

A lot depend on the module names in the code so I would prefer to not suffix them.

I was thinking of 2 options (both in the HomepageModule schema):

  1. Add a "mode" field that is always populated with the current mode (desktop, mobile summary, mobile details). This field would be redundant with is_mobile in some cases but would be easy to populate and probably easy to query for you.
  2. Add the mode in the action_data for the actions where it makes sense (probably just impression and link-click). This is a bit more work on our side than option 1 but it's something more to think about in the future as we add new actions (does this new action require the render mode or not?).

Let me know if that's a good suggestion or not, happy to iterate on it or switch to a different approach!

Do you want to see the back button as a new action type in the same schema, or maybe a 'link-click' with link-id=back-button?#

I think having it as a new action type would be good (e.g. "close") because it's not really a link that's clicked. That would also keep it similar to the Help Panel, which makes it less confusing to work with.

A new action works well for us.

A lot depend on the module names in the code so I would prefer to not suffix them.

That makes sense, let's scrap that idea.

I was thinking of 2 options (both in the HomepageModule schema):

  1. Add a "mode" field that is always populated with the current mode (desktop, mobile summary, mobile details). This field would be redundant with is_mobile in some cases but would be easy to populate and probably easy to query for you.
  2. Add the mode in the action_data for the actions where it makes sense (probably just impression and link-click). This is a bit more work on our side than option 1 but it's something more to think about in the future as we add new actions (does this new action require the render mode or not?).

I had to think about this for a bit, because I'm unsure whether what we're looking at is open/close actions, or preview/full view of impression events, and how that will work out in the long term when we add additional modules. For example, I can imagine a recent activity or task recommendation module having a preview also on desktop, and then open up to a larger display to provide more detailed information.

It seems to me that this boils down to a question of naming things, and accounting for alternative views. Instead of "open/close", we have "impression/close", and I think that's perfectly fine and easy enough to work with. Adding a "mode" field as you suggest gives us the ability to account for different types of views. As you mention, there's a bit of redundancy on desktop at the moment, but that's fine, and it's also something that might change in the future.

Unless I'm missing something, adding the "mode" field seems like a good solution to me, and should be straightforward to work with.

Change 515140 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/GrowthExperiments@master] Record 'mode' in HomepageModule schema

https://gerrit.wikimedia.org/r/515140

The patch above adds the "mode" field to the HomepageModule schema. It is ready for review.

Instrumentation of the back button will be done here later or as part of T222833: Mobile Homepage: Show module details in a mobile overlay

Change 515140 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Record 'mode' in HomepageModule schema

https://gerrit.wikimedia.org/r/515140

ovasileva moved this task from Needs triage to Triaged on the Mobile board.Jun 11 2019, 1:50 PM

Should this move to Needs PM Review since the blocker of this ticket is complete, should the portion that will be completed later and is considered a "nice to have" be broken out so this task can be represented as complete? @MMiller_WMF @SBisson

Until yesterday we only had the basic server-side version of the mobile homepage so the instrumentation was complete.

Now that we are opening the module details in a mobile overlay, we have to update the instrumentation code to account for that. I am working on it right now.

Change 517635 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/GrowthExperiments@master] Mobile homepage overlay instrumentation

https://gerrit.wikimedia.org/r/517635

Change 517635 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Mobile homepage overlay instrumentation

https://gerrit.wikimedia.org/r/517635

Etonkovidova added a comment.EditedJun 19 2019, 7:04 PM

Notes - results of testing on testwiki (iPhone 6s):

  • "hover-out" events associated with "link-click"
  • Navigating back to Homepage from Start module - clicking on an back arrow (the browser back button gives the same error):

EventLogging Validation: [HomepageModule] Value "start" for property "module" is not one of ["start-account","start-tutorial","start-userpage","start-email","impact","help","mentorship"]
Other modules seems to be fine.

Change 517925 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] HomepageModule: Use newer schema with start module name

https://gerrit.wikimedia.org/r/517925

Change 517929 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] HomepageModule: Disable hover logging for mobile modes

https://gerrit.wikimedia.org/r/517929

Change 517925 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] HomepageModule: Use newer schema with start module name

https://gerrit.wikimedia.org/r/517925

Change 518047 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@wmf/1.34.0-wmf.10] HomepageModule: Use newer schema with start module name

https://gerrit.wikimedia.org/r/518047

Change 517929 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] HomepageModule: Disable hover logging for mobile modes

https://gerrit.wikimedia.org/r/517929

Change 518047 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.34.0-wmf.10] HomepageModule: Use newer schema with start module name

https://gerrit.wikimedia.org/r/518047

Mentioned in SAL (#wikimedia-operations) [2019-06-20T18:47:06Z] <tgr@deploy1001> Synchronized php-1.34.0-wmf.10/extensions/GrowthExperiments/extension.json: SWAT: [[gerrit:518047|HomepageModule: Use newer schema with start module name (Bug: T222836)]] (duration: 00m 58s)

Hey @Etonkovidova and @Cntlsn can this go into Marshall's column?

@kostajh - the commit message says that "the schema did not accept "start" as a valid value", and event-module= 'start' is recorded as rexently as 20190704000148. Schema:HomepageModule lists start as a valid value for module. So the fix was to make start a valid value for HomepageMolule schema?

So the fix was to make start a valid value for HomepageMolule schema?

Yes, that's right.

@Etonkovidova how come this is in progress? Is there anything else to do for this task?

@Etonkovidova how come this is in progress? Is there anything else to do for this task?

It's now up to @nettrom_WMF to mark to see whether there is something to be done for this task. The task was in QA column only for preliminary evaluation in betal cluster whether instrumentation is working.

nettrom_WMF closed this task as Resolved.Aug 14 2019, 4:21 PM
nettrom_WMF triaged this task as Normal priority.
nettrom_WMF added a project: Product-Analytics.

With T226222 being closed, I see no reason to keep this open any longer. If a specific mobile instrumentation issue comes up, there'll be a separate task for it.