Page MenuHomePhabricator

Add event logging for eligibility, impression, and tap through for Quick Surveys funnel
Closed, ResolvedPublic5 Estimated Story Points

Description

This is a task to implement event logging that will allow high level aggregate funnel analysis for users eligible for quick surveys and shown quick surveys, and will allow finer grained access for users taking an intentional action to engage with quick survyes.

Staging it

  • Confirmed on localhost mobile
  • Confirmed on localhost desktop
  • Confirmed on beta labs mobile
  • Confirmed on beta labs desktop

Current schema: https://meta.wikimedia.org/wiki/Schema:QuickSurveysResponses

It's unclear if Yes / No Thanks taps for external surveys are actually being logged right now. That needs to be verified and should be implemented if it isn't being logged.

It's unclear if the presentation value is being correctly logged right now as well. That also needs to be verified and should be implemented if it isn't being logged. If it makes it easier for the instrumentation, concatenation of the form factor, a hyphen, and the skin is sufficient for the presentation field (as opposed to concatenating form factor, a hyphen, and one of "stable|beta|alpha|prototype"). As I understand it, for example 'minerva' and 'minerva-beta' are distinct things, so it will be obvious when it's stable versus beta on the mobile web.

For existing QuickSurveysResponses schema, add the following, and ensure that the button tap action is captured not only for on-wiki surveys but also when the user is navigating to an external survey:

namespaceId (int)
surveySessionToken (mediawiki.user.sessionid plus fixed suffix, in other words survives pages and in the rarest cases, tabs)
surveyInstanceToken (one time token per page with survey)

Define a new schema QuickSurveyInitiation and fire events for eligible and impression:

beaconCapable (sent for `eligible` event to kick off funnel)
surveySessionToken (mediawiki.user.sessionid plus fixed suffix, in other words survives pages and in the rarest cases, tabs)
surveyInstanceToken (one time token per page with survey)
surveyCodeName
eventName
   eligible (at the time eligibility is determined, sent if and only if user was eligible)
   impression (sent once, as is custom - when .5 of element shown)

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@leila, @ellery, we've put the proposed schema changes and the creation of a new eligible/impression schema into the Description.

By design, the eligible/impression schema is simple in order that we don't use extra bandwidth (and correspondingly don't collect the extended fields unless the user chooses to actually engage with the survey). It's sufficient to understand in aggregate whether there's a mismatch in eligibile : pages served ratios or impression : eligible ratios. For users who do choose to engage, the already available more sophisticated analysis is of course viable through the QuickSurveysResponses value.

Make sense?

@dr0ptp4kt thanks! :-)

General comments about the current version of Schema:QuickSurveysResponses (14136037):

  • We should be able to distinguish between Mobile Web and Desktop platforms. This is currently not captured via platform in Schema:QuickSurveysResponses.
  • If I remember correctly, we had wiki as part of the schema earlier. Can we bring that back (not sure if that's part of the event capsule)? If we run the survey concurrently in two languages, we need to know which response is from what language.

Comments related to your suggestions:
Ellery and I reviewed them. They are capturing what we discussed. thanks.

@leila, thanks! I'll update the description to ensure that the 'platform' value is correctly captured as well.

The wiki is part of the event capsule.

Change 265406 had a related patch set uploaded (by Bmansurov):
Update Schema:QuickSurveysResponses fields

https://gerrit.wikimedia.org/r/265406

Change 265422 had a related patch set uploaded (by Bmansurov):
Add support for logging to Schema:QuickSurveyInitiation

https://gerrit.wikimedia.org/r/265422

@dr0ptp4kt:

If it makes it easier for the instrumentation, concatenation of the form factor, a hyphen, and the skin is sufficient for the presentation field (as opposed to concatenating form factor, a hyphen, and one of "stable|beta|alpha|prototype").

and

As I understand it, for example 'minerva' and 'minerva-beta' are distinct things, so it will be obvious when it's stable versus beta on the mobile web.

seem to contradict each other. Did you mean we can skip form factor on mobile web (as described in the schema definition) and just use 'minerva' or 'minerva-beta'?

@bmansurov, I meant to indicate we should have values of the following nature:

small-minerva
big-minerva
small-minervabeta
big-minervabeta
small-vector
big-vector

And so on. Basically our sizing heuristic for the first part, a hyphen, then like in related articles EL, the skin in force for the user (the skin part is like the 'skin' field in https://meta.wikimedia.org/wiki/Schema:RelatedArticles, but note we're removing the hyphen from minerva-beta and just calling it minervabeta).

Side note: we're willing to accept that someone may have their phone in landscape orientation and it would fall into "big".

@dr0ptp4kt @bmansurov rather than re-inventing the wheel here I suggest we be consistent with what we've done elsewhere and have a boolean field "isTablet" and just record skin alongside that. No need for these 'big-' and 'small-' prefixes, they're unnecessary.

@dr0ptp4kt @bmansurov rather than re-inventing the wheel here I suggest we be consistent with what we've done elsewhere and have a boolean field "isTablet" and just record skin alongside that. No need for these 'big-' and 'small-' prefixes, they're unnecessary.

Works for me! Go for it.

@leila, to answer your question from IRC, we had a pairing session with Nathaniel on this task, but I don't think he's doing anything more.

thanks, @bmansurov for following up. :-)
Does this mean that your team will take care of this task completely?

Perfect. Thanks for confirming! :-)

Because clarity is a virtue: this task is the event logging part in the beginning of the funnel. That's more or less done (there's going to be an additional field in the event logging to let us differentiate tablet/desktop form factor from smaller form factor).

The part for @schana would be the tap through name-value parameter (reviewing Monday).

Thanks for clarifying that. I had lost track of that.

@bmansurov: I've fixed the two minor issues with 265422. Let me know if you're happy PS6 and I'll merge it.

@phuedx, good catch of the dependency. See my comment on the patch.

Change 265422 merged by jenkins-bot:
Add support for logging to Schema:QuickSurveyInitiation

https://gerrit.wikimedia.org/r/265422

Change 265406 merged by jenkins-bot:
Update Schema:QuickSurveysResponses fields

https://gerrit.wikimedia.org/r/265406

Anyone interested can also test it by visiting http://en.m.wikipedia.beta.wmflabs.org/wiki/1?quicksurvey=true and looking for the world 'event' among the request URLs in the console network tab.

Before closing the task we need to test if this is actually working correctly in production.

Anyone interested can also test it by visiting http://en.m.wikipedia.beta.wmflabs.org/wiki/1?quicksurvey=true and looking for the world 'event' among the request URLs in the console network tab.

Or filtering network requests by "Other", which, in all likelihood, will only contain requests to event.gif.

@bmansurov that's what's sign off is for - so moving back :)

@dr0ptp4kt if you're happy with the above and want to sign off and mark as resolved - i'd suggest raising a card for next sprint to test in production.

@dr0ptp4kt if you're happy with the above and want to sign off and mark as resolved - i'd suggest raising a card for next sprint to test in production.

Or this task could be moved into Reading-Web-Sprint-65-Game of Phones Ready for Signoff.

@bd808: Do we still get 5 XP if we do ^?

I'm fine with a separate (train) deployment task. That's how it should have been structured in the first place - my mistake. But, @bd808, what do you recommend?

Where the points are recognized, so to speak, shouldn't inhibit this shipping to stable. Please ship to stable.

Since I was asked twice I guess I'll chime in with "either way works for me". If the Team is interested in tracking velocity then I'd recommend the split task approach as the real work has been done and "releasing" is just waiting for the train to roll and following up with monitoring correct? Without a uniform Definition of Done and story grooming guidelines we end up in these choose your own adventure scenarios with a lot of tasks.

Okay, consider this accepted and I'll create the task to observe the outcome of the train.

Please ready it for the train if it hasn't been done so already.