Page MenuHomePhabricator

Instrument banner module on newcomer homepage
Closed, ResolvedPublic

Description

In T269804, we built a simple banner module that communities can configure. Depending on how communities use it, it may pull newcomers away from the homepage. We want to be able to monitor that, and so we would like to record when users click on links that are in the module.

Event Timeline

That reminds me, we need to add the module name to the HomepageModule schema for logging impressions.

Change 650031 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[schemas/event/secondary@master] analytics/legacy/homepagemodule: Add banner to module list

https://gerrit.wikimedia.org/r/650031

Change 650033 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Allow use of link containers on the homepage for link click logging

https://gerrit.wikimedia.org/r/650033

Change 650033 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Allow use of link containers on the homepage for link click logging

https://gerrit.wikimedia.org/r/650033

Moving to PM review just to inform that HomepageModule schema has been changed.

The page Schema:HomepageModule has not been updated yet.

"banner" was added to "module", so clicking on a link in the banner is recorded as following:

event.HomepageModule 
{action: "link-click", action_data: "linkId=banner", user_id: 6768, user_editcount: 709, user_variant: "D", …}
action: "link-click"
action_data: "linkId=banner"
homepage_pageview_token: "g0kg88961j97dp9uv37gj3rpfv9bir07"
is_mobile: false
mode: "desktop"
module: "banner"
state: "enabled"
user_editcount: 709
user_id: 6768
user_variant: "D"

The page Schema:HomepageModule has not been updated yet.

Should we try to keep metawiki pages up to date when they are not used by the software anymore? (See also T270136: Automate deprecation of schema on metawiki after migration to Event Platform for some related discussion.)

(That said, the real schema has not been updated yet either.)

It looks like https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/650033 has been deployed, as well as version 1.1.0 in https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/650031, however, the schema version still used is 1.0.0.

I think this happened because the Growth schemas are still being migrated in T267333: Migrate Growth EventLogging schemas to Event Platform, so the schema version URI is currently overridden in mediawiki/config.

This needs to be updated in extension.json and in mediawiki/config ASAP. Making patches.

Ping @mforns

OH! excuse me no, https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/650031 is not merged. So, it looks like your code is now updated to log conforming to the new schema, but the HomepageModule events are failing now validation because they are validating using the 1.0.0 schema.

https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-2021.01.13?id=oj4p_XYBsCn0xdb8X731

This is fine, those events didn't exist before the change to the logging code, and preexisting events should not be affected, so the schema errors just mean we aren't collecting banner module data yet. We didn't have a deadline for starting to collect it, so this is not any kind of emergency.

Change 650031 merged by Mholloway:
[schemas/event/secondary@master] analytics/legacy/homepagemodule: Add banner to module list

https://gerrit.wikimedia.org/r/650031

Thank you! I'm going to check out the data on Monday to see how many banner events have been coming in.

Still needs a config change.

@Tgr -- when will we do the config change?

Change 656284 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Update /analytics/legacy/homepagemodule/ schema version to 1.1.0

https://gerrit.wikimedia.org/r/656284

Scheduled it for the next backport window which is Tuesday 4PM PST.

Change 656284 merged by jenkins-bot:
[operations/mediawiki-config@master] Update /analytics/legacy/homepagemodule/ schema version to 1.1.0

https://gerrit.wikimedia.org/r/656284

Mentioned in SAL (#wikimedia-operations) [2021-01-20T00:43:29Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:656284|Update /analytics/legacy/homepagemodule/ schema version to 1.1.0 (T270309)]] (duration: 01m 03s)

Checked in arwiki - the events action_data: "linkId=banner" are recorded.

@Etonkovidova @Tgr -- I just checked for this data in the data lake, and I don't see any events where module = "banner". These are the only modules for which there are events:

  • generic
  • help
  • impact
  • mentorship
  • start-startediting
  • startemail
  • suggested-edits

What do you think is going on?

@MMiller_WMF - I re-checked arwiki wmf.27 - and the Console shows that client-side events are being recorded: impression for module: "banner" and when the banner link is clicked - "linkId=banner".

event.HomepageModule 
{action: "impression", action_data: "", user_id: 1353631, user_editcount: 0, user_variant: "D", …}
action: "impression"
action_data: ""
homepage_pageview_token: "t49q8kclco95ke35kr7a36frj6pqropb"
is_mobile: false
mode: "desktop"
module: "banner"
state: "enabled"
user_editcount: 0
user_id: 1353631
user_variant: "D"

event.HomepageModule {action: "link-click", action_data: "linkId=banner", user_id: 1353631, user_editcount: 0, user_variant: "D", …}

Is it only banner module is missing in the data lake for the specific user? Can you re-check if you see the banner and click on it and then in the data lake all your user actions will be recorded but not for the 'banner' module?

@kostajh is going to work on this issue.

@Etonkovidova this isn't deployed to group2 yet but should be shortly, as soon as wmf.28 arrives there.

...releng decided to abandon .28 and proceed with .29 directly. Just FYI.

This is working now, with events since 2021-02-02 in the data lake. Below are the counts of how many distinct users had impressions each day, and the percent that clicked on the link in the banner. Each day about 3.5% of homepage visitors (a dozen or two) click the banner in Arabic Wikipedia.