Page MenuHomePhabricator

Add logging for measuring impact of our work on improving the mobile block messages
Open, MediumPublic3 Estimated Story PointsTue, Sep 22

Description

Goal

This ticket relates to T189717: Better handle block reasons on mobile (specifically templates and HTML comments) which has some more context. We are making significant improvements to block messages on mobile. This has been brought up time and again as a problem for our end users who unexpectedly find themselves blocked.
This is important work and we want to make sure we can measure and highlight the impact we're having by doing this work.

Acceptance criteria:
  • Log the following:
    • How many times does the mobile block drawer display for end users?
    • How many times does a 'call to action' display (other than 'OK')?
    • How many times do users click on the 'call to action' (other than 'OK') (T259775)?
    • How many times do users access the special page or the expanded drawer to see the reason (T259755)?
  • We will use statsd to capture these numbers and we don't log the type of CTA.
Note

This has been done in the past. Related task - T201719: Tracking blocks: Log when the mobile web editor block notice is displayed
Q: Do we have any of the past data still available?

Event Timeline

Niharika triaged this task as Medium priority.Aug 12 2020, 4:51 AM
Niharika created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 12 2020, 4:51 AM
Niharika added a subscriber: jwang.Aug 12 2020, 5:06 AM

@jwang I want to flag this task for you. We can talk about it when we meet next.

Niharika updated the task description. (Show Details)Aug 19 2020, 4:46 PM
ARamirez_WMF set the point value for this task to 3.Aug 19 2020, 4:47 PM
Niharika updated the task description. (Show Details)Aug 19 2020, 4:48 PM
Niharika removed the point value for this task.
Niharika set the point value for this task to 3.Aug 19 2020, 5:06 PM

As discussed with @Niharika and @jwang:

  • The calls to action are "Log in" and "See random page". There's also "OK", but that dismisses the drawer and is probably less interesting.
  • We need to know (1) how many times was drawer shown, (2) how many times were the interesting calls to action shown, (3) how many times were the interesting calls to action clicked on

Thanks @mpopov! This is helpful.

Tchanders updated the task description. (Show Details)Mon, Aug 31, 12:06 PM
Tchanders claimed this task.Tue, Sep 8, 3:08 PM
Tchanders moved this task from Ready to In Progress on the Anti-Harassment (The Letter Song) board.
ARamirez_WMF changed the subtype of this task from "Task" to "Deadline".Wed, Sep 9, 8:45 PM
ARamirez_WMF set Due Date to Tue, Sep 22, 4:00 AM.

Change 626710 had a related patch set uploaded (by Tchanders; owner: Tchanders):
[mediawiki/extensions/MobileFrontend@master] Reinstate analytics tracking for the block message drawer

https://gerrit.wikimedia.org/r/626710

Change 626710 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Reinstate analytics tracking for the block message drawer

https://gerrit.wikimedia.org/r/626710

@Tchanders we are not seeing this data in Grafana yet. Jennifer built a dashboard to track it here. The old data is available in the dashboard if you zoom out.
If we are capturing the data the same way, it should show up in the dashboard, correct?

Change 629691 had a related patch set uploaded (by Tchanders; owner: Tchanders):
[operations/mediawiki-config@master] Enable mobile block notice tracking in MobileFrontend

https://gerrit.wikimedia.org/r/629691

@Niharika This is only on 1.36.0-wmf.10, so it wouldn't be giving any significant data yet. It also needs to be enabled via a config change, which we'll deploy later today - thanks for the reminder.

Change 629691 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable mobile block notice tracking in MobileFrontend

https://gerrit.wikimedia.org/r/629691

Mentioned in SAL (#wikimedia-operations) [2020-09-24T19:04:05Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: bcf9fcbe3b82ab85b8f97206ceca45b64619c362: Enable mobile block notice tracking in MobileFrontend (T260218) (duration: 01m 04s)

@Niharika This is only on 1.36.0-wmf.10, so it wouldn't be giving any significant data yet. It also needs to be enabled via a config change, which we'll deploy later today - thanks for the reminder.

Awesome, thanks! @jwang can you double check if the data is showing up now?