Page MenuHomePhabricator

Newcomer tasks: guidance instrumentation
Open, Needs TriagePublic

Description

The help panel feature is already instrumented, per the work done in T206719: Help panel: instrumentation. In using it to display guidance for newcomer tasks, we have modifications and additions to make. This task will track the work, and the individual tasks about the different components of guidance will also contain notes on the relevant instrumentation for those components.

  • Changes to EventLogging schemas
  • Code for instrumentation
  • Post-deployment QA by data analyst
  • Publish changes to measurement plan

The specifications are recorded in the existing measurement plan for newcomer tasks at these headers:

  • Measurements: the numbers we want to be able to produce
  • Rules: specific interactions to be recorded. These are also pasted below for convenience. But note that the Google Doc contains useful comments from team members.

Additional rules for version 1.2 (guidance)

  • General
    • Since guidance will largely be implemented via the help panel, we want to record all the same things we already record via the help panel, in terms of it being opened and closed, preferences cog being clicked, back button being clicked, etc. The below points highlight things that are new with the guidance feature (and also includes some of the standard elements, just for completeness).
    • For all events, we need to be able to tell that they happened as part of the suggested edits workflow, and not as part of the help panel displaying in a normal editing session.
    • For all events, we need to be able to tell whether they happened in read mode or in edit mode.
  • Suggested edit screen in read mode
    • This screen displays on desktop when the user arrives on the article. On mobile, it displays when the user has tapped the mobile peek.
    • impression: we want to record that the user saw the help panel and which set of content it displays, in terms of which “task type” it is guiding, and whether it is showing the desktop VE, mobile VE, desktop wikitext, or mobile wikitext content.
    • clicks: we want to record exactly which tab the user clicks on when they click through the quick tips.
    • auto-advance: the quick start tabs will advance automatically every five seconds until the user clicks one. We should not record an event for auto-advances.
    • link clicks: the suggested edit screen will contain links to help documents, to which we want to record clicks.
  • Mobile peek
    • peek impression: we want to record that the user saw the mobile peek and which set of content it displays, in terms of which “task type” it is guiding, and whether it is showing the desktop VE, mobile VE, desktop wikitext, or mobile wikitext content.
    • tap on peek: whether the user taps the mobile peek.
    • dismiss peek: whether the user taps outside the mobile peek, which dismisses it.
  • Clicking edit
    • We want to record when the user clicks the “Edit” button. In two-tab wikis on desktop, we need to know whether they chose the VE or wikitext option. One way to figure this out may be with the editattemptstep schema with appropriate oversampling.
  • In edit mode
    • impression: we want to record whether the user saw the help panel in edit mode.
    • root screen button click: the new design of the “root screen” will contain several buttons. We want to record an event indicating which one the user clicks.
    • Once the user has clicked one of the buttons, normal help panel logging ensues.
  • Post-edit dialog
    • impression: we want to record that the user received the post-edit dialog. Because this dialog will contain a next article suggestion, we want to record the same details of that article suggestion that we record in se-task-impression events (e.g. title, whether it has an image, task type, maintenance template, etc.), while also recording that they received the impression via this post-edit dialog, as opposed to from the suggested edits module.
    • close: whether the user clicks away or clicks the “X” to dismiss the dialog.
    • clicks: whether the user clicks the link to go to their homepage.
    • task click: if the user clicks the suggested article, we want to record that click event along with the article details, similar to se-task-click events on the homepage, but specifying that they clicked via this post-edit dialog, as opposed to from the suggested edits module.
    • homepage visit: in the homepageVisit schema, we record the referer_route. We should add a referer_route for arriving on the homepage via the link in the post-edit dialog.

Event Timeline

Schema changes: HomepageVisit, HomepageModule, HelpPanel, new schema: NewcomerTask

HomepageVisit:

  • new referer_route value postedit-panel

HomepageModule:

  • task data is factored out into NewcomerTask and replaced with a newcomer task token that references it

HelpPanel:

  • new actions for the post-edit panel (impression, task click, link click, close)
  • editor_interface values which are not about the editor (reading, homepage) are split into a new context field
  • new is_suggested_task for logging whether we are in a suggested task session

Change 593972 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Instrumentation schema changes

https://gerrit.wikimedia.org/r/593972

More changes that came up:

  • a few more fields in HelpPanel.action_data for action=postedit-impression, to handle loading errors and impressions where no task is shown in the first place (diff)
  • a few more fields in HelpPanel.action_data for action=postedit-impression to handle T245790#6097815 (diff)

Change 595173 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Instrumentation for the post-edit dialog

https://gerrit.wikimedia.org/r/595173

Change 596719 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Instument mobile peek

https://gerrit.wikimedia.org/r/596719

So, the general changes (and the changes needed to ensure consistency with EditAttemptStep) are handled in c593972, c594780 and c595224, those specific to the post-edit dialog in 595173; most of the help panel event logging was in place already (a little bit inconsistent, T250327: Newcomer tasks: clean up help panel EventLogging fixes that); the mobile peek is handled in c596719. All that's left is events related to guidance content; that's left for later since the functionality is still in progress.

hi @Tgr and @MMiller_WMF - do we also record time elapsed before someone dismisses the help panel? Asking as I wonder if this can be a proxy to determining the bounce rate somehow (e.g., if they close the panel before 5 seconds have elapsed and without interacting with the panel, this is a sign they do not find the information interesting or useful).

hi @Tgr and @MMiller_WMF - do we also record time elapsed before someone dismisses the help panel? Asking as I wonder if this can be a proxy to determining the bounce rate somehow (e.g., if they close the panel before 5 seconds have elapsed and without interacting with the panel, this is a sign they do not find the information interesting or useful).

All events have a timestamp; per the discussion in today's meeting, that seems to be sufficient for getting elapsed times. We can add a JS timer later if @nettrom_WMF feels calculating event time deltas is not practical.

Change 593972 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Instrumentation schema changes

https://gerrit.wikimedia.org/r/593972

Change 595173 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Instrumentation for the post-edit dialog

https://gerrit.wikimedia.org/r/595173

Change 596719 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Instument mobile peek

https://gerrit.wikimedia.org/r/596719

Schema changes: HomepageVisit, HomepageModule, HelpPanel, new schema: NewcomerTask

HelpPanel:

  • new actions for the post-edit panel (impression, task click, link click, close)
  • editor_interface values which are not about the editor (reading, homepage) are split into a new context field
  • new is_suggested_task for logging whether we are in a suggested task session

Checked HelpPanel in betalabs along with changes done in T250327: Newcomer tasks: clean up help panel EventLogging.
Also, checked that all events listed on Schema:HelpPanel are being recorded.
Note:
When the guidance panel gets minimized (by clicking on 'X') the actions might look a little confusing:

  • impression
  • open
  • close
  • impression - again (since the minimized "Get help with editing" panel is viewed as a separate entity). This event identical to the first impression (might be confusing if some queries will not take it into account)

Most likely it's a no-issue but Schema:HelpPanel will record search requests depending on the speed of user typing (and the speed was not extremely low). So the logged events for searching might look like multiple requests, e.g. the below is for typing test
event.HelpPanel {action: "search", action_data: "queryLength=1;resultCount=10", user_id: 47, user_editcount: 8, context: "editing", …} action: "search" action_data: "queryLength=1;
event.HelpPanel {action: "search", action_data: "queryLength=2;resultCount=10", user_id: 47, user_editcount: 8, context: "editing", …}action: "search" action_data: "queryLength=2;
event.HelpPanel {action: "search", action_data: "queryLength=4;resultCount=10", user_id: 47, user_editcount: 8, context: "editing", …} action: "search" action_data: "queryLength=4

@MMiller_WMF - moving for PM review since

  • the tasks listed in the description are either wer PM-reviewed and Resolved

and

  • I went through all requirements and checked them in betalabs. Aside from my comment https://phabricator.wikimedia.org/T251526#6246417 (certain context values are not shown) and my minor observation points in that comment and in this task below.

Also,

Schema changes: HomepageVisit, HomepageModule, HelpPanel, new schema: NewcomerTask

HelpPanel:

  • new actions for the post-edit panel (impression, task click, link click, close)
  • editor_interface values which are not about the editor (reading, homepage) are split into a new context field
  • new is_suggested_task for logging whether we are in a suggested task session

For context ( editor_interface) I verified only editing, reading and`postedit.`

In testing this, I also noticed that I receive a help panel impression event when I minimize the help panel. Is that deliberate? I don't think that was the case before we added guidance, because the close event covers that. This is the only issue I see that needs to be addressed, except for my separate comment in T251526#6262946.

In testing this, I also noticed that I receive a help panel impression event when I minimize the help panel. Is that deliberate?

That's the same issue that was discussed in T244431#6249231. T255255: Newcomer tasks: Enable easier toggling to open and close Desktop help panel will probably fix it, since it's caused by removing/readding the help button as the panel opens/closes.

In testing this, I also noticed that I receive a help panel impression event when I minimize the help panel. Is that deliberate?

That's the same issue that was discussed in T244431#6249231. T255255: Newcomer tasks: Enable easier toggling to open and close Desktop help panel will probably fix it, since it's caused by removing/readding the help button as the panel opens/closes.

There's also a patch to address this: Help panel: Fix duplicate impression events when guidance is enabled

That fixes a similar issue (logging an impression event when the page is loaded with the help panel open), but not the one of logging an impression when closing the help panel, as far as I can tell.

In testing this, I also noticed that I receive a help panel impression event when I minimize the help panel.

I'll reassess this (and fix if it still needs fixing at that point) once T255255 has been merged. (And follow up on T244431#6269930 which asks about a related behavior.)

@Tgr -- we think that now enough has been merged for you to take the next steps on this task.