Page MenuHomePhabricator

Newcomer tasks: instrumentation for suggested edit screen tips
Closed, ResolvedPublic

Description

From the parent task:

  • This screen displays on desktop when the user arrives on the article. On mobile, it displays when the user has tapped the mobile peek.
    • impression: we want to record that the user saw the help panel and which set of content it displays, in terms of which “task type” it is guiding, and whether it is showing the desktop VE, mobile VE, desktop wikitext, or mobile wikitext content.
    • clicks: we want to record exactly which tab the user clicks on when they click through the quick tips.
    • auto-advance: the quick start tabs will advance automatically every five seconds until the user clicks one. We want to record an event for each auto-advance, indicating which tab it advanced to. (see T253009#6176438)
    • link clicks: the suggested edit screen will contain links to help documents, to which we want to record clicks.

Related Objects

StatusSubtypeAssignedTask
OpenNone
Resolvedkostajh
ResolvedCatrope
Resolvedkostajh
Resolvedkostajh
Resolvedkostajh
ResolvedBUG REPORTkostajh
Resolvedkostajh
Resolvedkostajh
ResolvedCatrope
Resolvedkostajh
Resolvedkostajh
ResolvedBUG REPORTkostajh
ResolvedTgr
OpenNone
ResolvedTgr
ResolvedTgr
Resolvedkostajh
OpenNone
ResolvedTgr
Resolvedkostajh
ResolvedTgr
InvalidNone
ResolvedTgr
Resolvedkostajh
ResolvedTgr
ResolvedTgr
DeclinedNone
Resolvedkostajh
Resolvedkostajh
ResolvedBUG REPORTTgr
ResolvedTgr
ResolvedCatrope
Resolvedkostajh
ResolvedBUG REPORTCatrope
OpenTgr
ResolvedTgr

Event Timeline

@kostajh @Tgr -- I just noticed this task wasn't on our sprint board. Where should it be? Is it In Progress?

MMiller_WMF renamed this task from Instrumentation for suggested edit screen tips to Newcomer tasks: instrumentation for suggested edit screen tips.Jun 1 2020, 6:07 AM

impression: we want to record that the user saw the help panel and which set of content it displays, in terms of which “task type” it is guiding, and whether it is showing the desktop VE, mobile VE, desktop wikitext, or mobile wikitext content.

We are already logging whether the user is on desktop or mobile and whether they are using VE or the source editor. (The latter is not necessarily accurate when in reading mode, but the same guess is used by the logging and the tips panel.) That's not necessarily the same as which content is shown (for example, if the user is using the wikitext editor, currently we'll show the VE content because we do not have any wikitext content) but the latter is currently not a well-defined property as the API handles content fallback field-by-field. Not sure if that's a problem.

link clicks: the suggested edit screen will contain links to help documents, to which we want to record clicks.

I don't see any, am I missing something? Or does this refer to the links in the settings menu?

@Tgr this is the “learn more” link that appears in the last tip for each task type, and sometimes as an inline link on the second to last tip on each task type. Its presence is contingent on being defined in the NewcomerTasks.json file

@Tgr -- your comment about logging the editor sounds fine. But I want to make sure we'll be able to tell when the user is receiving the help panel content in read mode. In my memory, the helppanel schema's hack for that is having a value of read for the editor? Will we be continuing that?

Change 602529 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Instrumentation for guidance tips panel

https://gerrit.wikimedia.org/r/602529

Thanks Kosta, forgot about those!

But I want to make sure we'll be able to tell when the user is receiving the help panel content in read mode. In my memory, the helppanel schema's hack for that is having a value of read for the editor?

It used to be; since T251526: Newcomer tasks: create separate EventLogging schema for newcomer task impressions got merged, there's a context field which is editing or reading (or something else for the homepage).

The part that depends on T253009: Newcomer tasks: automatically advance tips still needs to be done.

As I understand T253009#6176438 we are not logging automatically advancing tips, so I think this task can go to QA.

Change 602529 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Instrumentation for guidance tips panel

https://gerrit.wikimedia.org/r/602529

As I understand T253009#6176438 we are not logging automatically advancing tips, so I think this task can go to QA.

Thanks, I missed that. Not logging them still needs to be done, though - the current code just naively logs tab changes (set events), some extra effort will be needed to avoid that for auto-advance.

Also, the auto-advance event needs to be cleared from the schema.

Change 602775 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Clean up unused HelpPanel schema field

https://gerrit.wikimedia.org/r/602775

Change 602775 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Clean up unused HelpPanel schema field

https://gerrit.wikimedia.org/r/602775

This had to be re-written significantly for T253009: Newcomer tasks: automatically advance tips so it should probably be re-tested when that lands.

Tested in betalabs - action=guidance-tab-advance is not present.

Will wait on T253009 to re-test.

Tested in betalabs - action=guidance-tab-advance is not present.

We are not logging that, see the link in the task description.

Tested in betalabs - action=guidance-tab-advance is not present.

We are not logging that, see the link in the task description.

Yes, my comment was supposed to sound affirmative :) "Tested in betalabs - action=guidance-tab-advance is not present." - correct behavior.

Checked if HelpPanel records events according to this task specs - my notes in (1) are only to make sure that "impression", "open", and "close" events do not cause any over- or under-counting.
(1)

impression: we want to record that the user saw the help panel and which set of content it displays, in terms of which “task type” it is guiding, and whether it is showing the desktop VE, mobile VE, desktop wikitext, or mobile wikitext content.

  • I did not see impression event when landing on SE article, only open. I do have enabled "Preserve log" (Chrome) and even used mw.loader.using( 'mediawiki.api' ).then( function () { new mw.Api().saveOption( 'eventlogging-display-web', '1' ); } ); to see all available eventlogging schemas events.
  • minimizing (closing on 'X') the Guidance panel will produce the following sequence

event.HelpPanel {action: "close", action_data: "", user_id: 24, user_editcount: 55, context: "reading", …}
event.HelpPanel {action: "impression", action_data: "taskType=expand", user_id: 24, user_editcount: 55, context: "reading", …}

  • also there will be impression event after I click on 'Edit'

event.HelpPanel {action: "open", action_data: "", user_id: 24, user_editcount: 55, context: "editing", …}
event.HelpPanel {action: "impression", action_data: "taskType=expand", user_id: 24, user_editcount: 55, context: "editing", …}
"open" in the above it's not a user action, it's a state in which the panel was before switching to Editing mode.
(2)

clicks: we want to record exactly which tab the user clicks on when they click through the quick tips.

Seems to be recorded correctly, e.g.

event.HelpPanel {action: "open", action_data: "", user_id: 24, user_editcount: 53, context: "reading", …}
 event.HelpPanel {action: "guidance-tab-click", action_data: "taskType=links;tabName=tipset-4", user_id: 24, user_editcount: 53, context: "reading", …}
 event.HelpPanel {action: "guidance-tab-click", action_data: "taskType=links;tabName=tipset-6", user_id: 24, user_editcount: 53, context: "reading", …}
 event.HelpPanel {action: "guidance-tab-click", action_data: "taskType=links;tabName=tipset-1", user_id: 24, user_editcount: 53, context: "reading", …}

(3)

link clicks: the suggested edit screen will contain links to help documents, to which we want to record clicks.

Seems to be recorded correctly, e.g.

event.HelpPanel {action: "guidance-tab-click", action_data: "taskType=expand;tabName=tipset-5", user_id: 24, user_editcount: 55, context: "editing", …}
 event.HelpPanel {action: "link-click", action_data: "guidance-expand-tipset-5", user_id: 24, user_editcount: 55, context: "editing", …}