Page MenuHomePhabricator

Leveling up: Instrumentation for new post-edit dialog for non-suggested edits
Closed, ResolvedPublic

Description

Per T328060: Leveling up: Instrumentation, capture:

  • Everything the HelpPanel schema currently captures about that event (e.g. time, platform, wiki, user ID, edit count, etc)
  • That the user is seeing this “try a Suggested edit” dialogue. This should be reflected in the type value in the action_data field (because the context and action fields already reflect that it's an impression of the post-edit dialogue).
  • The action the user took, whether it was “try suggested edits” or “no thanks”.

That is:

  • Log the following values:
    • on impression: context=postedit-nonsuggested,action=postedit-impression
    • on closing (this can happen via the "No thanks" button, the X icon, the Esc key, clicking outside - I assume it's OK not to differentiate since we don't do that elsewhere either) context=postedit-nonsuggested,action=postedit-close
    • on clicking the "Try suggested edits" button: context=postedit-nonsuggested,action=postedit-link-click,action_data=suggested-edits (and savedTaskType in action_data will become optional since we don't have it)
  • on collapsing/expanding the postedit dialog: context=postedit-nonsuggested,action=postedit-collapse / context=postedit-nonsuggested,action=postedit-expand
  • help_panel_session_id is a random value that's the same for events within the same pageview.
  • editor_interface will take the value not-known (it can probably be recovered during analysis by checking the change tags on the user's last edit)
  • update schema docs for savedTaskType

Event Timeline

@nettrom_WMF does that sound OK? Especially the type field in action_data that differentiates post-edit notices after suggested vs. non-suggested edits, so far we only had it in the impression action - do you want to stick with that and use the session ID to match further events, or put type into the action data for all events to differentiate them from normal post-edit? (Or maybe use a new context value like postedit-nonsuggested? That seems the cleanest to me.) Currently for link-click actions the action data is just the link name, without any key-value structure; we should probably change that (here and elsewhere for consistency) if we want to add another piece of information for link click events.

kostajh lowered the priority of this task from High to Medium.

@nettrom_WMF does that sound OK? Especially the type field in action_data that differentiates post-edit notices after suggested vs. non-suggested edits, so far we only had it in the impression action - do you want to stick with that and use the session ID to match further events, or put type into the action data for all events to differentiate them from normal post-edit? (Or maybe use a new context value like postedit-nonsuggested? That seems the cleanest to me.) Currently for link-click actions the action data is just the link name, without any key-value structure; we should probably change that (here and elsewhere for consistency) if we want to add another piece of information for link click events.

This largely looks good to me! I have a couple of thoughts/comments:

  1. I would love to keep help_panel_session_id in the schema, to be used to connect the various events in the same post-edit dialogue. I recognize that it duplicates session_token in this context, and also that there might only be a handful of event per session for these dialogues. The main reason for me to keep help_panel_session_id would be that it would result in the three post-edit dialogues (this one, the regular "post-suggested-edit" dialogue, and the new "try a new task type" dialogue) using the same short-term token when combining events. Let me know if that complicates the engineering of this, as we can also choose to document this in the measurement specification so I don't forget that this particular dialogue requires me to use session_token instead.
  2. I don't have a strong opinion about whether to have a new context value or whether to identify these through type in action_data. When I was drafting the measurement specification I think the main dialogue I was focused on was the "try new task type" dialogue, where the suggestion to try a new task type is always followed by the list of suggested edits (with either the new task type or an old one). In that case having context=postedit made sense to me since that interaction has a continuous context. Either way, I'm definitely comfortable with using type in action_data on an action=postedit-impression event to define the start of the interaction funnel for these dialogues.
  3. I don't see a need to differentiate how the user closed the dialogue, happy to only log that they closed it.

I think that's all, as mentioned this largely looks okay to me.

Thanks! I'll use context=postedit-nonsuggested then, that's marginally simpler; and have help_panel_session_id duplicate session_token.

Change 893092 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[schemas/event/secondary@master] helppanel: Add postedit-nonsuggested context

https://gerrit.wikimedia.org/r/893092

The non-suggested postedit panel also seems sufficiently different for tracking homepage visits from it separately, so I'll add the new referer value postedit-panel-nonsuggested.

Change 893098 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[schemas/event/secondary@master] homepagevisit: Add referer_route:postedit-panel-nonsuggested

https://gerrit.wikimedia.org/r/893098

Added this per the discussion in today's meeting:

editor_interface will take the value not-known (it can probably be recovered during analysis by checking the change tags on the user's last edit)

We should probably figure out a more natural way to handle postedit actions since we are relying on them quite a bit. I'll file a tech debt task about that.

@nettrom_WMF FWIW it doesn't seem like events right before navigation (such as button click events when the button is a link or JS redirect) get logged. This is not specific to this dialog (I can see that the dialog triggers the mw.track event but then EventLogging doesn't react to it) - maybe caused by the EventLogging library rewrite a while ago? From skimming the metric library code the handling of this scenario seems solid, so not sure what's going on.

I assume in this specific case it is not super important as there will be a homepagevisit event right afterwards anyway.

Change 893102 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Leveling up: InviteToSuggestedEdits instrumentation

https://gerrit.wikimedia.org/r/893102

Patch notes:

  • It makes the logging framework set the editor schema field to not-known when it can't guess the editor. This affects some existing instrumentation, as there were some rare cases where the editor couldn't be guessed, but previously we just guessed visualeditor.
  • It makes the framework log the page title/ID in the post-edit panel (both this one and others). That didn't happen previously. I think that was a bug: we skip those fields for help panel events in reading mode because logging what the user reads would be invasive, and the post-edit panel is technically in reading mode; but it's an editing-related event so that was the wrong check to apply.

@nettrom_WMF FWIW it doesn't seem like events right before navigation (such as button click events when the button is a link or JS redirect) get logged. This is not specific to this dialog (I can see that the dialog triggers the mw.track event but then EventLogging doesn't react to it) - maybe caused by the EventLogging library rewrite a while ago? From skimming the metric library code the handling of this scenario seems solid, so not sure what's going on.

I assume in this specific case it is not super important as there will be a homepagevisit event right afterwards anyway.

You're right that since there'll be a HomepageVisit event right afterwards if they click that link we can catch it that way. I'm somewhat concerned about this and have made a note to specifically test when this is deployed to testwiki. If the click events don't make it, I'll follow up with the Event platform folks.

I read through the task description and your comments, and everything looks good from my end!

Change 893098 merged by jenkins-bot:

[schemas/event/secondary@master] homepagevisit: Add referer_route:postedit-panel-nonsuggested

https://gerrit.wikimedia.org/r/893098

Change 893092 merged by jenkins-bot:

[schemas/event/secondary@master] helppanel: Add postedit-nonsuggested context

https://gerrit.wikimedia.org/r/893092

Change 893842 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[schemas/event/secondary@master] helppanel: Add not-known editor type

https://gerrit.wikimedia.org/r/893842

Change 893842 merged by jenkins-bot:

[schemas/event/secondary@master] helppanel: Add not-known editor type

https://gerrit.wikimedia.org/r/893842

Change 893102 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Leveling up: InviteToSuggestedEdits instrumentation

https://gerrit.wikimedia.org/r/893102

Patch notes:

  • It makes the logging framework set the editor schema field to not-known when it can't guess the editor. This affects some existing instrumentation, as there were some rare cases where the editor couldn't be guessed, but previously we just guessed visualeditor.
  • It makes the framework log the page title/ID in the post-edit panel (both this one and others). That didn't happen previously. I think that was a bug: we skip those fields for help panel events in reading mode because logging what the user reads would be invasive, and the post-edit panel is technically in reading mode; but it's an editing-related event so that was the wrong check to apply.

For HelpPanel editor_interface: "not-known" is displayed when context: "reading" or context: "postedit-nonsuggested":

action: "impression"
action_data: "taskType=copyedit"
context: "reading"
editor_interface: "not-known"

action: "open"
action_data: ""
context: "reading"
editor_interface: "not-known"
  • a user closes the Help panel
action: "close"
action_data: ""
context: "reading"
editor_interface: "not-known"
 
action: "impression"
action_data: "taskType=copyedit"
context: "reading"
editor_interface: "not-known"
  • non-suggested dialog appears
action: "postedit-impression"
action_data: ""
context: "postedit-nonsuggested"
editor_interface: "not-known"
  • for a non-suggested dialog actions
action: "postedit-impression"
action_data: ""
context: "postedit-nonsuggested"
editor_interface: "not-known"
action: "postedit-collapse"
action_data: ""
context: "postedit-nonsuggested"
editor_interface: "not-known"

Checked in testwiki wmf.2 - the events look
e.g.

action: "postedit-impression"
action_data: ""
context: "postedit-nonsuggested"
editor_interface: "not-known"
help_panel_session_id: "70328c2101f5cfb3e79b"
is_mobile: false
is_suggested_task: false
page_id: 101596
page_ns: 0
page_protection: ""
page_title: "MavetunaZialnt30_2"
session_token: [removed]
user_can_edit: true
user_editcount: 2
user_id: [removed]

Note: Still need to find a way to track all events in production, e.g. I cannot see the details of due to the redirection to Special:Homepage:

on clicking the "Try suggested edits" button: context=postedit-nonsuggested,action=postedit-link-click,action_data=suggested-edits