Page MenuHomePhabricator

Collect on-going numbers related to VE template dialog and TemplateWizard use
Closed, ResolvedPublic2 Estimated Story Points

Description

We'd like to track the number of successful interactions with the template Dialog in visual editor. We'll define a successful interaction as the user opening the dialog, making a change within it, and saving the change.

Template Dialog in VE:

  • Collect number times the VE template dialog is opened per day, split by edit count into five groups of users (0-10, 11-100, 101-1.000, 1.001-10.000, 10.001+)
    • Existing metric: Schema:VisualEditorFeatureUse, {feature: "transclusion", action: "window-open-*"}
  • Collect number of times the VE template dialog is opened per day from the "Insert" > "Template" menu
    • Existing metric: Schema:VisualEditorFeatureUse, {feature: "transclusion", action: "window-open-from-tool"}
    • Note that this is not necessarily inserting a template, it will open the Edit Template dialog if the user has already selected a template invocation.
  • Collect number of times the VE template dialog is opened per day by keyboard action (by typing "{{")
    • Existing metric: Schema:VisualEditorFeatureUse, {feature: "transclusion", action: "window-open-from-sequence"}
  • Collect number of times it is opened per day to edit an existing template (by clicking a template and then clicking edit button)
    • Existing metric: Schema:VisualEditorFeatureUse, {feature: "transclusion", action: "window-open-from-context"}
    • Editors may also double-click the template to open this dialog, recorded as {feature: "transclusion", action: "window-open-from-command"}.
  • Collect number of times the VE template dialog is opened per day, a change is made, and the change is saved successfully vs the workflow being abandoned.
    • Existing metric: Schema:VisualEditorFeatureUse, {feature: "transclusion", action: "dialog-done"}
    • Existing metric: Schema:VisualEditorFeatureUse, {feature: "transclusion", action: "dialog-abort"}

For interactions in the Template Dialog, track:

  • When a documented parameter is successfully added (i.e., and not abandoned) to a template
    • New action for existing metric: Schema:VisualEditorFeatureUse, {feature: "transclusion", "add-known-parameter"}. https://www.mediawiki.org/w/index.php?diff=4249302&diffmode=source
    • Note that we will have to aggregate by editingSessionId to determine whether the template dialog was closed happily, and whether the changes were finally synced to the article.
  • When an undocumented parameter is successfully added to a template.
    • New action for existing metric: Schema:VisualEditorFeatureUse, {feature: "transclusion", "add-unknown-parameter"}
  • When a parameter value is edited
  • Clicks on help page/ template page link

For comparison, also collect basic data on the Template Wizard:

  • Collect number of times the TemplateWizard is opened per day, split by edit count into five groups of users (0-10, 11-100, 101-1.000, 1.001-10.000, 10.001+)
  • Collect number of times the TemplateWizard is opened per day, a change is made, and the change is saved successfully.
    • Existing metric for successfully saved: Schema:TemplateWizard {action: "insert-template"}
    • Existing metric for unsuccessful abandonment: Schema:TemplateWizard {action: "cancel-dialog"}
    • Note that the page being constructed may or may not be successfully saved after exiting the TemplateWizard dialog. This seems like a relevant workflow detail, if we want it can be approximated by joining higher-level save and abort events on {page_id, revision_id, user_id}...

Other requirements:

  • Add any new topics to the inclusion list for sanitization
    • No new topics were added.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Lena_WMDE renamed this task from Collect on-going numbers related to VE template dialog use to Collect on-going numbers related to VE template dialog and TemplateWizard use.Sep 7 2020, 2:26 PM
Lena_WMDE updated the task description. (Show Details)
awight set the point value for this task to 3.Oct 1 2020, 9:23 AM
Lena_WMDE removed the point value for this task.
Lena_WMDE set the point value for this task to 8.Nov 16 2020, 1:38 PM

A general warning, most of the events we would rely on and add to are currently sampled at 6.25%

Please see the question about the second interaction ("Partial:" ...)

I don't understand the difference between these two criteria,

When an existing parameter is added (and filled) to a template using the VE template dialog
When a parameter which was not auto added (optional params) is added to a template

  • Here "existing" seems to mean, a named parameter which appears in TemplateData but not required. Isn't that the same as an optional (or suggested) parameter?
  • I assume that "optional params" should also include suggested params?
  • What does "and filled" mean? That the value is non-empty? I agree this would make sense from some perspectives, since an empty parameter is indistinguishable from a missing parameter when rendering a template. However, maybe the fact that the field exists in the interface is what actually matters for our case? Finally, this might be a slightly difficult metric to implement because it requires keeping track of which parameters were freshly added, and which have values when the dialog is closed. I suggest that we skip the "filled" requirement for our first pass.

When an existing parameter is edited using the VE template dialog

Now I have doubts about this criteria, too. Is this a different "existing" which has no relevant meaning? In other words, "Track whenever any parameter value is edited?" Or should we filter these edits to only report on "parameter listed in TemplateData" or "parameter which was already present when the dialog was opened"?

Change 643020 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/TemplateWizard@master] [DNM] Track user ID and edit count

https://gerrit.wikimedia.org/r/643020

When an existing parameter is added (and filled) to a template using the VE template dialog
When an existing parameter is edited using the VE template dialog
When a parameter which was not auto added (optional params) is added to a template
When an unnamed parameter is added to a template

It seems we need to talk about some of the terminology here.

  • "Unnamed" refers to template parameters that don't have a name, but a number. For example, in {{Messagebox|Hello}} the text "Hello" is the value of the first parameter, but the parameter doesn't have a name, i.e. is "unnamed". That's the same as {{Messagebox|1=Hello}}, i.e. "unnamed" parameters are automatically numbered, but specifying that number is optional when the template is used.
    • For what this ticket here is about, there is nothing special about unnamed parameters. I don't see a need to track them separately.
  • "Optional parameters" are all parameters that are not required, i.e. parameters I can remove.
  • I'm not sure what "existing" refers to. "Existing" in which context?

@Lena_WMDE?

Thanks @thiemowmde !

  • Unnamed parameter actually refers to undocumented parameters i.e. parameters that were not defined in the template documentation (TemplateData) and are added by the user in the template dialog using the "add parameter" function (right now in the bottom left menu bar). I believe the name may have been adjusted for usability testing, so that may be what has caused confusion. So, we are interested in tracking whether users want to add parameters to a template that are not predefined in its TemplateData. Does that make more sense?
  • By existing, we mean a documented parameter i.e. parameter defined in the template documentaiton (as opposed to the undocumented paramter above) that is added to a template and the value field is filled, in the case of the first bullet point. In the case of the second bullet point, we mean that the value of a documented parameter, already added in an existing template invocation, is changed.
  • Optional parameters as defined by the parameter's priority setting. Optional parameters are not auto-added to a tempalte invocation in the template dialog and so must be added (current functionality) and explicitly selected (likely future functionality) by the user. What is the confusing related to this one?

Let me know if that's clearer or if we should schedule some time to talk through them!

Thanks for the clarifications! I get the feeling that we're asking the same questions multiple times, however... It might be helpful to have a call, but let me collect the remaining open questions as I see them:

  • There is an assumption that "Insert" > "Template" menu always leads to the "Add template" dialog. It does not—if this menu item is chosen while a template is selected, then the normal edit dialog is opened. Meanwhile, there are other ways to add a template such as the "add template" puzzle piece in the bottom-left of the template dialog toolbar. Should we try to instrument just the "insert > template" menu action, when it results in "Add Template"? Should we instrument all ways of adding a template?
  • Do we really care whether an added parameter gets filled in with a value or left blank?
  • "In the case of the second bullet point, we mean that the value of a documented parameter, already added in an existing template invocation, is changed." I can do that, but it feels very abstract without understanding the motivation behind this metric. We don't want to count edits to undocumented parameters which were already present in the invocation?

"Optional" is clear to me now. The confusion is that we have a priority level called "optional", but also a "suggested" priority which makes a parameter optional. The criterion in the task is simply "not auto-added", in other words any parameter which is not required. Actually, this point is redundant because we're also tallying the "documented" and "undocumented" parameters, which adds up to include any parameter which might be added.

... the remaining open questions as I see them ...

Update: we chatted about these and I'll make corresponding changes to the task description.

awight updated the task description. (Show Details)
awight updated the task description. (Show Details)

Change 643494 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/VisualEditor@master] Send EventLogging when adding template parameters

https://gerrit.wikimedia.org/r/643494

@Lena_WMDE One more question, about:

When a documented parameter value is edited
When an undocumented parameter value is edited

For reasons, it's not easy to tell whether a parameter was "documented" or "undocumented" by the time the user edits its value. Can we collapse this metric into,

  • When a parameter value is edited

?

@Lena_WMDE Okay I need a few more hints about "when a parameter value is edited". There are several levels at which we could record a parameter value change. I'll just make a proposal rather than going through all the possibilities:

  • Send an event only the first time a parameter's value is changed. Subsequent edits to that parameter will not send more events. Edits to a different parameter will send one more event.
  • This allows us to aggregate over each editing session (VisualEditor pageview) and answer "How many parameters were edited during a session?".

Change 643502 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/VisualEditor@master] [WIP] Send EventLogging when editing parameter values

https://gerrit.wikimedia.org/r/643502

Thanks @awight. That sounds good for individual editing sessions. One question would be if it "resets" after the user closes the dialog? So, can we track:

  • user makes change to parameter (send event)
  • saves
  • reopens dialog and edits the same parameter again because the results weren't as expected (send new event)

Thanks @awight. That sounds good for individual editing sessions. One question would be if it "resets" after the user closes the dialog?

Yes, this is just what I was planning to do, there would be one parameter edit event per dialog per parameter. I'll change the task description to reflect!

Change 643692 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/VisualEditor@master] Send an event when the template doc link is clicked

https://gerrit.wikimedia.org/r/643692

Change 643020 merged by jenkins-bot:
[mediawiki/extensions/TemplateWizard@master] Track user ID and edit count

https://gerrit.wikimedia.org/r/643020

Lena_WMDE changed the point value for this task from 8 to 2.Dec 2 2020, 10:32 AM

Change 643692 merged by jenkins-bot:
[mediawiki/extensions/VisualEditor@master] Send an event when the template doc link is clicked

https://gerrit.wikimedia.org/r/643692

Change 643502 merged by jenkins-bot:
[mediawiki/extensions/VisualEditor@master] Send EventLogging when editing parameter values

https://gerrit.wikimedia.org/r/643502

Change 643494 merged by jenkins-bot:
[mediawiki/extensions/VisualEditor@master] Send EventLogging when adding template parameters

https://gerrit.wikimedia.org/r/643494

@ECohen_WMDE @Lena_WMDE
Before we put a medium amount of extra effort into this point, I'd like to clarify how strictly we need to follow the guidance:

When a documented parameter is successfully added (i.e., and not abandoned) to a template

  • Note that we will have to aggregate by editingSessionId to determine whether the template dialog was closed happily, and whether the changes were finally synced to the article.

The way I understand the requirement is that we need to join the parameter being added to the template eventually being saved. This can be done, but maybe we're also happy with the much simpler measurement of how many times a documented parameter is added to the template, regardless of whether the template dialog was eventually aborted?

Change 647742 had a related patch set uploaded (by Andrew-WMDE; owner: Andrew-WMDE):
[analytics/reportupdater-queries@master] [WIP] Process EventLogging events for VisualEditor

https://gerrit.wikimedia.org/r/647742

Change 647742 merged by Mforns:
[analytics/reportupdater-queries@master] Process EventLogging events for VisualEditor

https://gerrit.wikimedia.org/r/647742

Just to collect all of our changes to the VisualEditorFeatureUse schema,
https://www.mediawiki.org/w/index.php?title=VisualEditor/FeatureUse_data_dictionary&type=revision&diff=4250295&oldid=4229021&diffmode=source

We added four new actions under the transclusion feature:

  • add-known-parameter
  • add-unknown-parameter
  • edit-parameter-value
  • template-doc-link-click

Since the emitter implementation is already deployed, it's easiest for us to not adjust these names again, but if anything here will cause a problem of course we can be flexible. No data will be lost, our aggregation can account for changed names.

Thank you for the heads up about these instrumentation changes, @awight and @Lena_WMDE. A couple of resulting comments and questions...

Question

  • Have you all consulted the WMF Legal Team (WMF-Legal) about the VisualEditorTemplateDialogUse schema y'all are creating? If not, this will need to happen before the schema is deployed. I'm happy to send an email introducing you to the person within WMF legal who can help out with the schema review.

Comments

  • The naming conventions you all have followed and documented in T258920#6701120 look good to us. Also, thank you for updating the data dictionary.
  • RE VisualEditorTemplateDialogUse: should you want to be able to answer questions about editing sessions, @MNeisler and @DLynch recommend that you all add an editing sessionId field to the VisualEditorTemplateDialogUse schema so you can look at session-level data by way of joining it to the existing EditAttemptStep or VEFU schemas. David and Megan can offer guidance in the new calendar year should y'all value it.

hi @ppelberg thanks for your comments and questions.

Have you all consulted the WMF Legal Team (WMF-Legal) about the VisualEditorTemplateDialogUse schema y'all are creating? If not, this will need to happen before the schema is deployed. I'm happy to send an email introducing you to the person within WMF legal who can help out with the schema review.

We didn't realise this was a requirement, so it would be great if you could put me in touch with the right person asap! I should note that the schema has already been deployed, however, we were sensitive to data privacy concerns when setting it up and took steps to ensure no unnecessary private data would be processed or stored for longer than 90 days (as dictated by GDPR regulations).

...it would be great if you could put me in touch with the right person asap!

On it. You can expect an email from me tomorrow introducing you to the WMF Legal team tomorrow, 14-Jan.

...it would be great if you could put me in touch with the right person asap!

On it. You can expect an email from me tomorrowl introducing you to the WMF Legal team tomorrow, 14-Jan.

✅ Done.

WMDE-Fisch moved this task from Doing to Demo on the WMDE-TechWish-Sprint-2021-07-07 board.

For the record: Data is collected and aggregated as noted in the checklist above. Grafana boards might need polishing here and there.

WMDE-Fisch moved this task from Demo to Done on the WMDE-TechWish-Sprint-2021-07-07 board.