Page MenuHomePhabricator

Apply VisualEditorFeatureUse schema changes to all editing interfaces
Closed, ResolvedPublic

Description

T252924 and patch 594809 will add the events to the VisuallEditorFeatureUse schema we will need for the analysis we have planned in T247139.

This task is about applying those changes to all of the editing interfaces.

Editing interfaces needing updating

  • WikiEditor = source mode on desktop
  • MobileFrontend = source mode on mobile
  • VisualEditor = visual editor and new wikitext editor on desktop; visual mode on mobile

See for more information here: https://www.mediawiki.org/wiki/User:DLynch_(WMF)/VisualEditor/Configurations

Timing

  • Work on this task can begin once T254291 for additional events is resolved.
  • Ideally, Editing QA can start testing this task the week of 20-July.

Schema changes

Test instructions

  1. Scenarios to be tested provided in this section of the QC Replying Document
  2. Added new column called 'Test Artifact' to provide event as seen in the browser console
  3. Feel free to test either logged in or logged out. If logged out, user_id=0
  4. Please enter values observed for the following in each Scenario:
  5. user_id
  6. user_editcount
  7. bucket
  8. integration
  9. platform
  10. editing interface

Observed issues

The issues below surfaced during testing.

IssueDescriptionAction itemStatus
#1In Scenarios 3a and 3b (rows 169 & 170 ), the editor_interface field is not being logged when editing in MobileFrontend✅No action neededFixed
#2In Scenario 2b (row 165 ), the editor_interface and editingSessionID fields are not being logged when editing in the New Wikitext EditorNo action needednot reproducible
#3No VisualEditorFeatureUse events are being logged when testing at this URL: https://nl.wikipedia.org/wiki/Gebruiker:RYasmeen_(WMF)/Kladblok?editingStatsOversample=1✅No action neededFixed

Issue #1:

Done

  • 1. The "Events" and "Fields" are added to the VisualEditorFeatureUse schema in the following editing interfaces:
    • WikiEditor
    • MobileFrontend
    • VisualEditor
  • 2. QA has determined whether the fields and events defined in the "Test instructions" above are firing as expected.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Question
@DLynch will the patch needed to implement what this task is asking for [1] be the same patch needed to implement what T252924 is asking for [2]? Will this task and T252924 require separate patches? Something else entirely?


  1. Adding the new fields to all editing interfaces that use Schema:VisualEditorFeatureUse
  2. Adding fields to Schema:VisualEditorFeatureUse
ppelberg updated the task description. (Show Details)Jun 2 2020, 9:39 PM
DLynch added a comment.Jun 8 2020, 5:01 PM

We talked about it elsewhere, but I forgot to update it here: the extra-fields got folded into T252924's patch for DiscussionTools. The other interfaces will get a patch here.

Change 603545 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/WikiEditor@master] Update VisualEditorFeatureUse with new fields

https://gerrit.wikimedia.org/r/603545

Change 603549 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/VisualEditor@master] Update VisualEditorFeatureUse with new fields

https://gerrit.wikimedia.org/r/603549

Change 603551 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/MobileFrontend@master] Update VisualEditorFeatureUse with new fields

https://gerrit.wikimedia.org/r/603551

matmarex moved this task from To Triage to Triaged on the VisualEditor board.Jun 10 2020, 7:10 PM

Change 603545 merged by jenkins-bot:
[mediawiki/extensions/WikiEditor@master] Update VisualEditorFeatureUse with new fields

https://gerrit.wikimedia.org/r/603545

Change 603549 merged by jenkins-bot:
[mediawiki/extensions/VisualEditor@master] Update VisualEditorFeatureUse with new fields

https://gerrit.wikimedia.org/r/603549

Change 603551 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Update VisualEditorFeatureUse with new fields

https://gerrit.wikimedia.org/r/603551

Esanders assigned this task to DLynch.Jun 11 2020, 3:13 PM

@Mayakp.wiki, a question for you about testing the additional fields and events we added to VisualEditorFeatureUse schema (T252931) in the WikiEditor, MobileFrontend and VisualEditor:

  • Where is the best place for @Ryasmeen to log whether the events in each of these editing interfaces are being logged correctly? One thought is for this to happen in the existing Replying QC template as the events being tested were implemented as part of our work to instrument the Reply tool. On the other hand, the events we are testing as part of this ticket are in non-DiscussionTools interfaces so perhaps the record of this testing should live somewhere separate? ...please let us know where Rummana should log the testing findings.

@ppelberg I agree. Its best to record it in the current document. @Ryasmeen should test all the editing interfaces in each test scenario under the Production section of the Replying QC Template.
If she sees an unexpected value in any of the fields she can mark that scenario as 'failed' and explain the bug in the issue description (this should work even if she encounters issues with one scenario in multiple editing interfaces).

ppelberg added a comment.EditedJul 15 2020, 7:27 PM

@ppelberg I agree. Its best to record it in the current document. @Ryasmeen should test all the editing interfaces in each test scenario under the Production section of the Replying QC Template.
If she sees an unexpected value in any of the fields she can mark that scenario as 'failed' and explain the bug in the issue description (this should work even if she encounters issues with one scenario in multiple editing interfaces).

Sounds good, @Mayakp.wiki. As we discussed during today's call, the next steps are listed below.

Next steps

Mayakp.wiki updated the task description. (Show Details)Jul 17 2020, 6:09 PM

@Ryasmeen I have added a new section to the QC document and provided instructions in the Test Description of this task.
We are primarily looking at the data being stored in the 6 new fields. For that, I've added a few scenarios where you can trigger specific actions to validate them.
Please feel free to ping/chat or let me know if you have any questions.

LGoto moved this task from Triage to Tracking on the Product-Analytics board.Jul 20 2020, 4:11 PM

I started checking this on Dutch wiki with both article and talk page namespace.
For talk page: I am not seeing those six new fields as part of VisualEditorFeatureUse events listed on the template. Below is an example of the event I see when I switched editor from source to visual mode.

dt.schemaVisualEditorFeatureUse {feature: "editor-switch", action: "visual-desktop", editingSessionId: "ef71538958b68abb6166", editor_interface: "wikitext"}action: "visual-desktop"editingSessionId: "ef71538958b68abb6166"editor_interface: "wikitext"feature: "editor-switch"__proto__: constructor: ƒ Object()hasOwnProperty: ƒ hasOwnProperty()isPrototypeOf: ƒ isPrototypeOf()propertyIsEnumerable: ƒ propertyIsEnumerable()toLocaleString: ƒ toLocaleString()toString: ƒ toString()valueOf: ƒ valueOf()__defineGetter__: ƒ __defineGetter__()__defineSetter__: ƒ __defineSetter__()__lookupGetter__: ƒ __lookupGetter__()__lookupSetter__: ƒ __lookupSetter__()get __proto__: ƒ __proto__()set __proto__: ƒ __proto__()

As for article page:

On Beta cluster, I see two events getting fired. And it seems thee values user_id, user_editcount, integration, platform, editing interface is part of activity.link not dt.schemaVisualEditorFeatureUse.

On Production, there is no event firing for dt.schemaVisualEditorFeatureUse, I only see activity.link.

There's a consistency issue with the trackdebug logging for VisualEditorFeatureUse in DiscussionTools -- it's not also including the event-defaults... which is where four of the fields are located. The one that's in the regular payload is editor_interface, which you're indeed seeing there. (You won't see anything for bucket ever, because nothing is being bucketed.)

I've got a patch up to fix the oversight. In the meantime...

Use the EventLogging debugging mechanism found here, along with the editingStatsOversample=1 URL parameter.

Using that, I visited this talk page: https://nl.wikipedia.org/wiki/Overleg:Dubbelplaneet?editingStatsOversample=1 and switched from source to visual, producing this logged event:

{
  "event": {
    "user_id": 725721,
    "user_editcount": 6,
    "platform": "desktop",
    "integration": "discussiontools",
    "feature": "editor-switch",
    "action": "visual-desktop",
    "editingSessionId": "2a0e8a71959540b1c4d1",
    "editor_interface": "wikitext"
  },
  "schema": "VisualEditorFeatureUse",
  "webHost": "nl.wikipedia.org",
  "wiki": "nlwiki",
  "revision": 20118253
}
ppelberg updated the task description. (Show Details)Jul 21 2020, 7:22 PM

Task description update
I've updated the "Editing interfaces needing updating" section of the task description to make it more explicit after confirming the meanings of the terms below with @DLynch.

  • WikiEditor = source mode on desktop
  • MobileFrontend = source mode on mobile
  • VisualEditor = visual mode on desktop and mobile

Oh, "VisualEditor" is also the new source mode on desktop from beta settings, in this case.

Oh, "VisualEditor" is also the new source mode on desktop from beta settings, in this case.

Good catch, @DLynch. I've updated the "Editing interfaces needing updating" section of the task description to reflect this. Please edit it if you see something that needs to be corrected.

ppelberg updated the task description. (Show Details)Jul 21 2020, 7:48 PM

There's a consistency issue with the trackdebug logging for VisualEditorFeatureUse in DiscussionTools -- it's not also including the event-defaults... which is where four of the fields are located. The one that's in the regular payload is editor_interface, which you're indeed seeing there. (You won't see anything for bucket ever, because nothing is being bucketed.)

I've got a patch up to fix the oversight. In the meantime...

Use the EventLogging debugging mechanism found here, along with the editingStatsOversample=1 URL parameter.

Using that, I visited this talk page: https://nl.wikipedia.org/wiki/Overleg:Dubbelplaneet?editingStatsOversample=1 and switched from source to visual, producing this logged event:

{
  "event": {
    "user_id": 725721,
    "user_editcount": 6,
    "platform": "desktop",
    "integration": "discussiontools",
    "feature": "editor-switch",
    "action": "visual-desktop",
    "editingSessionId": "2a0e8a71959540b1c4d1",
    "editor_interface": "wikitext"
  },
  "schema": "VisualEditorFeatureUse",
  "webHost": "nl.wikipedia.org",
  "wiki": "nlwiki",
  "revision": 20118253
}

Following @DLynch's instructions I verified the events 1a, 1b and 2a so far. They all look correct to me.

However I have two questions:

  1. I am getting these events on Talk page namespace when I switch from Source mode to Desktop mode, Desktop to Source and while adding a link using visual mode. But I am not getting these on article namespace, meaning while using my sandbox on Dutch wiki and switching editors/adding links there, Is that expected?
  1. Scenario 2b asks to check the event when adding a link using Source mode using link icon. Since there is no link icon for source mode on Reply tool, I am assuming this is for NWE. Since there is no event firing for articles. I am marking it as Failed for now.

Checked the events for MobileFrontends today, scenario 3a and 3b is working correctly and has been marked as Passed on the QC template.

Waiting for confirmation about the two questions I had before.

  1. I am getting these events on Talk page namespace when I switch from Source mode to Desktop mode, Desktop to Source and while adding a link using visual mode. But I am not getting these on article namespace, meaning while using my sandbox on Dutch wiki and switching editors/adding links there, Is that expected?

That's not expected, and I cannot reproduce it. Could you give me more details of the page you're using?

e.g. I just asked for a random page (https://nl.wikipedia.org/wiki/Tanasevitchia_uralensis?veaction=editsource&editingStatsOversample=1) and added a link and switched modes, and got the appropriate events such as:

{
  "event": {
    "feature": "editor-switch",
    "action": "source-nwe-desktop",
    "editingSessionId": "b51c9f407cdda35d3be8",
    "user_id": 725721,
    "user_editcount": 6,
    "editor_interface": "visualeditor",
    "integration": "page",
    "platform": "desktop"
  },
  "schema": "VisualEditorFeatureUse",
  "webHost": "nl.wikipedia.org",
  "wiki": "nlwiki",
  "revision": 20118253
}
{
  "event": {
    "feature": "editor-switch",
    "action": "source-nwe-desktop",
    "editingSessionId": "b51c9f407cdda35d3be8",
    "user_id": 725721,
    "user_editcount": 6,
    "editor_interface": "visualeditor",
    "integration": "page",
    "platform": "desktop"
  },
  "schema": "VisualEditorFeatureUse",
  "webHost": "nl.wikipedia.org",
  "wiki": "nlwiki",
  "revision": 20118253
}
  1. Scenario 2b asks to check the event when adding a link using Source mode using link icon. Since there is no link icon for source mode on Reply tool, I am assuming this is for NWE. Since there is no event firing for articles. I am marking it as Failed for now.

Yeah, that test assumes the someday-in-the-future NWE-for-source-in-replies feature. I wouldn't fail it per-se, just say it's inapplicable.

Change 615870 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/VisualEditor@master] VisualEditorFeatureUse should respect oversampling config as well

https://gerrit.wikimedia.org/r/615870

ppelberg updated the task description. (Show Details)Jul 23 2020, 11:14 PM

Task description update
The description, status and action items for each of the three issues @Ryasmeen identified are now represented in the task description's "Observed issues" section.

Ryasmeen updated the task description. (Show Details)Jul 23 2020, 11:55 PM

Change 615909 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/MobileFrontend@master] VisualEditorFeatureUse wasn't including editor_interface

https://gerrit.wikimedia.org/r/615909

Change 615870 merged by jenkins-bot:
[mediawiki/extensions/VisualEditor@master] VisualEditorFeatureUse should respect oversampling config as well

https://gerrit.wikimedia.org/r/615870

Change 615909 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] VisualEditorFeatureUse wasn't including editor_interface

https://gerrit.wikimedia.org/r/615909

ppelberg updated the task description. (Show Details)Jul 24 2020, 10:54 PM
ppelberg updated the task description. (Show Details)
Ryasmeen added a comment.EditedJul 25 2020, 2:01 AM

I have re-checked the scenario 1a, 1b, 2a and 2b on article pages using trackdebug=1. They seem to be working correctly now. Added the test artifacts on the QC template. @Mayakp.wiki/@DLynch might need to review those.

Change 616195 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/MobileFrontend@master] VisualEditorFeatureUse: trackdebug output consistency

https://gerrit.wikimedia.org/r/616195

DLynch added a comment.EditedJul 25 2020, 5:10 AM

Is it because I am using trackdebug=1 on Beta cluster? The reason I am using trackdebug=1 is because using editingStatsOversample=1 on Beta cluster does not fire any events.

This is actually the same issue as we earlier had with DiscussionTools (because their logging is based on the same code) -- the trackdebug logging isn't including the default values for the schema, which is where those four values come from. The above patch will fix that, as the earlier one did for DiscussionTools.

Fortunately, this does mean it's entirely a QA-process issue, rather than any reflection on what's actually being logged. :D

Change 616195 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] VisualEditorFeatureUse: trackdebug output consistency

https://gerrit.wikimedia.org/r/616195

While re-testing scenario 2b with trackdebug=1 I couldn't reproduce the issue about editor_interface and editingSessionID being missing. So, it could have been an intermittent issue.

Per the above, I am marking "Issue #2" in the task description as "No action needed."

ppelberg updated the task description. (Show Details)Jul 28 2020, 11:08 PM
Ryasmeen updated the task description. (Show Details)Jul 29 2020, 2:11 AM

I have re-checked the scenario 1a, 1b, 2a and 2b on article pages using trackdebug=1. They seem to be working correctly now. Added the test artifacts on the QC template. @Mayakp.wiki/@DLynch might need to review those.

@ppelberg: I have updated the status of "Issue#3" per the above as well. The only thing left is, verifying "Issue#1" on production.

I have re-checked the scenario 1a, 1b, 2a and 2b on article pages using trackdebug=1. They seem to be working correctly now. Added the test artifacts on the QC template. @Mayakp.wiki/@DLynch might need to review those.

@ppelberg: I have updated the status of "Issue#3" per the above as well. The only thing left is, verifying "Issue#1" on production.

Excellent – thank you, Rummana.

ppelberg updated the task description. (Show Details)Jul 29 2020, 7:11 PM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)Jul 29 2020, 7:13 PM

@ppelberg/@DLynch/@Mayakp.wiki: Checked Issue#1 (Scenario 3a and 3b) on nl wiki using the mobile site on desktop and on iOS safari after wmf.2 deployment today. The events are now firing with editor_interface being included. I have updated the QA template with the updated test artifacts and changed the status to "Passed" for both the scenarios.

ppelberg closed this task as Resolved.Aug 4 2020, 11:23 PM

....Checked Issue#1 (Scenario 3a and 3b) on nl wiki using the mobile site on desktop and on iOS safari after wmf.2 deployment today. The events are now firing with editor_interface being included. I have updated the QA template with the updated test artifacts and changed the status to "Passed" for both the scenarios.

Excellent – thank you, Rummana. I'm updating the task description's "Observed issues" section to reflect the finding above and resolving this task.

Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptAug 4 2020, 11:23 PM