Page MenuHomePhabricator

Post-deployment QA: verify ad blocking instrumentation working as expected
Closed, ResolvedPublic

Description

This task is about verifying the new instrumentation added in T263505 is functioning as expected.

Requirements

This section will contain the behavior we are expecting to see.

Timing

Ideally, work on this task finishes on Friday, 14-May-2021.

Done

Event Timeline

Task description update

  • ADDED target finish date for this task to the task description – Friday, 14-May-2021 – per the conversation @MNeisler and I had on 28-April-2021.

Per T263505#7054699, work on this task can technically begin as early as this upcoming Thursday, 6 May. Although, @MNeisler and I are planning for work on this task to begin a week from today, on Monday, 10 May.

ldelench_wmf triaged this task as High priority.
ldelench_wmf moved this task from Triage to Kanban on the Product-Analytics board.

@DLynch - I checked VisualEditorFeatureUse today and have not seen any of the new events logged since deployment last Thursday. I'm wondering if it's possible that there have been no completed edits by users without JS logged since deployment (seems unlikely but may be possible due to sampling?) or if there might be an issue with the instrumentation.

Here are all the event actions logged in May 2021 for event.feature = 'mwSave'. There are no instances of event.action = 'source-has-js'.

actionnum_events
checkbox-wpMinoredit30197
checkbox-wpWatchthis6505
dialog-approve5469
dialog-save172396
review-initial-schnark7
review-initial-visual7579
review-switch-visual1697
window-open-from-command173841
checkbox-wpReviewEdit88
dialog-abort19589
dialog-preview3614
dialog-report706
dialog-resolve1933
dialog-review8835
review-initial-source1232
review-switch-source1613

Data via

SELECT
    event.action,
    COUNT(*)
FROM event.visualeditorfeatureuse
WHERE
     YEAR = 2021
    AND MONTH = 05
    AND event.feature = 'mwSave'
GROUP BY
    event.action

Hm. It should be about as common as the dialog-save action. I'll look into it.

Change 689984 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikiEditor@master] PHP VisualEditorFeatureUse logging: properly record session id

https://gerrit.wikimedia.org/r/689984

Change 689984 merged by DLynch:

[mediawiki/extensions/WikiEditor@master] PHP VisualEditorFeatureUse logging: properly record session id

https://gerrit.wikimedia.org/r/689984

Change 689901 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikiEditor@wmf/1.37.0-wmf.4] PHP VisualEditorFeatureUse logging: properly record session id

https://gerrit.wikimedia.org/r/689901

Change 689902 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikiEditor@wmf/1.37.0-wmf.5] PHP VisualEditorFeatureUse logging: properly record session id

https://gerrit.wikimedia.org/r/689902

Change 689902 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@wmf/1.37.0-wmf.5] PHP VisualEditorFeatureUse logging: properly record session id

https://gerrit.wikimedia.org/r/689902

Mentioned in SAL (#wikimedia-operations) [2021-05-12T23:40:11Z] <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.5/extensions/WikiEditor/includes/WikiEditorHooks.php: ef4139628a36eb8b747c610c8d769a802faf2fc3: PHP VisualEditorFeatureUse logging: properly record session id (T281409) (duration: 01m 08s)

Change 689901 merged by Urbanecm:

[mediawiki/extensions/WikiEditor@wmf/1.37.0-wmf.4] PHP VisualEditorFeatureUse logging: properly record session id

https://gerrit.wikimedia.org/r/689901

Mentioned in SAL (#wikimedia-operations) [2021-05-12T23:48:30Z] <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.4/extensions/WikiEditor/includes/WikiEditorHooks.php: 2f6af514c49d47bbec5ce51f9f7263015e039003? PHP VisualEditorFeatureUse logging: properly record session id (T281409) (duration: 01m 07s)

We have confirmed that the related errors have stopped, and data is being logged. Checks on the quality of said data have not yet occurred.

@DLynch

Working on post-deployment QA and noticed just one potential issue so far:

We have not logged any event.action = 'source-has-js' events for the mobile platform since the fix was deployed yesterday. There have been 12,409 no js wikitext events logged to date and they have all been recorded as coming from desktop.

Any ideas on why that might be? For comparison, there were 9,165 event.action = 'dialog-save' events on the mobile platform logged during the same timeframe.

I'll follow-up with my notes from the QA but everything else looks good so far.

Here's the notebook I'm performing the QA checks for reference.

That'd be expected -- WikiEditor only logs events as being desktop. See T249944 for a discussion of the issue.

(Conveniently, anything logging as phone like those dialog-save events inherently is using JS, since it can only possibly be logged via MobileFrontend.)

That'd be expected -- WikiEditor only logs events as being desktop. See T249944 for a discussion of the issue.

Per discussion with @ppelberg and @DLynch , we decided to use the user-agent data to approximate the mobile vs desktop wikitext edits to complete the analysis described in T240697. Note: User agent data will only be able to review for the last 90 days.

I've completed the remaining QA checks and everything else appears to be logging as expected.

Summary of checks and observations re event.action = 'source-has-js events:

  • We have recorded a total of 25,492 events by 9,214 distinct users (12 May to 14 May 2021). These numbers are similar in scale to the number of events marked as event.action = 'dialog-save' recorded around the same time.
  • Confirmed that events are only recorded for the event.feature = 'mwSave'
  • We've recorded events for 250 projects so far. The larger wikis (English, German, Spanish, and French) have the most total events logged as expected. A look at the percentage of all wikitext edits by js users for each project (which will be done in the analysis) will provide more info on how frequently these types of edits occur.
  • The majority of sessions (66%) are by logged-in users.
  • Confirmed that all events are also recorded as saved (event.action = saveSuccess) in EditAttemptStep. I will use a join to EditAttemptStep to determine the percent of edits coming from users without JS for the analysis.

Please see QA notebook for further details

@ppelberg - Reassigning to you for sign-off. Let me know if you have any questions.

ppelberg reopened this task as Open.

I've completed the remaining QA checks and everything else appears to be logging as expected.

Excellent.

@ppelberg - Reassigning to you for sign-off. Let me know if you have any questions.

@MNeisler: does any instrumentation documentation need to be updated? [i]


i.

Done

@MNeisler: does any instrumentation documentation need to be updated?

@ppelberg Good call - thanks for checking on this. I just updated the VisualEditorFeatureUse Dictionary with the new values used to track these events.

See: https://www.mediawiki.org/w/index.php?title=VisualEditor/FeatureUse_data_dictionary&diff=4603143&oldid=4412666

@MNeisler: does any instrumentation documentation need to be updated?

@ppelberg Good call - thanks for checking on this. I just updated the VisualEditorFeatureUse Dictionary with the new values used to track these events.

See: https://www.mediawiki.org/w/index.php?title=VisualEditor/FeatureUse_data_dictionary&diff=4603143&oldid=4412666

Oh, perfect. Resolving this now.