Round 1 of testing complete. I have added a section in the notebook called "Issues" where I am noting down data-QA issues.
https://github.com/wikimedia-research/metrics-platform-data-qa/blob/main/QA_event.mediawiki_edit_attempt.ipynb

Round 2 testing update: I was trying to compare metrics between editattemptstep(EAS) and mediawiki_edit_attempt(MEA) on testwiki and found that EAS has more events than MEA. see QA2_event.mediawiki_edit_attempt.ipynb
I believe 100% sampling is enabled on EAS for group 0 wikis T312016#8361090 (which includes testwiki) since Nov 1.

Round 3 testing update: Created Test Data using 2 user accounts and posted results in the notebook QA3_event.mediawiki_edit_attempt

Summary:
Discusstion Tools

• events were logged even with Ad Blocker on
• all events logged successfully

VE

• init, ready, loaded, abort were logged successfully
• Some events were missing save success, intent and attempt events

Wikitext editor

• none of the events were logged

I have communicated this with Metrics Platform team.

Change 855990 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/WikiEditor@master] ext.wikiEditor: Also log EditAttemptStep events via Metrics Platform

https://gerrit.wikimedia.org/r/855990

In T320281#8384972, @Mayakp.wiki wrote:

Wikitext editor

• none of the events were logged

I have communicated this with Metrics Platform team.

… It turns out that I missed a part of the EditAttemptStep instrument. https://gerrit.wikimedia.org/r/855990 addresses this.

Summary of Data-QA Issues

• Some logged-in users dont have the following fields populated -
  • user_name (performer.name)
  • registration_dt (performer.registration_dt)
  • edit_count_bucket (performer.edit_count_bucket)
• Wikitext events are not logged by the Metrics Platform. I believe this is being fixed in T320281#8391386
• Some VisualEditor events were missing save success, intent and attempt events but init, ready, loaded, abort were logged successfully (See cell #33 and test case #1 )
• Discusstion Tools events were logged even with Ad Blocker on (see test case #7)

Other feedback

• To do the QA, I had to use Superset for looking at sample data and HUE for the table structure, because Superset does not display struct columns well. Until Datahub provides the option for viewing sample data, along with breakdown of custom_data fields, I don't think we can use it as an effective tool for referencing datasets frequently.

In T320281#8404509, @Mayakp.wiki wrote:

• Some logged-in users dont have the following fields populated -
  • user_name (performer.name)
  • registration_dt (performer.registration_dt)
  • edit_count_bucket (performer.edit_count_bucket)

This is consistent with the EAS instrument. For what it's worth, if, in the future, we decided that the MEA instrument should collect this information, then all we need to do is update the stream config!

• Discusstion Tools events were logged even with Ad Blocker on (see test case #7)

Since this issue affects both instruments – one that uses the Event Platform and one that uses the Metrics Platform – would you mind double checking that your ad blocker is enabled and that it's blocking the intake-analytics.wikimedia.org domain?

@phuedx : When I try to open intake-analytics.wikimedia.org with Ad blocker on it says Cannot GET /
I re-tested again today and I can confirm events are still logged with ad blocker on

Metrics Platform:

In T320281#8406315, @Mayakp.wiki wrote:

@phuedx : When I try to open intake-analytics.wikimedia.org with Ad blocker on it says Cannot GET /

That's a genuine response from the service backing https://intake-analytics.wikimedia.org/. With the intake-analytics.wikimedia.org domain unblocked, I see the same thing:

What you're reporting is consistent with your ad blocker not blocking the domain.

@EChetty @phuedx this is on our "Needs Review" column on Product Analytics. Maya will be out until Jan 3, so I wanted to check if should stay here (is it being reviewed still?) or move (blocked? or has it been reviewed and is it ready for sign-off?)

@kzimmerman: Thanks for checking in. I think the Blocked is the best column as Maya and I will need to discuss the above comments when she gets back.

Being bold…

Change 855990 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] ext.wikiEditor: Also log EditAttemptStep events via Metrics Platform

https://gerrit.wikimedia.org/r/855990

I will retest the following -

• compare metrics between editattemptstep(EAS) and mediawiki_edit_attempt(MEA) on testwiki compare metrics between editattemptstep(EAS) and mediawiki_edit_attempt(MEA) on testwiki ( are there any wikis where EAS is enabled 100% sampling?)
• log events on Wikitext Editor
• log savesuccess events on VE

I think (based on my possibly-out-of-date understanding of the metrics platform) that the WikiEditor instrumentation is going to be incomplete, because the patch for it only added the client-side instrumentation. For WikiEditor the logging of the init, saveAttempt, saveSuccess, and saveFailure events is done server-side via EventLogging::logEvent.

Retested the metrics platform version mediawiki_edit_attempt with similar conclusions as last time. Report link

On Desktop

• server side wikitext editor (all Save and Init) events were not being logged
• only client side (ready, loaded, firstchange, abort) events are logged

On Mobile

• some wiki editor Mobile events were being logged using wt and mf mediawiki extensions i.e. the same events were being logged twice and double counted. After discussing with @phuedx, this seems to be an issue affecting both Metrics Platform edit_attempt instrument as well as legacy editattemptstep instrument and happens when the user navigates to /wiki/$title?action=edit on a mobile device.. /wiki/$title?action=edit is the url when editing on desktop. on mobile the editing url looks something like .. /wiki/$title#/editor/section# so this issue happens with anyone who edits on a mobile device using a desktop edit link.

Action items from QA test

• get server side events logging on Metrics Platform
• enable wikitext logging on Metrics platform

We cannot proceed with the implementation until server side event logging is enabled on Metrics Platform, which is currently blocked by T281762 .

Once this is unblocked, I will re test wiki text editor events in another task. Closing this task.

In T320281#8664778, @Mayakp.wiki wrote:

• some wiki editor Mobile events were being logged using wt and mf mediawiki extensions i.e. the same events were being logged twice and double counted. After discussing with @phuedx, this seems to be an issue affecting both Metrics Platform edit_attempt instrument as well as legacy editattemptstep instrument and happens when the user navigates to /wiki/$title?action=edit on a mobile device.. /wiki/$title?action=edit is the url when editing on desktop. on mobile the editing url looks something like .. /wiki/$title#/editor/section# so this issue happens with anyone who edits on a mobile device using a desktop edit link.

I want to make sure that this bug isn't due to some user setting that I've enabled. I'll try to replicate this logged out in a fresh browsing session on Tuesday morning (UTC) and check back in here after.

The described issue sounds consistent with the wikieditor JS and the mobile JS both running at the same time (such that the desktop wikieditor is loaded in the background), or possibly with the redirect-to-mobile code not running as expected (such that the desktop wikieditor loads, logs events, and then the redirect happens, rather than the redirect occurring before the form is loaded at all). However, I can't actually make this happen by directly navigating to an ?action=edit URL on mobile, so it's a bit confusing.

@Mayakp.wiki, @DLynch:

I was able to reproduce this in a fresh browsing session by emulating an iPad Air and navigating to https://en.wikipedia.org/wiki/Foobar?action=edit. After the mobile redirect finishes, the wikitext editor is visible, and then the wikitext editor overlay editor overlay provided by the MobileFrontend extension shows. After ~30 seconds, I saw the following events:

1. metrics_event: eas.mf.init
2. editattemptstep
   • action: ready
   • platform: desktop
3. metrics_event: eas.wt.ready
4. editattemptstep
   • action: loaded
   • platform: desktop
5. metrics_event: eas.wt.loaded
6. editattemptstep
   • action: init
   • platform: phone
7. metrics_event: eas.mf.ready
8. editattemptstep
   • action: ready
   • platform: phone
9. metrics_event: eas.mf.loaded
10. editattemptstep
    • action: loaded
    • platform: phone

Notes:

• All events have the same session and page tokens
• The eas.mf.* metrics_event events and the EditAttemptStep{platform=phone} events have the same editing session ID
• The EditAttemptStep{platform=desktop} have a 32-character editing session ID
• I was able to reproduce the above (though the events were submitted in a different order) by navigating to https://en.m.wikipedia.org/wiki/Foobar?action=edit directly whilst logged in

I'll send you both a video and figure out if I can upload it to Phab too…

Change 895337 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikiEditor@master] Don't do logging if MobileFrontend is active for the current page

https://gerrit.wikimedia.org/r/895337

Change 895337 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Don't do logging if MobileFrontend is active for the current page

https://gerrit.wikimedia.org/r/895337

Patch https://gerrit.wikimedia.org/r/895337 will be deployed in this week's train (03/13-17)
as for T281762, @phuedx notes that it will also be done this week. Metrics Platform team is following up with reviewers to get this moving forward.
Keeping it in Blocked until I can retest in a few days.

@phuedx is currently on sabbatical. @cjming confirmed that there's one outstanding patch in the WikiEditor extension that needs attention so for editattemptstep, we will not see wikitext events yet. She will get it merged this week and will be available end of next week i.e. after 5/25

Completed Round 2 of testing event.mediawiki_edit_attempt: notebook here

Takeaways and Status Update:

On Desktop

• server side wikitext editor (including Save and Init) events are now being logged!!
• All client side (ready, loaded, firstchange, abort) events are logged

On Mobile

• some wiki editor Mobile events were being logged using wt and mf mediawiki extensions i.e. the same events were being logged twice and double counted. We had reported this in January 2023 to @DLynch/@phuedx and this patch was merged in March 2023, and later reverted in Apr (4/14/23) when the Editing team did big centralizations of editattemptstep logging.

From @DLynch: the reason we removed that patch was that we thought we’d stopped WikiEditor and the MobileFrontend editor from appearing on the page at the same time with this patch for T334263. This suggests that we didn’t catch every case that could cause that.

After discussing with @DLynch and @cjming, @DLynch made https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiEditor/+/928926 to reapply the original patch which was merged and deployed Jun 15, 2023. As of Jun 20, we have not retested to see if the revert was successful in removing duplicate events.

Action items from QA test

• retest Mobile events on wikitext editor and check any discrepancies between mobile and desktop events on MEA and EAS tables. If they don't match, dig deeper and talk with Metrics Platform team and make aware to the implementation team in Product.