All of the events and event names proposed by @DLynch in T301582#7735903 look good to me. These events will allow us to measure all the proposed metrics identified in the task description except for the following:

What percentage of people make changes to the comment they had been drafting after updating the page to see the latest comments that have been posted?

@DLynch - Will a new editing session be created after the page is updated to see the latest comments? Assuming that's not the case, calculating this metric will not be feasible with the current instrumentation. Within a single editing session, we track when a user starts typing (event.action = 'firstChange') but do not track subsequent typing activity so we will not know if someone made additional edits to a current draft after updating the page.

We could look at the percentage of users (or editing sessions) that saved their replies after engaging with the new comment notification with the current instrumentation.

@MNeisler Sorry for the slow response here. Currently whenever you init a widget it resets the session ID -- this includes when the widget is recreated for the update, because internally that's basically done just by replacing the entire talk page contents and then letting the autosave-recovery do its work. As such you would see two sessions, one which ends with an interaction with the new comment notification in VisualEditorFeatureUse, and then immediately-after another which has its own init and firstChange.

Change 857713 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] Track events for the new comment notice

https://gerrit.wikimedia.org/r/857713

...and my vm froze while loading for the close one. I'll get back to that. :D

Also, @MNeisler, I noticed that the way the page is refreshed skips the normal abort event in editattemptstep. Should I fix this, or is it okay-for-the-situation?

There, the close:

Change 857713 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Track events for the new comment notice

https://gerrit.wikimedia.org/r/857713

@MNeisler Sorry for the slow response here. Currently whenever you init a widget it resets the session ID -- this includes when the widget is recreated for the update, because internally that's basically done just by replacing the entire talk page contents and then letting the autosave-recovery do its work. As such you would see two sessions, one which ends with an interaction with the new comment notification in VisualEditorFeatureUse, and then immediately-after another which has its own init and firstChange.

@DLynch Thanks, that helps clarify. For the first session that ends with the interaction with the new comment notification, I'm assuming that this would look like an incomplete edit session because there would be an init saved without a subsequent saveSuccess. I don't think it would impact our overall edit completion rate metrics too much but am thinking through the best way to filter out these events if needed when calculating edit completion rates.

Options:

• I could currently look for any sessions that engaged with a widget by joining with VisualEditorFeatureUse.
• We could add a new action in EditAttemptStep to specifically log instances where a session is reset due to interaction with a widget. Do you think that's feasible?

Also, @MNeisler, I noticed that the way the page is refreshed skips the normal abort event in editattemptstep. Should I fix this, or is it okay-for-the-situation?

I think that's fine for this situation. I'm also not sure if this should be grouped into the abort event and am wondering if it would be worth adding a new event for this per my comment above

For the first session that ends with the interaction with the new comment notification, I'm assuming that this would look like an incomplete edit session because there would be an init saved without a subsequent saveSuccess.

Yup, it should just vanish suddenly. Well, now it'll vanish after that page-update event in visualeditorfeatureuse, so it'll be detectable if you dig for it.

We could add a new action in EditAttemptStep to specifically log instances where a session is reset due to interaction with a widget. Do you think that's feasible?
[...]
I'm also not sure if this should be grouped into the abort event and am wondering if it would be worth adding a new event for this per my comment above

I think an abort variant makes sense with what's already there -- what's happening here is basically a variant on the existing switch abort-types, after all.

Change 903845 had a related patch set uploaded (by DLynch; author: DLynch):

[schemas/event/secondary@master] EditAttemptStep: Add a new abort type for page updates

https://gerrit.wikimedia.org/r/903845

Change 903846 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikimediaEvents@master] Bump EditAttemptStep schema version to 1.3.2

https://gerrit.wikimedia.org/r/903846

Change 903847 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] Log an EditAttemptStep session-abort when refreshing the page contents

https://gerrit.wikimedia.org/r/903847

Example event with the new abort type:

Change 903845 merged by jenkins-bot:

[schemas/event/secondary@master] EditAttemptStep: Add a new abort type for page updates

https://gerrit.wikimedia.org/r/903845

Change 903846 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Bump EditAttemptStep schema version to 1.3.2

https://gerrit.wikimedia.org/r/903846

Change 903847 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Log an EditAttemptStep session-abort when refreshing the page contents

https://gerrit.wikimedia.org/r/903847

@DLynch Thanks for the ping.

All the new instrumentation for this new feature appears to be logging as expected. See summary of checks and some initial data from the QA below.

New comment notice events

• All of the events for the new comment notice are being logged as expected in visualeditorfeatureuse

Total events logged for the new comment notice event.feature = 'notificationNewComments) since changes deployed

• There are more action=show events than other event types as expected. New comments are reloaded (action = page-update) for about 44% of the editing sessions where the new comment notification is shown and the notice is dismissed about 2% of the time.
• Events have currently been logged at 129 distinct wikis
• The majority of events are associated with integration = discussiontools as expected. Only 4 events logged with integration = page.
• Events logged by both logged-in (92%) and logged-out users (8%).
• The majority of events (93%) have been logged on desktop so far (7% platform = phone)

New abort type for page update events

• New abort_type = 'pageupdate' is logging as expected with all expected associated events

Summary of events logged since new instrumentation deployed

• Logged on both desktop (91% of editing sessions with this abort type) and phone platforms (9% of editing sessions with this abort type)
• 100% of these events have occurred with the discussiontools integration.
• Was able to join these abort type events with the new comment notice events logged visualeditorfeatureuse