1. Should mobile be treated differently for all those questions? (Because the display of checks/suggestions is so different, some don't even apply.)
2. Should checks also be tracked? Is that going to be different when they're pre-save vs mid-edit?

3. On mobile, when multiple checks are combined into a single icon in the gutter because they overlap, does that count for all of them? Only the ones whose relevant content is actually visible?
4. On desktop/mobile, in pre-save when we only show a single check and give you pagination arrows, does it log only when you page to each one? (Even if the rest of the check trappings -- the rail, the content it's about -- is visible in the viewport.)
5. What percentage of a suggestion was visible within someone's viewport? <-- what if less than that percentage of a suggestion can fit in the full viewport? (I.e. what if the suggestion is really tall? I'm specifically thinking of the merged mobile gutter again, but a textmatch rule could user-configure a reaaaaaaallllllly long description.)
6. Should this trigger every time the check is seen, or only once? If once, once for each instance of the check, or once for the entire check-type?

Great questions, @DLynch. Proposal below.

Can you please hold off on implementing until @MNeisler have given an explicit +1?

In T412334#11555575, @DLynch wrote:

1. Should mobile be treated differently for all those questions? (Because the display of checks/suggestions is so different, some don't even apply.)

Yes

1. Should checks also be tracked? Is that going to be different when they're pre-save vs mid-edit?

Good question. What do you think, @MNeisler?

My instinct/current understanding...

• Currently, we implicitly assume all Edit Checks are seen because they're either A) shown alongside what you're typing, as you're typing it and/or B) shown in Pre-Save
  • Note: on desktop and mobile someone might be more likely to see an Edit Check in Pre-Save as opposed to Mid-Edit seeing as how in the latter, Checks are surfaced via the presence of an icon in the rail.
• It makes sense to make this "shown/seen" event explicit so we don't need to hold thinking like the above in our minds.

In T412334#11556204, @DLynch wrote:

3. On mobile, when multiple checks are combined into a single icon in the gutter because they overlap, does that count for all of them? Only the ones whose relevant content is actually visible?

The latter. On mobile, let's assume "shown" means the actual card is shown in an expanded state.

4. On desktop/mobile, in pre-save when we only show a single check and give you pagination arrows, does it log only when you page to each one? (Even if the rest of the check trappings -- the rail, the content it's about -- is visible in the viewport.)

To me, it seems simplest/most coherent to say that we only log when you page to each Check/Suggestion.

5. What percentage of a suggestion was visible within someone's viewport? <-- what if less than that percentage of a suggestion can fit in the full viewport? (I.e. what if the suggestion is really tall? I'm specifically thinking of the merged mobile gutter again, but a textmatch rule could user-configure a reaaaaaaallllllly long description.)

Good point. Let's skip this nuance for now and define it as any portion of the Suggestion/Check being visible in someone's viewport.

6. Should this trigger every time the check is seen, or only once? If once, once for each instance of the check, or once for the entire check-type?

Once for each instance of a Check.

Thinking: we're most immediately interested in this info. for the purpose of learning things like: 1) the proportion of Suggestions people see and act on, 2) the proportion of people who engage with a particular type of Suggestion, 3) the proportion of Suggestions that are available within a given edit session that people are actually seeing.

On mobile, let's assume "shown" means the actual card is shown in an expanded state.

Just to confirm: on mobile the icons in the gutter don't count for this metric at all. It's 100% counting when something appears in the drawer?

In T412334#11571640, @DLynch wrote:

On mobile, let's assume "shown" means the actual card is shown in an expanded state.

Just to confirm: on mobile the icons in the gutter don't count for this metric at all. It's 100% counting when something appears in the drawer?

Yep. This is what I'm proposing. I'm glad you asked to clarify.

@medelius: the requirements you drafted in T412334#11580819 look great to me.

Meta: @MNeisler, we decided to boldly move forward with implementing the spec above and will block merging on you reviewing and providing feedback on the initial implementation editing engineering will have implemented.

Change #1236822 had a related patch set uploaded (by Medelius; author: Medelius):

[mediawiki/extensions/VisualEditor@master] EditCheck: add instrumentation for checks seen during edit session

https://gerrit.wikimedia.org/r/1236822

The instrumentation spec looks good to me and I agree with logging the visible event for both checks and suggestions.

Just a couple clarifying questions to make sure I understand what's being logged:

Event logged on the first time a suggestion or check instance is visible within VisualEditor, with the following definitions

Does this mean we log an event once for every Check/suggestion seen or just once for every Check/suggestion type? For example, if 3 Add a Citation suggestions and 1 Revise tone suggestion are visible within an editing session, would we log 4 total events or just 2?

we're most immediately interested in this info. for the purpose of learning things like: 1) the proportion of Suggestions people see and act on, 2) the proportion of people who engage with a particular type of Suggestion,

Do we plan to add new events to track engagement (e.g. clicks) with a specific check or suggestion? I know we have instrumentation available in VEFU to track engagement with edit checks and wondering if this will be expanded to include suggestion clicks.

cc @medelius

Does this mean we log an event once for every Check/suggestion seen or just once for every Check/suggestion type? For example, if 3 Add a Citation suggestions and 1 Revise tone suggestion are visible within an editing session, would we log 4 total events or just 2?

This currently logs once for each distinct instance of a check/suggestion. In your example we'd log up to 4 total events, assuming that someone clicked through and expanded all of those suggestions.

As a caveat, note that "distinct" means "not similar enough that we reused the same action object for it", so there's going to be some cases where what a human would call "the same" check gets logged twice. (Imagine: you have a text match check suggested on a word in a sentence; you cut that sentence and paste it in elsewhere; that counts as two distinct "seen" checks.)

Do we plan to add new events to track engagement (e.g. clicks) with a specific check or suggestion? I know we have instrumentation available in VEFU to track engagement with edit checks and wondering if this will be expanded to include suggestion clicks.

The existing engagement events will still apply. Any of the ones that explicitly reference check in their name should already be using suggestion when appropriate. There's some that were added which don't reference that distinction (the "learn more" event) and so are still generic. The events in the feedback form haven't been updated, because there's no way to reach them in suggestion mode.

There's also some old events which say "check" which aren't updated (window-open-from-check-[moment] just means "the check system" and will trigger whenever either type opens the sidebar, for instance).

Change #1236822 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] EditCheck: add instrumentation for checks seen during edit session

https://gerrit.wikimedia.org/r/1236822

Change #1237606 had a related patch set uploaded (by Medelius; author: Medelius):

[mediawiki/extensions/VisualEditor@wmf/1.46.0-wmf.14] EditCheck: add instrumentation for checks seen during edit session

https://gerrit.wikimedia.org/r/1237606

Change #1237606 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@wmf/1.46.0-wmf.14] EditCheck: add instrumentation for checks seen during edit session

https://gerrit.wikimedia.org/r/1237606

Mentioned in SAL (#wikimedia-operations) [2026-02-09T21:18:27Z] <zabe@deploy2002> Started scap sync-world: Backport for [[gerrit:1238031|Add alias for arwikibooks namespace]], [[gerrit:1237902|Change wgSiteName and wgMetaNamespace for Arabic Wikibooks (ويكي الكتب => ويكي كتب). (T416779)]], [[gerrit:1237606|EditCheck: add instrumentation for checks seen during edit session (T413419 T412334)]]

Mentioned in SAL (#wikimedia-operations) [2026-02-09T21:21:06Z] <zabe@deploy2002> sync-world aborted: Backport for [[gerrit:1238031|Add alias for arwikibooks namespace]], [[gerrit:1237902|Change wgSiteName and wgMetaNamespace for Arabic Wikibooks (ويكي الكتب => ويكي كتب). (T416779)]], [[gerrit:1237606|EditCheck: add instrumentation for checks seen during edit session (T413419 T412334)]] (duration: 02m 38s)

Change #1238033 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/VisualEditor@wmf/1.46.0-wmf.14] Revert^2 "EditCheck: add instrumentation for checks seen during edit session"

https://gerrit.wikimedia.org/r/1238033

Change #1238033 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@wmf/1.46.0-wmf.14] Revert^2 "EditCheck: add instrumentation for checks seen during edit session"

https://gerrit.wikimedia.org/r/1238033

Mentioned in SAL (#wikimedia-operations) [2026-02-09T23:26:40Z] <zabe@deploy2002> Started scap sync-world: Backport for [[gerrit:1238033|Revert^2 "EditCheck: add instrumentation for checks seen during edit session" (T413419 T412334)]], [[gerrit:1224793|Add MultiTitle to extension list (T404461)]]

Mentioned in SAL (#wikimedia-operations) [2026-02-09T23:51:27Z] <zabe@deploy2002> tbodt, zabe: Backport for [[gerrit:1238033|Revert^2 "EditCheck: add instrumentation for checks seen during edit session" (T413419 T412334)]], [[gerrit:1224793|Add MultiTitle to extension list (T404461)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-02-10T00:06:17Z] <zabe@deploy2002> Finished scap sync-world: Backport for [[gerrit:1238033|Revert^2 "EditCheck: add instrumentation for checks seen during edit session" (T413419 T412334)]], [[gerrit:1224793|Add MultiTitle to extension list (T404461)]] (duration: 39m 37s)

Started testing this instrumentation:

❌ 1. Events are being logged only the first time a suggestion is visible (expanded dialog) within VisualEditor within an edit session.

Events are being logged not just the first time a suggestion is visible but also the second time while backing out of progress workflow. Steps:

• Click on the "Yes" button on Add a Citation workflow
• Cross out the "Add a Citation" dialog

You will see another event for the same suggestion being visible again. Are we considering them as two separate instances of the same check?

✅ 2. Events are being logged only the first time a check is visible (expanded dialog) within VisualEditor within an edit session.
✅ 3. Events are being logged when any portion of the expanded suggestion is visible in the viewport.
✅ 4. Events are being logged when any portion of the expanded check is visible in the viewport.
✅ 5. Events include information on the type of suggestion/check.
✅ 6. On mobile, events are being logged when the suggestion appears in the drawer.
✅ 7. On mobile, events are being logged when the check card appears in the drawer.
✅ 8. On mobile, events are not being logged when just the gutter icon is visible
✅ 9. On mobile, when multiple checks are combined into a single icon in the gutter because they overlap, events are being logged for only the ones whose relevant content is actually visible
✅10. In pre-save (desktop and mobile) events are being logged any time a check is paged to.

Issue #1 there is being tracked as T417718.

1. Technically in that situation it's a "new" dialog/widget the second time around. As such, when it's expanded, editcheck (and the instrumentation) treats that as the first expansion (and thus marks it again as "seen").

All that to say, the underlying issue is with ReferenceCheck making that dialog disappear at all - and that's being resolved in T417718.

Thanks for the thorough testing, Rummana.