Page MenuHomePhabricator

Log sub-types of textmatch checks to VEFU
Closed, ResolvedPublic

Description

In T407170, we are introducing a view that will enable us to, among other things, look at how specific TextMatching suggestions (T398478) perform.

To enable this, we'll need to log data about specific TextMatch implementations rather than the aggregating metrics about – what might be – tens or hundreds of different TextMatch suggestions into one broad category as is currently done.

Story

As a member of the Editing Team motivated to introduce TextMatch-based Edit Suggestions that volunteers find compelling and constructive, I'd like be able to see and track how individual suggestions are performing, so that I can decide 1) what new Suggestions to introduce, 2) what suggestions to refine further, 3) what suggestions to deactivate

Requirements

When a user/admin adds a new TextMatch suggestion (matchItem) to the TextMatch config, we will require it to have a unique id, such as potentialLLM or idioms.
Any time an editor selects an action (accept, dismiss, etc.) in the TextMatchEditCheck dialog, we can log the matchItem and their action as textmatch-[matchitemUID]-[action].

Done

  • Requirements drafted
  • @MNeisler to review + approve proposed requirements
  • Implement requirements
  • On wikis where TextMatch is already available within MediaWiki:Editcheck-config.json, update file to include new syntax this ticket introduces

Event Timeline

Currently we have a method of creating a "tag name" for the rules, but that's just based on their numeric index in the config. This is good for us because it doesn't require the users to work out a unique identifier, but it's also very fragile for persistent references between edit sessions.

A main confusion with this data is going to be that it'll be tied to the state of the on-wiki config at the moment it's logged, so there's going to be some need to manually investigate it to work out what's actually meant by any given log entry and also we won't be able to generalize the meanings between wikis.

Currently we have a method of creating a "tag name" for the rules, but that's just based on their numeric index in the config. This is good for us because it doesn't require the users to work out a unique identifier, but it's also very fragile for persistent references between edit sessions.

A main confusion with this data is going to be that it'll be tied to the state of the on-wiki config at the moment it's logged, so there's going to be some need to manually investigate it to work out what's actually meant by any given log entry and also we won't be able to generalize the meanings between wikis.

The more I think about matchItem specific logging, the more I’m liking the idea of requiring users to provide a unique identifier. I think it's a worthwhile ask; minimal effort but would greatly help with logs and persistence of references between edit sessions.

Alternatively, we could require just a unique title, and then generate the id ourselves from the concatenated, camel-cased version of it, though that might get ugly.

Both of these would still be somewhat tied to the state of the on-wiki config, by being tied to the state of that specific matchitem in the on-wiki config (query could change, etc). But I imagine it would be more digestible.

Change #1204696 had a related patch set uploaded (by Medelius; author: Medelius):

[mediawiki/extensions/VisualEditor@master] TextMatchEditCheck: Add sub-type logging

https://gerrit.wikimedia.org/r/1204696

Change #1204696 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] TextMatchEditCheck: Add sub-type logging

https://gerrit.wikimedia.org/r/1204696

Change #1206971 had a related patch set uploaded (by Medelius; author: Medelius):

[mediawiki/extensions/VisualEditor@master] TextMatchEditCheck: undo duplicate sub-type logging

https://gerrit.wikimedia.org/r/1206971

Change #1206971 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] TextMatchEditCheck: undo duplicate sub-type logging

https://gerrit.wikimedia.org/r/1206971

Change #1207201 had a related patch set uploaded (by DLynch; author: Medelius):

[mediawiki/extensions/VisualEditor@wmf/1.46.0-wmf.3] TextMatchEditCheck: undo duplicate sub-type logging

https://gerrit.wikimedia.org/r/1207201

Change #1207201 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@wmf/1.46.0-wmf.3] TextMatchEditCheck: undo duplicate sub-type logging

https://gerrit.wikimedia.org/r/1207201

Mentioned in SAL (#wikimedia-operations) [2025-11-19T22:05:55Z] <kemayo@deploy2002> Started scap sync-world: Backport for [[gerrit:1207201|TextMatchEditCheck: undo duplicate sub-type logging (T407286)]], [[gerrit:1207245|Remove action_context from page_load events in ReadingList A/B test (T410535)]], [[gerrit:1207246|Remove action_context from page_load events in ReadingList A/B test (T410535)]], [[gerrit:1207241|README: remove outdated advice about dblists]]

Mentioned in SAL (#wikimedia-operations) [2025-11-19T22:10:56Z] <kemayo@deploy2002> aude, kemayo, novemlinguae: Backport for [[gerrit:1207201|TextMatchEditCheck: undo duplicate sub-type logging (T407286)]], [[gerrit:1207245|Remove action_context from page_load events in ReadingList A/B test (T410535)]], [[gerrit:1207246|Remove action_context from page_load events in ReadingList A/B test (T410535)]], [[gerrit:1207241|README: remove outdated advice about dblists]] synced to the tests

Mentioned in SAL (#wikimedia-operations) [2025-11-19T22:16:52Z] <kemayo@deploy2002> Finished scap sync-world: Backport for [[gerrit:1207201|TextMatchEditCheck: undo duplicate sub-type logging (T407286)]], [[gerrit:1207245|Remove action_context from page_load events in ReadingList A/B test (T410535)]], [[gerrit:1207246|Remove action_context from page_load events in ReadingList A/B test (T410535)]], [[gerrit:1207241|README: remove outdated advice about dblists]] (duration: 10m 57s)

@medelius: I was checking this on Beta cluster today. I am noticing every time I select an action (accept, dismiss, etc.) in the TextMatchEditCheck dialog, I see along with suggestion-action-delete, lot of other events with action: suggestion-shown-midedit being emitted again for other existing TextMatchEditCheck that's on the page.

Screenshot 2025-11-19 at 8.31.17 PM.png (1×2 px, 974 KB)

Are these expected?

Thanks for sharing @Ryasmeen. Let me look into this and get back to you.