Page MenuHomePhabricator

Clarify the meaning of the editcheck-references-activated tag
Closed, ResolvedPublic

Description

This task involves the work of deciding how we'll evolve the set of Edit Check tags, in the context of Multi-check deployment, to help us (volunteers and staff):

  1. See the full range of edits people publish in edit sessions where the Reference Check is shown
  2. Detect false positives. Read: identify edits where the Reference Check was shown when it does not make sense for it to have been.

Requirements

Current tag nameCurrent meaningRevised tag nameRevised meaningNotes
editcheck-reference-decline-uncertainTag applied to edits when people decline to add a reference and indicate they are not sure what citation to add as the reason for doing so.DeprecateDeprecateThis tag will be deprecated. Reason: this tag loses its meaning in the future where multiple Reference Checks have the potential of being shown within a single edit session.
editcheck-reference-decline-otherTag applied to edits when people decline to add a reference and select "Other" when prompted to share why they made this choice.DeprecateDeprecateSee above
editcheck-reference-decline-common-knowledgeTag applied to edits when people decline to add a reference and indicate the information they are adding is common knowledge when prompted to share why they made this choice.DeprecateDeprecateSee above
editcheck-reference-decline-irrelevantTag applied to edits when people decline to add a reference and indicate the information they are adding is common knowledge when prompted to share why they made this choice.DeprecateDeprecateSee above
editcheck-referencesTag applied to all edits that meet the conditions that could cause the reference edit check to be shown.No changes to tag nameNo changes to existing meaningBoth the name and meaning of this tag will remain as-is.
editcheck-newreferenceTag appended to all edits made using the visual editor to pages in the main namespace that involve an edit where people add a net new reference.No changes to existing nameNo changes to existing meaning
editcheck-newcontentTag applied to all edits in which new content is added. Where "new content" in this context is defined by the conditions that were defined in T324730 and are now codified in editcheck/modules/init.js.No changes to tag nameNo changes to existing meaning
editcheck-references-activatedTag applied to edits where the reference check is actually shown to people in the process of publishing an edit.editcheck-references-shownNo changes to existing meaningRevise tag name to make meaning more clear.

Done

Background

editcheck-references-activated is implemented in such a way that the tag is appended to all published edits if, at any point, during said browser pageview the Reference Check was shown.

This approach is helpful in so far as it enables us to see the full range of edits people publish in response to seeing the Reference Check...even when the edits people end up publishing should not have caused the Reference Check to become activated in the first place.

This approach can also be confusing in so far as it can be difficult to differentiate between the following two cases:

  1. The Reference Check was shown when it shouldn't have been
  2. The Reference Check was shown as expected and in response, someone revised their edit in a way that makes it look like the Reference Check was shown in error

i. E.g. maybe someone decides to "back out" of the Reference Check workflow and remove the new content they were seeking to add.

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Event Timeline

It might be less-overloaded to say "browser pageview" rather than "edit session". The latter sort of overlaps with some analytics terminology, and might imply that the tag is sticky if you move to a different article and start editing, which it is not.

It might be less-overloaded to say "browser pageview" rather than "edit session". The latter sort of overlaps with some analytics terminology, and might imply that the tag is sticky if you move to a different article and start editing, which it is not.

Good spot, David. Updated.

We absolutely could reset this flag when you exit VE, if that was the preferred behavior. (We could even reset it when you back out of the save process, though I think it'd become kind of meaningless then.)

Should we use this task to think about how Multi-check will handle multiple tagging?

Yeah, we very likely need to do a major adjustment to how this tagging works for that. Even before we have multiple *types* of check, having multiple instances of the same check will make this confusing enough.

Should we use this task to think about how Multi-check will handle multiple tagging?

Mmm, needing to converge on how the existing tagging approach will need to evolve to handle the complexities @DLynch described...great spot, @Trizek-WMF.

I'm thinking we use T342460 for converging on that broader approach and using this ticket to as an input to it.

I'm thinking we use T342460 for converging on that broader approach and using this ticket to as an input to it.

This task has a focus on databases. We need to think about tags in matters of how each history line will display that information. The more checks we will have the more crowded history pages will be.

Next steps
To ensure Edit Check tags remain useful and meaningful in a future state where multiple Edit Checks (of the same and/or different types have the potential to be shown within a single edit session, we are going to update the Edit Tags available in the following ways...

Current tag nameCurrent meaningRevised tag nameRevised meaningNotes
editcheck-reference-decline-uncertainTag applied to edits when people decline to add a reference and indicate they are not sure what citation to add as the reason for doing so.DeprecateDeprecateThis tag will be deprecated. Reason: this tag loses its meaning in the future where multiple Reference Checks have the potential of being shown within a single edit session.
editcheck-reference-decline-otherTag applied to edits when people decline to add a reference and select "Other" when prompted to share why they made this choice.DeprecateDeprecateSee above
editcheck-reference-decline-common-knowledgeTag applied to edits when people decline to add a reference and indicate the information they are adding is common knowledge when prompted to share why they made this choice.DeprecateDeprecateSee above
editcheck-reference-decline-irrelevantTag applied to edits when people decline to add a reference and indicate the information they are adding is common knowledge when prompted to share why they made this choice.DeprecateDeprecateSee above
editcheck-referencesTag applied to all edits that meet the conditions that could cause the reference edit check to be shown.No changes to tag nameNo changes to existing meaningBoth the name and meaning of this tag will remain as-is.
editcheck-newreferenceTag appended to all edits made using the visual editor to pages in the main namespace that involve an edit where people add a net new reference.reference-addedNo changes to existing meaningRevise tag name to make meaning more clear.
editcheck-newcontentTag applied to all edits in which new content is added. Where "new content" in this context is defined by the conditions that were defined in T324730 and are now codified in editcheck/modules/init.js.No changes to tag nameNo changes to existing meaning
editcheck-references-activatedTag applied to edits where the reference check is actually shown to people in the process of publishing an edit.editcheck-references-shownNo changes to existing meaningRevise tag name to make meaning more clear.
ppelberg moved this task from Ready to Be Worked On to Inbox on the Editing-team (Kanban Board) board.

@MNeisler: could you please give the ===Requirements section a quick review before Editing Engineering begins work on implementation?

The main things I'm eager to learn through you reviewing:

  1. When you consider A) the changes this ticket as proposing alongside B) the instrumentation we're implementing in T352092, what – if any – meaning/information are we currently logging that will no longer be available to us?
  2. What – if anything – about the changes proposed in this ticket would complicate the analyses we have planned?

When the tags are updated, I'll change the documentation at https://www.mediawiki.org/wiki/Edit_check/Tags.

We should also consider T353726: Change MediaWiki:Tag-editcheck-references-activated target while editing the tags.

When you consider A) the changes this ticket as proposing alongside B) the instrumentation we're implementing in T352092, what – if any – meaning/information are we currently logging that will no longer be available to us?

  • The primary change is that the non-deprecated tags cannot be used to determine the number of checks presented to a user within a single editing session or number of changes made in response. We were previously able to determine the net change from reference checks (#number of checks shown -> # of new content edits with change) using only revision tags as only one check was shown in a single editing session. We will now need to depend on instrumentation in VEFU (proposed in T352092) if we want to determine the number of checks shown and number of checks accepted within a single editing session.
    • I don't believe this should cause us to lose any information for Reference Check assuming that the edit check workflow still requires the user to engage with the check before they are allowed to save. In other words, we can safely assume that if a user clicks to accept a check and then publishes an edit, it means that they addressed the requested change to their edit.
    • If a user is not required to engage with the check before publishing or can click accept and publish without making the change, we will not know the exact number of changes made (i.e. new references added) to the published edit. We will just know that they added at least one new reference via the reference-added tag.
  • No concerns with removing the editcheck-reference-decline- tags as we will continue to track user decline selection within the VEFU instrumentation.

What – if anything – about the changes proposed in this ticket would complicate the analyses we have planned?

  • No significant complications; however, we will need to consider the 90-day data retention policy for any metrics that require us to join VEFU data to mediawiki_history. This would include revert rate broken down by the number of checks shown within a single session .
  • Note: With the proposed tag changes, we would still be able to effectively monitor longer-term revert rate trends based on the number of edits where edit check was activated at least once and reverted but could not review breakdowns by number of checks shown within a session. Those analyses would need to be done within 90 days.

When the tags are updated, I'll change the documentation at https://www.mediawiki.org/wiki/Edit_check/Tags.

Great spot, @Trizek-WMF – I've updated the task description to include this step.

We should also consider T353726: Change MediaWiki:Tag-editcheck-references-activated target while editing the tags.

Another great spot. Let's do it. I'll mark this ticket as a sub-task of T353726 and move T353726 to our upcoming work.

Thank you for reviewing this, @MNeisler. Everything that you described sounds good to me. Although, responses to each point you raised in line below...

When you consider A) the changes this ticket as proposing alongside B) the instrumentation we're implementing in T352092, what – if any – meaning/information are we currently logging that will no longer be available to us?

  • The primary change is that the non-deprecated tags cannot be used to determine the number of checks presented to a user within a single editing session or number of changes made in response.

I'm glad you stated this explicitly and what you described is expected to me.

We were previously able to determine the net change from reference checks (#number of checks shown -> # of new content edits with change) using only revision tags as only one check was shown in a single editing session. We will now need to depend on instrumentation in VEFU (proposed in T352092) if we want to determine the number of checks shown and number of checks accepted within a single editing session.

Understood!

  • I don't believe this should cause us to lose any information for Reference Check assuming that the edit check workflow still requires the user to engage with the check before they are allowed to save. In other words, we can safely assume that if a user clicks to accept a check and then publishes an edit, it means that they addressed the requested change to their edit.

Got it and to double confirm: we can, as you described, "safely assume that if a user clicks to accept a check and then publishes an edit, it means that they addressed the requested change to their edit." for Reference Check only.

Note: with the initial iterations of the Paste Check (T359107) and Peacock Check (T365301) we cannot assume that if someone accepts a Check that they implemented the change the Edit Check they are being shown invited them to make.

  • If a user is not required to engage with the check before publishing or can click accept and publish without making the change, we will not know the exact number of changes made (i.e. new references added) to the published edit. We will just know that they added at least one new reference via the reference-added tag.

Understood and see comment immediately above.

  • No concerns with removing the editcheck-reference-decline- tags as we will continue to track user decline selection within the VEFU instrumentation.

Excellent.

What – if anything – about the changes proposed in this ticket would complicate the analyses we have planned?

  • No significant complications; however, we will need to consider the 90-day data retention policy for any metrics that require us to join VEFU data to mediawiki_history. This would include revert rate broken down by the number of checks shown within a single session .

Great spot.

  • Note: With the proposed tag changes, we would still be able to effectively monitor longer-term revert rate trends based on the number of edits where edit check was activated at least once and reverted but could not review breakdowns by number of checks shown within a session. Those analyses would need to be done within 90 days.

Understood.

Per today's planning meeting, I'm assigning this ticket over to @DLynch to review the proposed Requirements and the review Megan posted of them.

NOTE: someone else on Editing Engineering may end up taking on the implementation work.

Requirements seem reasonable to me.

Arguably we might want to add some sort of editcheck-shown-presave and editcheck-shown-midedit tags to take over from editcheck-references-activated if we want to support reviewers in the coming multiple-check-types future. For our current instrumentation needs that's superfluous, though.

Requirements seem reasonable to me.

Arguably we might want to add some sort of editcheck-shown-presave and editcheck-shown-midedit tags to take over from editcheck-references-activated if we want to support reviewers in the coming multiple-check-types future. For our current instrumentation needs that's superfluous, though.

Great spot and I agree with what I understand you to be proposing here: consider adding tags that enable us to identify what moment(s) within an editing session Edit Checks are presented within. Let's use T387936 to track this work.

Change #1124851 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Edit check: update tagging

https://gerrit.wikimedia.org/r/1124851

ppelberg updated the task description. (Show Details)

Change #1124851 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Edit check: update tagging

https://gerrit.wikimedia.org/r/1124851

Where the information about what answer editor choose will be available to all users after this change?

->T388920
->https://www.mediawiki.org/wiki/Talk:Edit_check#c-PPelberg_(WMF)-20250314191100-Matěj_Suchánek-20250312170700

Verified all the tags on the table. All the deprecated tags are correctly not being applied anymore to relevant edits. However, I am not seeing the editcheck-references-activated being revised as editcheck-references-shown as mentioned on the table in the task description here: https://en.wikipedia.beta.wmflabs.org/w/index.php?hidebots=1&hidecategorization=1&hideWikibase=1&tagfilter=editcheck-references-activated&limit=50&days=7&title=Special:RecentChanges&urlversion=2

Also, on Beta cluster, it seems no tags are being added to the edits where the reference check is actually shown to people in the process of publishing an edit.

Change #1130775 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Edit check: add editcheck-references-shown to the allowed tags list

https://gerrit.wikimedia.org/r/1130775

Change #1130775 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Edit check: add editcheck-references-shown to the allowed tags list

https://gerrit.wikimedia.org/r/1130775

Change #1131003 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@wmf/1.44.0-wmf.22] Edit check: add editcheck-references-shown to the allowed tags list

https://gerrit.wikimedia.org/r/1131003

Change #1131003 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@wmf/1.44.0-wmf.22] Edit check: add editcheck-references-shown to the allowed tags list

https://gerrit.wikimedia.org/r/1131003

Mentioned in SAL (#wikimedia-operations) [2025-03-25T14:43:46Z] <kemayo@deploy1003> Started scap sync-world: Backport for [[gerrit:1131003|Edit check: add editcheck-references-shown to the allowed tags list (T373949)]], [[gerrit:1131004|Edit check: don't close the sidebar on context change on desktop (T389906)]]

Mentioned in SAL (#wikimedia-operations) [2025-03-25T14:50:18Z] <kemayo@deploy1003> kemayo: Backport for [[gerrit:1131003|Edit check: add editcheck-references-shown to the allowed tags list (T373949)]], [[gerrit:1131004|Edit check: don't close the sidebar on context change on desktop (T389906)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-03-25T15:01:16Z] <kemayo@deploy1003> Finished scap sync-world: Backport for [[gerrit:1131003|Edit check: add editcheck-references-shown to the allowed tags list (T373949)]], [[gerrit:1131004|Edit check: don't close the sidebar on context change on desktop (T389906)]] (duration: 17m 30s)

Confirming that, editcheck-references-activated now got correctly revised as editcheck-references-shown and the tag is being applied to all the edits where the reference check is actually shown to people in the process of publishing an edit.