Page MenuHomePhabricator

Introduce a tag to identify edits that meet the Edit Check heuristic
Closed, ResolvedPublic

Description

This task involves the work of introducing a new hidden change tag that will be appended to edits that meet the conditions defined in T324730.

This hidden tag will enable us (volunteers and members of the Editing Team) to evaluate the extent to which the reference check heuristic would be initiated in expected cases.
Where "expected" in this context means an acceptable level of false positives and negatives.

Knowing the above will enable us to decide what – if any – adjustments (T327959) ought to be made to the said heuristic (T324730).

User Story

As someone (e.g. members of the Editing Team and volunteers) motivated to ensure that Edit Check is prompting people to add references in cases where references are in fact needed, I need be able see the edits (on a per project basis) that meet the criteria defined in T324730 so that I can propose adjustments to said criteria to minimize the rate of false positives and, eventually, false negatives.

Requirements

  1. A hidden change tag is appended/associated with all edits, regardless of the editing interface used to make said edits that meet the conditions from T324730:
  2. This hidden change tag should be called: editcheck-references
  3. The logic that determines whether the change tag this task is describing needs to be able to be kept in sync with the logic that we're likely to iterate upon by way of T327959

Done

  • All "Requirements" are met

Related Objects

Event Timeline

ppelberg moved this task from Untriaged to Upcoming on the Editing-team board.
ppelberg moved this task from To Triage to Triaged on the VisualEditor board.
VPuffetMichel renamed this task from Introduce a tag to identify edits made when an edit check was activated to [edit check] Introduce a tag to identify edits made when an edit check was activated.Dec 8 2022, 1:37 PM
ppelberg renamed this task from [edit check] Introduce a tag to identify edits made when an edit check was activated to Introduce a tag to identify edits that cause reference check to be initiated.Feb 18 2023, 12:50 AM
ppelberg updated the task description. (Show Details)
ppelberg edited projects, added Editing-team (Kanban Board); removed Editing-team.

On reflection, I suggest we consider removing requirement 1B ("the new paragraph does not include a reference"), because:

  1. It may be helpful to examine cases where our conditions apply, but a reference has been added, e.g. to evaluate our heuristics.
  2. We can always just filter out any tagged edits that do contain a reference, effectively applying 1B ourselves.

On reflection, I suggest we consider removing requirement 1B ("the new paragraph does not include a reference"), because:

  1. It may be helpful to examine cases where our conditions apply, but a reference has been added, e.g. to evaluate our heuristics.

@dchan: can you say a bit more about the above? Asked another way: what do you see us potentially inhibiting ourselves from learning/evaluating were we to continue with the plan NOT to cause Edit Check to become initiated for edits that involve people adding new content and accompanying said new content with a reference?

Oh, probably I'm not being clear. I was thinking:

  • we want to tag edits that should contain a reference (according to our heuristic)
  • regardless whether or not they do contain one
  • and regardless whether the reference check UI is deployed

That would let us obtain useful things like:

  • The proportion of tagged edits that already come with references (even before the reference check UI is deployed).
  • A sample of tagged edits that do have references, and a sample of tagged edits that don't. Through comparing these two samples, we can learn if there are important cases where our heuristic tends to give false positives (saying references are needed when they're not).

Also, wouldn't requirement 1B would mean that once the reference check UI is deployed, we'd never flag an edit if the user actually adds a reference through the UI? Or am I misunderstanding something?

ppelberg renamed this task from Introduce a tag to identify edits that cause reference check to be initiated to Introduce a tag to identify edits that meet the Edit Check heuristic .Mar 8 2023, 10:30 PM
ppelberg updated the task description. (Show Details)

Change 900378 had a related patch set uploaded (by Bartosz Dziewoński; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Add editcheck tags when VE thinks a reference is needed

https://gerrit.wikimedia.org/r/900378

Change 900378 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Add editcheck tags when VE thinks a reference is needed

https://gerrit.wikimedia.org/r/900378

Change 903664 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Change name of the editcheck-needreference tag to editcheck-references

https://gerrit.wikimedia.org/r/903664

Change 903664 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Change name of the editcheck-needreference tag to editcheck-references

https://gerrit.wikimedia.org/r/903664

Change 903759 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Enable hidden tag for "Edit Check" project on Wikipedias

https://gerrit.wikimedia.org/r/903759

Change 903759 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable hidden tag for "Edit Check" project on Wikipedias

https://gerrit.wikimedia.org/r/903759

Mentioned in SAL (#wikimedia-operations) [2023-03-28T20:34:22Z] <urbanecm@deploy2002> Started scap: Backport for [[gerrit:903684|Only run edit check on main namespace]], [[gerrit:903685|Change name of the editcheck-needreference tag to editcheck-references]], [[gerrit:903759|Enable hidden tag for "Edit Check" project on Wikipedias (T324733)]]

Mentioned in SAL (#wikimedia-operations) [2023-03-28T20:51:10Z] <urbanecm@deploy2002> urbanecm and matmarex: Backport for [[gerrit:903684|Only run edit check on main namespace]], [[gerrit:903685|Change name of the editcheck-needreference tag to editcheck-references]], [[gerrit:903759|Enable hidden tag for "Edit Check" project on Wikipedias (T324733)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-03-28T21:03:15Z] <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:903684|Only run edit check on main namespace]], [[gerrit:903685|Change name of the editcheck-needreference tag to editcheck-references]], [[gerrit:903759|Enable hidden tag for "Edit Check" project on Wikipedias (T324733)]] (duration: 28m 53s)

Change 904700 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Revert "Enable hidden tag for "Edit Check" project on Wikipedias"

https://gerrit.wikimedia.org/r/904700

Change 904700 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Enable hidden tag for "Edit Check" project on Wikipedias"

https://gerrit.wikimedia.org/r/904700

Mentioned in SAL (#wikimedia-operations) [2023-03-31T16:02:30Z] <ladsgroup@deploy2002> Started scap: Backport for [[gerrit:904700|Revert "Enable hidden tag for "Edit Check" project on Wikipedias" (T324733 T333612)]]

Mentioned in SAL (#wikimedia-operations) [2023-03-31T16:03:50Z] <ladsgroup@deploy2002> matmarex and ladsgroup: Backport for [[gerrit:904700|Revert "Enable hidden tag for "Edit Check" project on Wikipedias" (T324733 T333612)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-03-31T16:10:49Z] <ladsgroup@deploy2002> Finished scap: Backport for [[gerrit:904700|Revert "Enable hidden tag for "Edit Check" project on Wikipedias" (T324733 T333612)]] (duration: 08m 18s)

Change 905263 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] EditCheck: catch errors from TransactionSquasher

https://gerrit.wikimedia.org/r/905263

Change 905263 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] EditCheck: catch errors from TransactionSquasher

https://gerrit.wikimedia.org/r/905263

Change 905685 had a related patch set uploaded (by Bartosz Dziewoński; author: DLynch):

[mediawiki/extensions/VisualEditor@wmf/1.41.0-wmf.2] EditCheck: catch errors from TransactionSquasher

https://gerrit.wikimedia.org/r/905685

Change 905727 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Revert "Revert "Enable hidden tag for "Edit Check" project on Wikipedias""

https://gerrit.wikimedia.org/r/905727

Change 905727 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Revert "Enable hidden tag for "Edit Check" project on Wikipedias""

https://gerrit.wikimedia.org/r/905727

Mentioned in SAL (#wikimedia-operations) [2023-04-04T20:03:36Z] <samtar@deploy2002> Started scap: Backport for [[gerrit:905727|Revert "Revert "Enable hidden tag for "Edit Check" project on Wikipedias"" (T324733)]]

Mentioned in SAL (#wikimedia-operations) [2023-04-04T20:05:02Z] <samtar@deploy2002> matmarex and samtar: Backport for [[gerrit:905727|Revert "Revert "Enable hidden tag for "Edit Check" project on Wikipedias"" (T324733)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-04-04T20:11:07Z] <samtar@deploy2002> Finished scap: Backport for [[gerrit:905727|Revert "Revert "Enable hidden tag for "Edit Check" project on Wikipedias"" (T324733)]] (duration: 07m 30s)

Change 905685 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@wmf/1.41.0-wmf.2] EditCheck: catch errors from TransactionSquasher

https://gerrit.wikimedia.org/r/905685

Mentioned in SAL (#wikimedia-operations) [2023-04-04T20:18:57Z] <samtar@deploy2002> Started scap: Backport for [[gerrit:905685|EditCheck: catch errors from TransactionSquasher (T324733)]]

Mentioned in SAL (#wikimedia-operations) [2023-04-04T20:20:21Z] <samtar@deploy2002> matmarex and samtar: Backport for [[gerrit:905685|EditCheck: catch errors from TransactionSquasher (T324733)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-04-04T20:27:20Z] <samtar@deploy2002> Finished scap: Backport for [[gerrit:905685|EditCheck: catch errors from TransactionSquasher (T324733)]] (duration: 08m 23s)

matmarex updated the task description. (Show Details)