Page MenuHomePhabricator

Compare the revert risk scores with proposed criteria for vandalism
Closed, ResolvedPublic

Description

We want to avoid using revert risk scores for identifying potential vandalism and using that in our evaluations, as Automoderator itself will be using the revert risk. Instead, we are proposing a criteria on what usually makes an edit an vandalizing edit. The goal of this data is analyse how this compares with the revert scores and if we need to modify the criteria based on patterns observed.

Proposed criteria

  • Edit from account with <25 edits or anonymous user
  • Reverted by a different editor
  • Revert happens within 24 hours
  • Edit is in the main namespace

Event Timeline

KCVelaga_WMF changed the task status from Open to In Progress.Nov 20 2023, 9:33 AM
KCVelaga_WMF triaged this task as High priority.
KCVelaga_WMF updated the task description. (Show Details)

@Samwalton9-WMF

Based on the analysis, the following additions/modifications in addition to the initial criteria can improve the median risk score

  • Reverted within 12 hours
  • User edit count less 15 edits
  • Time since user's first edit is less than 48 hours
  • Absolute bytes difference is more than 5 bytes
wikiInitial+Reverted within 12 hours+User Edit Count <= 15 edits+Time Since First Edit <= 48 hrs+Absolute Bytes Diff >= 5 bytes
median_riskn_editsmedian_riskn_editsmedian_riskn_editsmedian_riskn_editsmedian_riskn_edits
dewiki0.901974168290.904239160770.904503160610.907555154680.91721411281
enwiki0.9106791725840.9122051624390.9128471608890.9151961538580.920194115997
eswiki0.922596551050.923474529220.92385526960.924792516960.93048339239
fawiki0.91636699670.91679292280.91805691360.92046885390.9243526734
frwiki0.903316193750.905588184010.906304182850.909034174890.91370913492
idwiki0.90246435540.90199432310.90289231900.90507130670.9100192361
itwiki0.919648234400.921301220770.921365220110.922709216330.92453315505
jawiki0.875682101700.87978994010.88011691090.88252588280.883676679
ptwiki0.91306433610.91436331470.91691630790.93066924580.9342281855
ruwiki0.914291235870.916403222500.916746222040.918103216610.92378816914
zhwiki0.88345475680.88698968800.88758868190.8903864810.8963374813

  • Restricting user related related metrics make minor improvements to the median risk, as majority of the reverted edits are made by anonymous users.
  • While having at least an n number of absolute bytes difference, improves the median risk, a substantial number of edits are elimiated, as compared to the initial criteria.
  • In addition to the time to revert, absolute bytes difference is only the control factor available for anonymous edits.

Here is the full analysis at various intervals for each dimension. We can discuss these results when we meet and decide on the adjustments.

After discussing with @Samwalton9-WMF

We have decided to finalize on the following criteria:

  • Edit from account with <15 edits or anonymous user
  • Reverted by a different editor
  • Reverted within 12 hours
  • User edit count less 15 edits
  • Time since user's first edit is less than 48 hours
  • Edit is in the main namespace

We decided not to add "Absolute bytes difference is more than 5 bytes" as we are not confident it will be a useful addition given the number of edits being eliminated from consideration, and subtle vandalism can be below 5 bytes.