Here is the pre and post analysis report reviewing the impact of the change in Reference Check logic on identified metrics.

Some key insights (copied from report):

• After the change in reference change logic, there was a 46.4% increase in the number of editing sessions where reference check was shown.
• Edit Completion Rate
  • There were no significant changes in edit completion rate overall or by platform.
  • Results also vary on a per wikis basis but all decreases in edit completion rate were less than 3%.
• False Positive Rate
  • Overall no significant changes in the proportion of edits that selected the "irrelevant" option for declining to add a reference (6.6% pre and 6.5% post).
  • However, we did observe a 5.4% increase in the proportion of edits that provided any reason for declining to add a reference.
  • The largest percent increase (12.7%) was for users that selected "Other" as the decline option.

• New Content Edit Revert Rate
  • There was 19% increase in new content revert rate following the deployment of the logic change. We also observed an increase in new content edits as identified by the editcheck-newcontent tag. Need to confirm if there were any changes in this tag that caused this increase.
  • By platform, there was a 6% increase on desktop and 10.7% increase on mobile.
  • We observed increases across all wikis where reference check was available as default during the reviewed timeframe except zhwiki and ptwiki.

• Absolute number of people who are adding a citation as a result of this rule change
  • There was a 10.2% increase in the number of distinct users that added a new content edit with a reference following the deployment of the reference change logic.
  • Increases were observed on both mobile and desktop platforms but we obsered the most significant increase in number of edits and users on mobile. There was a 65% increase in new content edits with a reference on mobile.
• Proportion of edits Reference Check was activated within that result in a new reference being added
  • There was -14.8% decrease in the proportion of new content edits with a reference following the deployment of the logic change.
  • While there was an increase in the total number of content edits with a reference following the change, there was a higher increase in the number of content edits without a reference.

Please see the report for additional breakdowns, charts, and other details.

cc @ppelberg

In T376345#10297576, @MNeisler wrote...

Wonderful work, @MNeisler. Per what we discussed offline today, our next steps are as follows:

1. Discuss these findings with the Editing Team and decide on whether to keep this change in logic in place or revert to the previous state (discussion scheduled for 20 November)
2. Publish these findings along with the decision we'll have converged on through "1)" on-wiki

In the meantime, a follow-up question for @DLynch...

• New Content Edit Revert Rate
  • There was 19% increase in new content revert rate following the deployment of the logic change. We also observed an increase in new content edits as identified by the editcheck-newcontent tag. Need to confirm if there were any changes in this tag that caused this increase.

David: what – if any – changes were made to the editcheck-newcontent tag between 3 June 2024 and 2 July 2024?

David: what – if any – changes were made to the editcheck-newcontent tag between 3 June 2024 and 2 July 2024?

I don't think there was any change to it in that time period.