Page MenuHomePhabricator

Make DiscussionTool dirty diff detection more accessible
Open, Needs TriagePublic

Description

This task is about enhancing the tool @matmarex built in T260393#6384098 for detecting when edits made with DiscussionTools make changes to the page other than adding content.

Background

A key principle of the Talk pages project (an outcome of the 2019 Talk page consultation) is making it easier and more efficient for people, across experience levels, to communicate with others on Wikipedia. To deliver on this, we are introducing new communication tools...communication tools that are required to be backwards compatible with existing Wikitext talk pages.

Core to this "backwards compatibility," is ensuring these new communication tools do not affect existing content in undesirable ways. The script Bartosz wrote enables us to detect suspicious diffs at scale and thus validate whether these new tool are in fact delivering on the agreement established in the Talk page consultation.

Trouble is, as the Reply Tool and New Discussion Tool is made more widely available, the number of suspicious diffs this script identifies will likely grow beyond a single person's ability to review them, thus this task.

Meta

Objectives

  • The Editing Team can more accurately prioritize issues knowing how frequently/infrequently they occur
  • The Editing Team can identify issues earlier and decrease the likelihood edits produced by DiscussionTools disrupt editors

Target user(s)

Requirements

Meta

  • "Script" [i] can be configured to run at a to-be-defined interval without any individual needing to manually start it
  • The script's output can be read in the browser without needing to install additional libraries
  • The script should look review all edits made with DiscussionTools, regardless of how DiscussionTools have been deployed at that project. Read: the script should identify edits whether they were made on projects where DT are available as opt-out preference, as a Beta Feature or not at all (someone used DT by appending ?dtenable=1 to the URL of a discussion page).

Consuming script's output

  • The page should include metadata to help the Editing Team know:
    • How many total DiscussionTool edits have been analyzed to-date
    • Of the DiscussionTool edits the script has "analyzed" to-date, the number of diffs the script has tagged as "suspicious"
  • When the table of suspicious edits was last updated (read: when the script was last run)
  • The page should contain a list of diffs the script has identified as suspicious
    • Each suspicious diff should have the following values associated with it:
      • The URL of the project on which the diff was made
      • The URL of the diff
      • The time the edit producing the diff was made

Open questions

  • Is it necessary to tag suspicious edits as they happen vs. identifying them en masse at run-time, as is currently being done?
  • Is it necessary to implement any kind of time out handling?

Done

  • All "Open questions" are implemented
  • All "Requirements" are met

i. "Script" is used here as a general term; it may not be the best way to describe the code that will need to be written to deliver on the required functionality


Ideas for the future

  • A way to relate diffs to each other to help categorize issues
  • A way to triage suspicious diffs. E.g. mark "Archive" or "Investigate"
  • A way to "tune" the script such that categories of diffs deemed not to be harmful can be ignored in the future

Event Timeline

ppelberg created this task.Thu, Sep 3, 11:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Sep 3, 11:14 PM
ppelberg updated the task description. (Show Details)EditedFri, Sep 11, 11:57 PM

Update: 11-Sep
Yesterday, @DLynch, @Esanders, @matmarex, @Whatamidoing-WMF and I met to discuss the value and feasibility of implementing what this task is asking for: a more scalable way of reviewing potentially problematic edits produced by DiscussionTools.

In this meeting we "agreed" and "wondered about" the following...

Agreed

  • It is premature, and potentially not valuable, to build this tool for use by people outside of the Editing Team
  • Being able to programmatically identify and review potentially problematic diffs would be valuable for the following reasons:
    • We can more accurately prioritize issues know that we'll have an understanding for how frequently/infrequently they occur
    • We can identify issues earlier and decrease the likelihood edits produced by DiscussionTools disrupt editors
  • An MVP would include the following:
    • A table of potentially problematic edits that is automatically updated at some defined interval and available to view in the browser

Wondered

  • Is it necessary to tag problematic edits as they happen vs. identifying them en masse at run-time, as is currently being done?
  • Is it necessary to implement any kind of time out handling?

I've updated the task description to include the above.

This comment was removed by Esanders.

Change 626848 had a related patch set uploaded (by Esanders; owner: Esanders):
[mediawiki/extensions/DiscussionTools@master] Add a hidden tag to replies which modify other parts of the page

https://gerrit.wikimedia.org/r/626848

The above patch adds a hidden tag to any reply edit that modifies another section (same detection as Bartosz's script). With this approach, users can monitor edits via RecentChanges. If we were to make a tool to aggregate all wikis this would be made easier too.

ppelberg updated the task description. (Show Details)Tue, Sep 15, 2:11 AM