Page MenuHomePhabricator

Create a system to encode best practices into editing experiences
Open, Needs TriagePublic

Assigned To
None
Authored By
ppelberg
Oct 9 2020, 6:01 PM
Referenced Files
F35060442: Screen Shot 2022-04-20 at 11.52.10 AM.png
Apr 20 2022, 6:52 PM
F35060440: image.png
Apr 20 2022, 6:51 PM
Tokens
"Orange Medal" token, awarded by Krinkle."Like" token, awarded by Sdkb."Love" token, awarded by Trizek-WMF.

Description

🌱 In the forest that is Phabricator, this ticket is very much a seedling on the forest floor. Read: this task is a gathering place/work in progress.


This parent task is intended to help gather and advance the thinking around how the visual editor might be enhanced to help people learn the social conventions (read: policies and guidelines) and exercise the judgement necessary to become productive and constructive Wikipedia editors.

Background

Visual editor's growing popularity among people new to editing Wikipedia [i] suggests it has been reasonably successful at helping people to learn the technical skills [ii] necessary to edit Wikipedia.

Trouble is, the edits these new people make often break/defy Wikipedia policies and guidelines.

This task is about exploring how the visual editor could be augmented/enhanced to help people learn these policies and guidelines and exercise the judgement necessary to become productive and constructive Wikipedia editors.

Potential impact

New and Junior Contributors
Working on talking pages has led us to notice that for many newcomers, the earliest human interactions they have on-wiki centers around something they did wrong, like not contributing in the Wikipedia Way [iii] [iv][v][vi][viii].

We wonder how newcomers perceive these interactions and further, whether newcomers having a positive first interaction with another Wikipedia editor could increase the likelihood that they continue editing. [vii]

Senior Contributors
We wonder whether adding more "productive friction" to publishing flows could free Senior Contributors up to do more high-value work by:

  • Reducing the amount of work they need to do reverting lower quality edits by increasing the quality of edits
  • Reducing the amount of work they need to do blocking bad faith/vandalistic editors by lowering the likelihood these actors are able to complete/publish the changes that cause them to be blocked
  • Reducing the amount of work and time they spend writing on new users' talk pages to alert them of policies and/or guidelines they've [likely] unknowingly broken
  • Reducing the amount of time and energy they need to spend worrying about protecting the wikis. This could potentially relieve them of mental space to address new challenges.

Use cases

More in T284465.

Components

  • A way to codify rules and policies that machines could "read" / "interpret"
    • To start, a team like Editing could hardcode these rules and policies on a one-off basis. At scale, if/when the concept proves valuable, we envision a way for volunteers to write custom rules based on the consensus reached at their respective projects.
  • A way for the editing interface to locate where content exists that violate these rules and policies
    • Note: "content" in this context refers to content that has already been published on the wiki and content that volunteers have added and have not yet published.
  • A way to surface improvement suggestions to volunteers who have an editing interface open.
    • E.g. "This is the issue. This is where the issue exists within the artifact (read: article, talk page comment you are drafting, etc.). This is what you can do to remedy the issue."
  • A way to make volunteers aware that issues exist within content they are reading / have not yet started to edit.

Also see the ===Components section in T276857.

Naming ideas

  • Policy Check
  • Intelligent edits / Edit intelligence
    • I'm attracted to the idea of framing this as infusing the collective intelligence of the wikis into the editing interfaces.
  • Edit assistance
  • Assisted edits
  • Augmented edits

References

Related ticket

Data

Policy documentation

Research

  • Proposed: AI-Models-For-Knowledge-Integrity
  • Automatically Labeling Low Quality Content on Wikipedia By Leveraging Patterns in Editing Behaviors via @Halfak in T265163#7622952 non paywall link
  • Conversations Gone Awry: Detecting Early Signs of Conversational Failure
  • Wikipedian Self-Governance in Action: Motivating the Policy Lens
  • Twitter Prompts Findings
    • "If prompted, 34% of people revised their initial reply or decided to not send their reply at all."
    • "After being prompted once, people composed, on average, 11% fewer offensive replies in the future."
    • "If prompted, people were less likely to receive offensive and harmful replies back."
  • Unreliable Guidelines: Reliable Sources and Marginalized Communities in French, English and Spanish Wikipedias
    • "A new editor wrote in an email that they perceived Wikipedia’s reliable source guidelines to have exclusionary features."
    • "...community consensus is a foundational pillar of the Wikimedia movement. We learned trainers see this process as privileging those who participated first in the development of the encyclopedia’s editorial back channels. As well, the participants in our community conversations were uncomfortable with the presumption that agreement is communicated through silence, which privileges those who have the time and feel comfortable speaking up and participating in editorial conversations."
    • "In English, contributors from English-speaking countries in Africa said their contributions often faced scrutiny. One organizer from an unnamed African country who participated in our session said when they hosted events, contributions were deleted en-mass for lacking reliability. This was demoralizing for the participants and required extra work by the trainers to stand up for their publications and citations, said one participant...To avoid new editors experiencing these disappointing responses, other trainers in English Wikipedia explained they would review sources before new editors begin editing."
    • "...the quantity of material that a trainer is required to parse in relation to reliable source guidelines is immense. As one participant said:"
      • "This bushy structure makes the guidelines pages unreadable. Who has the time and the meticulousness to read it completely without being lost to a certain point? It took me a decade to go through it [in French] and I must admit I’m not done yet!"

Ideas

  • Ideas for where/how we might introduce this feedback/interaction: T95500#6873217

Related on-wiki tools

On-wiki documentation

Related third-party tools

Open questions

  • 1. Responsibility: How much responsibility should the software lead people to think they have for "correcting" the issues within the content they're editing?
  • 2. Authority: How might this capability shift editors' perception of who is the authority on what's "best"? Might this tool cause people to listen the software more than they do fellow editors?
  • 3. Audibility: How will this tool adapt/cope with the fact that policies and guidelines are constantly evolving?
  • 4. Reverts: Might this capability be impactful for people whose edits have been reverted?

Note: I've added the above to T265163's newly-created ===Open questions section.


i. https://superset.wikimedia.org/r/345
ii. Where "technical skills" could mean: adding text, formatting text, adding links, adding citations, publishing edits, etc.
iii. https://www.mediawiki.org/wiki/New_Editor_Experiences#Research_findings
iv. https://w.wiki/eMc
v. https://w.wiki/dSx
vi. https://w.wiki/dSv
vii. Perhaps the Research Team's "Thanks" research might be instructive here for it explore how positive interactions/sentiment affect editing behavior: https://meta.wikimedia.org/wiki/Research:Understanding_thanks
viii. https://en.wikipedia.org/w/index.php?title=User_talk:Morgancurry0430&oldid=1038113933

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenOTichonova
OpenNone
OpenNone
OpenFeatureNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
Openppelberg
OpenNone
OpenNone
Resolvedppelberg
OpenDLynch
OpenFeatureNone
Resolvedppelberg
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ppelberg updated the task description. (Show Details)
  • ADDED the work @NRodriguez and the Community-Tech team are doing to discourage editors from linking to disambiguation pages to the task description's === Use cases section
ppelberg updated the task description. (Show Details)
  • ADDED the proof of concept @NRodriguez and the Community-Tech team are developing to make people aware, in real-time, when they've added a link to a disambiguation page: T288589.

Thanks for adding me as a subscriber! I just made https://en.wikipedia.org/wiki/Wikipedia:Making_editing_easier_2021 today, but there's nothing there yet. I've seen some community interest, though. Hopefully we can have some useful things to say at some point.

Meta
I've added the ===Components section to the task description based on the conversation @Esanders and I had today.

@Sumit recently wrote a paper (https://dl.acm.org/doi/abs/10.1145/3479503) about building AIs that learn how to highlight content that is likely to need specific types of clean-up by learning directly from past edits. E.g. sentences that get edited for NPOV reasons tend of have a specific set of issues. The model learns those issues and then can be used to flag the same types of issues in new sentences. In effect, this encodes policy directly into the context in which someone is editing. It could be handy, so I'm bringing it up and pinging Sumit for comment :)

In T261352 a proposal was made for Odia Wikipedia to encourage editors to check the citing sources policy and encourage them to fix typos and grammatical errors more proactively:

[...]
The link to the help page "WP:CS" can be added with an icon and a text "ଲିଖନ ଶୈଳୀରେ ସହାୟତା?" ([Need] help with the writing style?) linking to the help page will be of great use.
There is a new education program -- or something of that sort -- going on that the community is unaware of, and we're experiencing influx of really long articles (as long as 40K words, the whole content created using probably Google Translate resulting extremely complex and gibberish sentences that will take really long time for even long time and experienced editors to fix). My assumption is that at least having a link to the help page might prompt some new editors to read through a rather brief details on the dos and don'ts of MoS and improve their writing before submitting.
[...]
When they click on PUBLISH, the tool can proactively ask a question "ଅନୁବାଦରେ ବନାନ ଓ ବ୍ୟାକରଣଗତ ଭୁଲସବୁ ସଜାଡ଼ିଛନ୍ତି ତ?" (have you fixed all the typos and grammatical errors in the translation) with a "ହଁ" (yes) "OR" "ନାଁ, ମୁଁ ବର୍ତ୍ତମାନ ପଢ଼ି ସଜାଡ଼ିଦେଉଛି" (no, let me read and fix [those mistakes]) where the latter option can take them back to the editing field to fix any mistakes.

The proposal suggests to add links to the documentation and prompt the user to review contents, but a contextual solution such as the one described in the current ticket may be more effective to support this.

I spoke to an editor from Tamil Wikipedia last week in the context of content moderation needs and they expressed a desire for editors to receive popups or prompts about basic formatting elements of the page when writing a new article. They mentioned title formatting, categories, having at least one interwiki link, and references, as potential areas of focus.

@Sumit recently wrote a paper (https://dl.acm.org/doi/abs/10.1145/3479503) about building AIs that learn how to highlight content that is likely to need specific types of clean-up by learning directly from past edits. E.g. sentences that get edited for NPOV reasons tend of have a specific set of issues. The model learns those issues and then can be used to flag the same types of issues in new sentences. In effect, this encodes policy directly into the context in which someone is editing. It could be handy, so I'm bringing it up and pinging Sumit for comment :)

@Halfak: "handy" sounds like an understatement :) Anywho, what you're describing above sounds precisely like the kind of thing this ticket will depend on; I'm grateful that you came here to share this. Once I've read the paper, I'll likely have questions for you and @Sumit.

In the meantime, two quick questions:

  1. Would it you be okay with me uploading a PDF copy of the paper in case anyone else happening upon this ticket would like to read it as well? I thankfully have access to it via the The Wikipedia Library.
  2. Do you recall how you come to learn about this ticket? No worries if you don't remember...I ask this curious to learn about spaces where this kind of thing is being discussed.
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg removed a subscriber: Quiddity.
ppelberg added a subscriber: Quiddity.

In T261352 a proposal was made for Odia Wikipedia to encourage editors to check the citing sources policy and encourage them to fix typos and grammatical errors more proactively...

@Pginer-WMF the immediate + in-context guidance this ticket is describing sounds like a great fit for helping newer people publish articles that are more readable and in-line with project policies...I'm glad you stopped by to mention this need. I've added this to the task description's ===Use Cases section. [i]


i. T265163#7726311

I spoke to an editor from Tamil Wikipedia last week in the context of content moderation needs and they expressed a desire for editors to receive popups or prompts about basic formatting elements of the page when writing a new article. They mentioned title formatting, categories, having at least one interwiki link, and references, as potential areas of focus.

Nice one, @Samwalton9! If you recall, I'd be keen to hear what led this volunteer to share these "particular areas of focus"? E.g. are these the mistakes they have personally noticed newcomers often making? Is there a particular policy at ta.wiki that includes a checklist of sorts for new articles? Etc.

In the meantime, I've added a new "Writing new articles" row to the task description's ===Use cases table: T265163#7726333.

If you recall, I'd be keen to hear what led this volunteer to share these "particular areas of focus"? E.g. are these the mistakes they have personally noticed newcomers often making? Is there a particular policy at ta.wiki that includes a checklist of sorts for new articles? Etc.

I don't think we spoke in much detail about them, they're just the examples they provided. I doubt there's such a policy, because ta.wiki is pretty light on fleshed out and consistently-actioned policies.

Another potential use case is the enforcement of UCoC policies, we could display a warning when someone tries to write potentially offensive content.

[...]two quick questions:

  1. Would it you be okay with me uploading a PDF copy of the paper in case anyone else happening upon this ticket would like to read it as well? I thankfully have access to it via the The Wikipedia Library.
  2. Do you recall how you come to learn about this ticket? No worries if you don't remember...I ask this curious to learn about spaces where this kind of thing is being discussed.

Here's a free version of the PDF: https://arxiv.org/abs/2108.02252

@Isaac pinged me via email about this ticket after I discussed that paper with him.

I've reached out to @Sumit and Nikola about your interest on this ticket. I suspect they might be interested in reinvesting in this space if the tech is likely to be used. We had a bit of a flop trying to work with a few WikiProjects to vet the usefulness of this type of modeling. I think to them it seemed like we were just adding more work or nit-picking their articles. See activity from https://en.wikipedia.org/wiki/Special:Contributions/Sumit.iitp after Sept 2020. I do think this has the potential to be useful if we can deliver the predictions at the right moment -- e.g., when someone is creating/contributing to an article.

Thanks @Halfak for the invite to this task! I am interested in understanding how long standing editor practices can be best encoded in the editing interface that can help editors do better and allow us to build robust models from structured editing data that can automatically flag outstanding issues for editors to fix. One of the problems with building effective automated content flaw detection to help Wikipedians is the lack of precise information around historical edits (like what exactly was improved in this edit?)

There are some aspects that I see as providing immediate value, e.g., grammar, style issues, broken links and some that are harder to implement and contentious, yet valuable (e.g.,WP:NPOV). In addition to thinking of this as "common suggestions that improve edits", we might benefit from thinking how can we best provide editors the tool to express this intent of the edit. E.g, if I am saving an edit, my intents could be "did I fix grammar?", "did I add new content?", "did I fix citations?".

We had attempted some engagements on English Wikipedia for our work, but its nice to see from the threads that there is interest in other Wikipedia editions as well.

One of the problems with building effective automated content flaw detection to help Wikipedians is the lack of precise information around historical edits (like what exactly was improved in this edit?)

If there was a way to get good examples of problematic content we might like to highlight, that would make the modeling approach much easier. It might also provide a route for modeling new types of content issues ad-hoc. E.g. this is a clean example of a POV issue from rev_id=539204272 with the problematic content highlighted:

The only artifacts known for Shepseskare's reign are several clay sealings from Abusir, where the king may have been buried, and two cylinder seals, according to the respected Czech Egyptologist Miroslav Verner.

In the referenced paper, Sumit developed a strategy for scanning old edits that explicitly reference WP:NPOV to find examples. This works because sometimes people will cite NPOV in their edit summary. But if they don't cite it or make multiple changes, that muddies the data and makes it unusable. In theory we could likely train a model to detect any type of issue if we have good examples of what it looks like in practice. Sumit mentioned a way that an editor might annotate their own edits. I wonder if annotating a diff might make sense since some changes will not be relevant to a specific issue.

In the referenced paper, Sumit developed a strategy for scanning old edits that explicitly reference WP:NPOV to find examples. This works because sometimes people will cite NPOV in their edit summary. But if they don't cite it or make multiple changes, that muddies the data and makes it unusable. In theory we could likely train a model to detect any type of issue if we have good examples of what it looks like in practice. Sumit mentioned a way that an editor might annotate their own edits. I wonder if annotating a diff might make sense since some changes will not be relevant to a specific issue.

I want to pile on and +1 this. A lack of high-quality structured training data is one of the main barriers to building new algorithmic models to aid editors (similar to some of the features imagined in this ticket). We can usually hack something together for English using the sort of approach that @Halfak mentioned but this is rarely possible in other languages without a lot of work. Adding structured ways to describe edits would be a huge improvement I imagine for editors but also for our ability to support editors. I went on at length about this in T290746#7385097, in particular highlighting the huge impact of the PageAssessments extension that the Community Tech team built a few years ago for our ability to build models to predict article quality and topic. I would love love love to see this sort of investment for edits and I know that folks appreciate the structured edit summaries on Wikidata and would like improvements to Wikipedia's edit summary box (search the page for "summary" in this year's Community Wishlist results).

One of the problems with building effective automated content flaw detection to help Wikipedians is the lack of precise information around historical edits (like what exactly was improved in this edit?)

@Sumit and others: I've been working on developing a tool to do I think some of what you want. It maps any Wikipedia article edit to a structured way of describing what was changed -- e.g., a reference was added and link removed. It focuses on basic edit actions instead of the more high-level intentions/quality-issues that you mention but I think still could be quite useful to help you build datasets to train your models. It's under development but works in all Wikipedia languages and is pretty complete. You can test it out here: https://wiki-topic.toolforge.org/diff-tagging?lang=en
If you have general questions about the tool (beyond its applicability to this task), I'd ask that you just email me or add them to the Meta talk page so it doesn't derail this phabricator thread: https://meta.wikimedia.org/wiki/Research_talk:Wikipedia_Edit_Types

Note: @aminalhazwani, @Halfak, @Isaac, @Samwalton9, and @Sumit: y'all can expect responses from me before next Friday, 8 April is over.

In the meantime, two quick updates:

  1. @Halfak: thank you for the link to the free version of the paper (https://arxiv.org/abs/2108.02252); I'm almost through with my first read of it
  2. I've added another use case to the task description that @MMiller_WMF identified by way of the suggestion @Kerry_Raymond shared on mediawiki about providing newcomers feedback that deters them from "correcting" the spelling of words written in American English to their British English equivalent, or vice versa.
ppelberg updated the task description. (Show Details)

Both creating articles and uploading images involve some amount of specialist understanding of Wikipedia to do correctly...

via @Caeciliusinhorto-public on en.wiki

Krinkle awarded a token.
Krinkle added a subscriber: Krinkle.

T315072: Add a new edit filter trigger action: pop-up box is a possible way to implement some of the components described in this ticket, which keeps the edit checks fully customisable on wikis locally using the existing AbuseFilter language familiar to communities.