Weekly update:

• @diego is attending WikiLibCon this week, no updates for corresponding tasks.
• @cwylo and I have been meeting individually with @Isaac to discuss our efforts to advance research on moderation work in Q3. I took the very relevant questions on distributed moderation from T383365 and created a Google doc, which incorporates new questions and a narrower focus on verifiability. The idea is for the three of us to collaboratively refine the research proposal before sharing it publicly on Phabricator (ideally next week).
• For the work related to elections, I plan to solve the task next week (one hypothesis required analyzing data from the election date through to the inauguration date, January 20)

Thanks @Pablo ! Adding something that came in late but there is now a kick-off meeting related to Peacock detection on 29 January.

Weekly update:

• @cwylo, @Isaac, and I met to organize work around distributed moderation, which will focus on issue-template space as we have a lot more understanding of the "patrolling" side.
• I identified hundreds of English articles related to the 2024 elections in India and the European Union. Work is ongoing to compute the revert risk scores for the related revisions (I also asked the Disinformation team to review if important articles are missing).
• No additional updates from other projects (no major updates are expected in the coming two weeks due to some colleagues being OoO).

Weekly update:

• I focused on getting familiar with the dataset of templates or infoboxes added/deleted on revisions from October 2024. @Isaac and I have schedule a co-working session to discuss improvements and potential directions.
• @Isaac met with Editing and ML Platform to discuss the scoping of the Peacock-detection Edit Check. They are open to adjusting but would ideally like an edit check that can operate at the sentence or paragraph level for any newcomer edit that adds new content and be able to detail which words are causing the issue. This is sensible but a different data distribution (new edits vs. existing articles) and granularity (sentences/paragraphs vs. articles) from the current model and also means that we need a more robust evaluation of the SHAP-based approach prototyped by @Aitolkyn. @Isaac suggested that valuable next steps for Editing/ML to bring more clarity would mean hosting the current model, hand-labeling some edits where the check would be triggered, and generating some groundtruth for which words are problematic when the edit should fail the peacock detection. This would provide a better understanding of whether the current model is still a good fit or needs to be re-worked with new data or approach.
• With the feedback of the Disinformation team and a Romanian wikimedian, I completed the final lists of English articles related to the 2024 elections in India and the European Union. I also identified the existing versions in other Wikipedia language editions of interest.

Weekly update:

• @Isaac and I met for a co-working session in which it was decided to first start improving the template parsing process to address the edge cases reported at the SDS 1.2.3 report. As result, code is under development to better match edge cases, including (1) dynamic messageboxes that report statistics on category usage or when the page was last edited, (2) minor maintenance edits in templates. In parallel, a Spark job was run to collect edge cases (very similar but not identical moderation signals between a consecutive revision and its parent revision) across multiple languages. Next steps will focus on using these cases to refine the parsing strategy.
• No updates were identified in other projects of this bucket.

Weekly update:

• Moderation public report T382614: @diego created a public Meta page based on the internal report.
• Distributed moderation (qualitative work) T383365: To align efforts between qualitative and quantitative work, a check-in meeting between @cwylo @Isaac and @Pablo has been scheduled for February 24 (next week is the quiet week for the WMF research team).
• Distributed moderation (quantitative work) T384600: @Pablo focused on creating a notebook with a refined the parsing strategy based on three steps for matching moderation signals between revisions: (1) exact HTML matching, (2), exact wikitext matching, and (3) fuzzy HMTL matching. The later is based on the Levenshtein ratio, so over 100 edge cases across wikis have been manually inspected. As a result, it is proposed for the third step that two moderation signals are the same if (a) the Levenshtein ratio is over 0.9, and (b) the wikitext template names match. @Isaac and @Pablo are currently examining and getting expert feedback on very extreme cases.
• Peacock support: @diego is currently advising the ML team to create an evaluation dataset at paragraph/sentence level.
• Client-hint consultation: No update (next week, @Pablo will review the project plan).
• Elections analysis: No update (next (quiet) week, @Pablo will focus largely on this project).

Weekly update:

• Moderation public report T382614: Completed.
• Distributed moderation (T383365/T384600): No updates this week.
• Peacock support: @diego provided hands-on support in evaluating the existing peacock detection model (T386645#10561213). From now on, he will take on an advisory role.
• Client-hint consultation: @Pablo reviewed the WE 4.2.10 Project Planning to provide feedback on (1) the identification sockpuppets and potential ban evasion, and (2) the process to create hash representation.
• Elections analysis: @Pablo built the dataset with revert risk predictions of revisions about the 2024 Indian elections (enwiki: 148,650; hiwiki: 9,911; mrwiki: 3,084) and the 2024 EU elections (eswiki: 10,590; frwiki: 32,979; itwiki: 20,288; rowiki: 1,700). Notebooks have also been prepared to test hypotheses H2 and H3, which will be discussed next week with the technical research partner of the DEM-Debate project, responsible for the analysis of Wikipedia data on the 2024 EU elections.

Weekly update:

• Distributed moderation (T383365/T384600): The refined version of the parser was executed in the cluster to generate a new dataset of distributed moderation actions. @cwylo, @Isaac, and I met this week to review progress and plan the next steps. On the quantitative side, we will prototype metrics using the new dataset. For the qualitative work, the main deliverable will be a taxonomy of distributed moderation based on these metrics.
• Peacock support (T386645): @diego kept supporting the ML team in refining the inline peacock dataset. He also attended coordination calls with the ML team and the Moderators Tool team. During the ML team meeting, @MGerlach shared his positive experience using the peacock model to detect puffery in LM-generated summaries.
• Client-hint consultation (T381031): No updates this week (no actions expected in the coming weeks either).
• Elections analysis (T384616): Results of the analysis have been published at T384616#10590184. Take home messages:
  • In most of the wikis analyzed, the lead up to the election attracted IP editors, but not newcomers.
  • In the lead up to the election, no increase in reverts to IP edits or newcomer edits was observed.
  • Low quality edits were usually moderated quickly.
  • Wikis with content moderation bots reverted more low-quality edits and more quickly (which may serve to highlight the great value of the work of Moderator Tools, led by @Samwalton9-WMF).
  • The data collection process could have missed election-related articles not existing in English Wikipedia (it is worth exploring if wikis with few edits to election-related articles (e.g., Marathi, Romanian) contain relevant articles not present in the datasets).

Weekly update:

• Distributed moderation (T383365/T384600): I uploaded to the Gitlab repository the notebook to generate the dataset of moderation actions in October 2024 and the notebook with the metrics listed at (T384600#10612914). The metrics already serve to quantify moderator/moderation activity. Next week I will focus on connecting them with the SDS 1.2.3 Moderator Action Spreadsheet (in particular, the article maintenance actions) to help define the categorization of the issue templates as a taxonomy. We are exploring the possibility of drafting the deliverable as a paper for a workshop on content moderation and platform governance.
• Peacock support (T386645): @diego supporting the ML team in comparing different versions of the model.
• Elections analysis (T384616): The results of the analysis were shared with colleagues and posted on Meta. The ticket is now resolved.

Weekly update:

• Distributed moderation (T383365/T384600): After a coordination meeting, I created datasets with stats of issue templates in the selected wikis (details: T384600#10637277). These datasets are assisting @cwylo in cataloguing and categorizing templates (details: T383365#10632956). I have also created an Overleaf project to start documenting the results from both efforts as a work-in-progress paper to be submitted to an academic workshop on content moderation and platform governance.
• Peacock support (T386645): @diego is advising the ML team on how to create evaluation data.

Weekly update:

• Distributed moderation (T383365/T384600): I structured the Overleaf project into sections and completed the preliminary data exploration section. @cwylo, @Isaac, and I met to coordinate the writing process. @cwylo will document the taxonomy of issue templates by Wednesday, allowing me to finalize a draft by the end of the next week.

Weekly update:

• Distributed moderation (T383365/T384600): Efforts this week have focused on running the last analysis of templates (details in T384600#10684072) to complete a first version of the taxonomy (details in T383365#10683259).
  • These two tasks will be resolved.
  • A new task will be created for the documentation of the results (a non-archival workshop paper and a page on https://meta.wikimedia.org/wiki/Research:Projects).

• Peacock support (T386645): @diegoparticipated in the "pre-mortem" meeting for this project, helping to anticipate risks on the implementation of this product.

• Client-hint consultation (T381031): Service was provided for requests in Q3 (details in T383610#10571044), so the task is complete.