Dataset for hackathon is created.
Please see the following notebook for a sample usage:
https://gitlab.wikimedia.org/repos/machine-learning/exploratory-notebook/-/blob/edit_suggestions_dataset/edit_suggestions/generate_suggestions.ipynb?ref_type=heads

models to pick
We currently discuss it here:
https://wikimedia.slack.com/archives/G01A0FNPLG4/p1768310436209719

@OKarakaya-WMF could you please add a weekly summary for me to pull into Asana?

Reporting format
Progress update on the hypothesis for the week, including if something has shipped:

• We have updated the format of the articles during html to text conversion.
• We have re-generated the most frequent edit types tables after the updates. (article level suggestions.)
• We have generated edit types in section level. We share the findings in scratchpad
• We have started working on creating a dataset for MOS and grammar related edits.

Any updates on metrics related to this hypothesis (including baseline, target, or actuals, if applicable):

Any emerging blockers or risks:

Any unresolved dependencies:

New lessons from the hypothesis:

Changes to the hypothesis scope or timeline:

• We have organized the phases as Preparing the edit suggestion types, Evaluation + Iteration and Data Access. We share the scope of the phases in the Timeline

Reporting (20/03/2026)

Progress update on the hypothesis for the week, including if something has shipped:

• We have identified a list of edit suggestion types that we aim to work on. The edit suggestion types are based on frequent edits from users, frequent suggestions from LLMs and suggestions from colleagues/community. We will share it with the editing team.
• Our first evaluation approach is to use user edits as much as possible. We have created a dataset to validate grammar and MOS (Manual of Style) related edit suggestions. We will use it after we create the related suggestions.
• We are working on the second evaluation method llm-as-a-judge. We are currently testing it on a small subset of suggestions.
• We are working on generating edit suggestions based on pre-defined list of edit suggestion types.

Any updates on metrics related to this hypothesis (including baseline, target, or actuals, if applicable):

Any emerging blockers or risks:
N/A
Any unresolved dependencies:
N/A
New lessons from the hypothesis:
N/A
Changes to the hypothesis scope or timeline:
N/A

Reporting (27/03/2026)

Progress update on the hypothesis for the week, including if something has shipped:

• We have generated pre-defined edit types for MOS and grammar related edits. Qualitatively, we find useful suggestions on grammar, mos and neutrality.
• We have evaluated pre-defined suggestions by using llm-as-a-judge. Based on a small sample of suggestions (20 suggestions) we have 62% human alignment with the llm-as-a-judge scores. We believe annotating a larger dataset can increase the human alignment and confidence of the llm-as-a-judge scores.
• We work on evaluation based on edits for grammar edit types. We find similarity based comparison useful. We will expand the evaluation based on edits to calculate precision, recall.

Any updates on metrics related to this hypothesis (including baseline, target, or actuals, if applicable):

Any emerging blockers or risks:
N/A
Any unresolved dependencies:
N/A
New lessons from the hypothesis:
N/A.
Changes to the hypothesis scope or timeline:
N/A