Thanks for creating this task! Might be useful to try to think through all of the potential ways of representing a diff and potential models upfront so we can split up the work a little? My gut feeling is that we should first see how we can do from content alone as it should hopefully be enough alone to determine the larger category of edit that's being made. I'm sure the user features will boost offline model performance a bit as there are definitely user patterns, but they will also will build in some long-term fragility to the model I suspect because those are things that could easily change.

Models:

• Super simple supervised classifier -- e.g., fastText over just the changed words?
• Do we want to do a fine-tuned SLM? Something like ModernBERT?
• A few size of LLMs with zero or few-shot prompts (similar to T414816: [WE1.7.3] Exploration of automated verifiability checks)?

As far as input data, I suspect almost all of this will have to be wikitext unfortunately because otherwise we won't be able to collect enough data for training the models. That said, I'll add a few more small-scale experiments we can do at least with the LLMs (only smaller-scale evaluation data needed) to test if they have any impact on quality. How to represent the diff:

• Side-by-side revisions (full thing, no diff)?
• Reduced diff -- i.e. remove sections that are common between the two revisions to reduce size?
• A list of changed tokens (no attempt to say whether added or removed, just split on whitespace and provide as list)?
• Same as above but split between "Inserted" and "Removed"?
• List of all changes from the more Structured Edit Types output? [the above ideas we can do cheaply outside of mwedittypes library but this would require mwedittypes]

Smaller-scale experiments:

• The above but based on HTML (we've been gathering HTML diffs for several months now for moderator metrics so have a pretty large sample actually available).
• Use the diff representation that comes via API:Compare. This is based on wikitext and unfortunately I'm not sure if there's a nice compact representation based on the HTML available.

fab opened https://gitlab.wikimedia.org/repos/research/research-datasets/-/merge_requests/113

edit_intent: add notebooks for predicting edit intent from diffs

fab merged https://gitlab.wikimedia.org/repos/research/research-datasets/-/merge_requests/113

edit_intent: add notebooks for predicting edit intent from diffs

A first set of notebooks can be found here.

• The raw features notebook generates research_dev.edit_intent_raw: The ~10.9M interpretable enwiki edits enriched with revision wikitext, the parent wikitext, the parent diff, and the structured edit-types JSON. This is the shared input for every use case below, and it uses a broadcast revision_id key-set to prune the large content tables rather than shuffle them.
• A shared library module holds the config and helpers, and an exploratory analysis notebook compute general statistics about the labelled dataset

Derived use cases (starting points on top of the raw features, aka no iteration on quality):

• The logistic regression notebook writes research_dev.edit_intent_features_logreg: simple top-level features (diff sizes, edit-type counts, user features) for a classifier of intent, with a stratified/grouped/temporal split cell and a baseline fit. The first run shows reasonable for well-solved structural intents (category, date, and link maintenance, page creation and move, stubs with a F1 ~0.85 to 0.93) and harder semantic ones (content add/remove/update, copyedit, unsourced, vandalism much worse at 0.1 to 0.5).
• The edit-type clustering notebook runs SVD then HDBSCAN on edit-type feature vectors (same as used for the logistic regression), coloured by intent, the plot of the first two PC shows clusters for some edit intents (matching the regression results).
• One notebook embeds the words each revision added (using intfloat/multilingual-e5-large-instruct) and writes research_dev.edit_intent_added_embeddings; a second notebook clusters those embeddings (SVD, HDBSCAN, t-SNE) to test whether added content alone recovers the intent categories unsupervised.

Note that the _export.ipynb linked above were just exported for the gitlab sharing, to work with these notebooks use the interactive versions with the same name.