As an engineer I'd like to create a prototype of a service that detects Wikipedia policy violations and surfaces them in the VisualEditor in order to assist new editors in making edits.
The prototype will be based on the existing work that has been done for peacock detection but use an LLM hosted locally using ollama. VisualEditor makes a post request to the model feeding it a paragraph each tim. The model uses a system prompt based on the Wikipedia:Manual of Style and returns a json in the following format { "violations":[], "message": ""}
Example:

Edit text: "Some people say that Wikipedia is the best encyclopedia ever."
Your response: {"violations": ["WP:NPOV", "WP:PEACOCK", "WP:WEASEL"], "message": "Uses 'best' which is a subjective term violating WP:NPOV and also employs 'ever' implying absoluteness, contravening WP:PEACOCK. Also the phrase 'Some people say' is vague and makes the statement unverifiable."}

Using an LLM is only part of a POC in order to explore how a multi edit-check like this would look like. Scaling this and putting it in front of editors for real time use cases would require a lot of additional work. To achieve this exploration and experimentation in the following areas would be required:

Target small-ish models and fine tune them:

• Collecting and labelling data for each of the policy violations
• Fine tuning smaller models that would allow us to serve real time within the Visual editor in < 100 ms latency

For additional POC experimentation:

• Identifying an open source model that performs well on this task and is small enough to achieve low or just tolerable latency for the user.
• Iterating on the prompt, figuring out edge cases and feed it a couple of working examples.