Thanks for creating a ticket for further investigation on the approach.

In my POC, I did some simplifications and I would consider removing that simplification and bring that close to a real world edit situation. And then evaluate it to get a better sense

1. I hand crafted the before and after texts. They are plain text content. However, wikitext might be the actual format we want to use.
   • Plain text version of the edit difference might miss vandalism in non-renderable parts. For example, adding a link to exernal website, while the link text is non-problematic text
2. In real edit diff situation, how effectively we can construct these 'before' and 'after' content
   • Need to evaluate how effectively LLM can 'summarize' the diff when the diff is compllicated - as in table edits, template edits, markup corruptions, Also long range diffs when small edits are done in various parts of a large article

I believe these two improvements require more work on data preparation. We might also need to expand the POC to allow us quickly check the edits. For example, let it accept a revision id and then evaluate the edit. Standalone(or sandboxed) evaluations like that can help us in quick iterations.

Known limitations: Any LLM approach is subjected to limited language coverage. But that should not prevent us from using its potential in the languages it supports.