Goal: Set up a process for regularly running a tally of VisualEditor edits, and classifying them whether they're problematic. This will be in addition to the existing automated tests and QA efforts.
- Review a batch of edits from recent changes with the VisualEditor tag on a selection of wikis. Initial numbers: ~500 edits total from ~5 biggest wikis, but this can be tweaked as long as we get a reasonably representative sample of edits.
- Classify them by handcoding them according to the categories below
- Do this before the weekly VisualEditor triage meetings so the bugs can be reviewed then.
Possible categories for edits:
- no problem
- unavoidable user error
- borderline (could be improved with UX)
- obvious bug
- unknown (not apparent what the user attempted to do, or what the problem was)
In order to save time, a simple user script or similar may help make the process faster. For example, the script could add a few buttons in the diff interface that match the different classifications, and automatically add the diff to the appropriate wikipage / spreadsheet / tally, instead of manually copy-pasting each URL.