Page MenuHomePhabricator

SPIKE: Enable Clean up in OCR Proofreading (4hours)
Closed, ResolvedPublic

Description

During a community comment reply, Satdeep shared there is a script users can add to their common.js that can help automate proofread cleanup.

This gave me an idea of incorporating a "cleanup" functionality to the proofread page.
Satdeep also suggested adding it as a gadget or extension depending on which makes sense.
However. there are some nuances with how to use this cleanup functionality which would need to be documented.

Example of script can be found here:
https://en.wikisource.org/wiki/User:Bodhisattwa/cleanup.js

This investigation would entail:

  • Documenting how to use the script
  • Tabling the cost of adding it to UI versus making it a gadget/extension

Event Timeline

@satdeep_gill do you mind documenting the nuances of the cleanup script in the comments so that @nayoub and i can figure out if it makes sense in the UI

Natalia to read through this OCR Wikisource script documentation list and prioritize functionality with Nicolas:
https://en.wikisource.org/wiki/Wikisource:Tools_and_scripts

There are multiple versions of cleanup script available on various user pages and wikis because OCR produces different types of errors across different languages. So, it might be interesting to investigate how to add cleanup.js to Wikisource extension but also allow community members to add their language specific common errors (maybe it all goes to the one global script), perhaps even while they are proofreading and encountering errors.

Here's a task about cleaning up Google's structured output into wikitext: T250185: Make Wikisource-OCR handle paragraphs better

I think a system for user-supplied sets of cleanup rules would be great. A huge amount could be done with a set of ordered regex replacements, perhaps bundling them together into tasks such as "remove line breaks", "fix punctuation", etc. so that users wouldn't have to actually read regex syntax themselves. A few cleanup scripts do more complicated things than that, but they can of course keep doing so; I think that anything built into the OCR tool should be generic and extensible, and as simple as possible.

NRodriguez renamed this task from SPIKE: Enable Clean up in OCR Proofreading to SPIKE: Enable Clean up in OCR Proofreading (4hours).May 3 2021, 5:19 PM