@kostajh -- thank you for filing this and including that useful list. @MGerlach is going to be doing research in the coming months about this whole space, and so these links might be useful for him. Here's the project page we created about it: https://www.mediawiki.org/wiki/Growth/Personalized_first_day/Structured_tasks/Copyedit

In order to directly query LanguageTool's suggestions for Wikipedia-articles, I have created an experimental API on toolforge. One needs to specify the language of the Wikipedia (e.g. “en”) and the page-title. Some example calls for different wikis/articles:

• https://copyedit.toolforge.org/api/v1/lt?lang=en&title=Chiara_Cordelli
• https://copyedit.toolforge.org/api/v1/lt?lang=simple&title=Sodium
• https://copyedit.toolforge.org/api/v1/lt?lang=de&title=Panama_Hotel_(Seattle)
• this works for 31 wikis (those where the language is supported by LanguageTool): ar, ast, be, br, ca, da, de, el, en (and simple), eo, es, fa, fr, ga, gl, it, ja, km, nl, pl, pt, ro, ru, sk, sl, sv, ta, tl, uk, zh
• Code and some documenation: https://gitlab.wikimedia.org/repos/research/copyedit-api

This allows for easier exploration on what are possible challenges. One of the main difficulties I encountered was that LanguageTool is very sensitive and thus yields many false positives (flagging an error when the text is in fact correct). In order to mitigate that, the tool applies an aggressive filtering to reduce the number of false positives. Using the HTML-version of the article makes this much easier: i) identify only plain text (and avoiding tables, infoboxes, or transcluded content from templates; ii) remove errors that overlap with annotated text such as links, bold, italics, etc, which often yields wrong errors. While in this case I query the HTML for individual articles via the MediaWiki-API, we could also do this in bulk thanks to the amazing HTML-dumps.

The tool queries an endpoint on cloud-vps running our own instance of LanguageTool such that we dont rely on limitations/restrictions of LanguageTool's public-facing API. The endpoint could easily feed into other tools as well.

• Test-query: https://copyedits.wmcloud.org/v2/check?language=en&text=my+text
• Code and documentation: https://github.com/wikimedia/research-api-endpoint-template/tree/language-tool

This allows for easier exploration on what are possible challenges. One of the main difficulties I encountered was that LanguageTool is very sensitive and thus yields many false positives (flagging an error when the text is in fact correct).

Hmm, I was a little surprised to read this as anecdotally in my day-to-day use of the tool, I rarely see false positives. Are there configurable thresholds for the suggestions?

@kostajh at this point it is also an anecdotal observation from my side. For example, copying the lead section of enwiki-article Roman Catholic Diocese of Bisceglie (a random article) into the LanguageTool-interface yields 7 errors in only 3 sentences -- all of which are false positives (bold are the errors from LanguageTool):

The Diocese of Bisceglie (Latin: Dioecesis Vigiliensis) was a Roman Catholic diocese located in the town of Bisceglie on the Adriatic Sea in the province of Barletta-Andria-Trani, Apulia in southern Italy. It is five miles south of Trani. In 1818, it was united with the Archdiocese of Trani to form the Archdiocese of Trani-Bisceglie.[1][2]

Not all of the errors are links such as the italics Latin spelling. Some false positives are recurring, e.g. Trani appears as plain text and is marked as an error. Thus, it is not trivial to exclude them and some effort is needed to filter out the false positives.

The aim is to get a quantitative evaluation of the false positive rate. However, this is difficult due to lack of good ground-truth data for two reasons: i) we dont have a complete annotated sample of true errors in Wikipedia articles and ii) we have even less data for languages other than English.

There are some configuration options though I havent played with that (for more details see the documentation of the endpoint https://github.com/wikimedia/research-api-endpoint-template/tree/language-tool).