Tokenize sentences with an acceptable level of accuracy across different wikiproject languages. Will be largely language agnostic.
== Background
[x] Literature Review
[x] Past tools survey ([[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/3|Issue 3]])
== Rule-based
[x] Build a global list of Unicode sentence terminators and use regex to split sentences ([[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/4|Issue 4]])
== Statistical
[x] Scrape likely abbreviations from all wiktionary projects ([[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/10|Issue 10]])
[x] Filter those abbreviations based on Wikipedia occurrences ([[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/10|Issue 10]])
[x] Incorporate language-specific abbreviation lists into sentence tokenization code ([[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/5|Issue 5]])
== Additional
[] Build and compile additional rules required (contraction, abbreviations, different punctuation schemes etc)
([[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/12](|issue 12)]] , [[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/14](|issue 14),]] ,[[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/7](|issue 7)]] , [[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/9](|issue 9]] )
== Performance / Evaluation
[x] Initial basic benchmark of how sentence tokenization does on different edge cases ([[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/8|Issue 8]])
[x] Add unit tests ([[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/15|Issue 15]])
[x] Refactor code to help simplify / optimize performance ([[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/20|Issue 20]])
[x] Fuller evaluation of how addition of abbreviation-handling impacts performance ([[https://gitlab.wikimedia.org/repos/research/wiki-nlp-tools/-/issues/11|Issue 11]])