Page MenuHomePhabricator

Fix text normalization edge cases in TTS prototype
Closed, ResolvedPublic

Description

In T424378, we deployed v0 of the TTS prototype on Toolforge. The Apps team has been testing it and reported mispronunciation of unit abbreviations. For example, "3 m" is read as "three em" instead of "three meters", and "10 ft" is read as "ten eff tee" instead of "ten feet".

We need to improve the text normalization pipeline so that the following edge cases are pronounced correctly:

  • Decimals: e.g. 3.14 -> "three point one four"
  • Percentages: e.g. 50% -> "fifty percent"
  • Temperature units: e.g. 3 °C -> "three degrees celsius", 4.2 °F -> "four point two degrees fahrenheit"
  • Measurements: e.g. 1 kg -> "one kilogram", 10 kg -> "ten kilograms"
  • Dates: e.g. 2025-01-15 -> "January fifteenth twenty twenty-five"
  • Currency: e.g. $99.99 -> "ninety-nine dollars and ninety-nine cents"
  • Ordinals: e.g. 1st -> "first", 2nd -> "second"
  • Time: e.g. 3:30pm -> "three thirty PM"
  • Abbreviations: e.g. "St." -> "Saint" or "Street" (disambiguated by context)
  • Number ranges: e.g. 100–900 -> "one hundred to nine hundred"
  • Compound units: e.g. m/s² -> "meters per second squared" (T426756#11944373)
  • Subscript and Superscript characters: e.g. ₂, ² should be spoken as words (T426756#11944373)
  • Orphaned punctuations (T426756#11966668)
  • Custom/Wikipedia-specific terms: Domain vocabulary not handled by general-purpose text normalization

Using an established text normalization library will allow us to handle a broader range of these cases without rolling our custom regex rules for each edge case.

Event Timeline

One thing we found while fixing subscript/superscript edge cases is that Wikipedia's plain-text extract API:
https://en.wikipedia.org/w/api.php?action=query&titles=Square_metre&prop=extracts&explaintext=1&format=json
strips all formatting. Content like m<sub>2</sub> i.e m₂ and m<sup>2</sup> i.e both arrive as m2. This ends up being read as "m two". The digit is pronounced correctly, but the superscript meaning ("squared") is lost.

The Wikipedia HTML API: https://en.wikipedia.org/w/api.php?action=query&titles=Square_metre&prop=extracts&format=json
preserves the original markup m<sup>2</sup>, which could allow us to handle squared units differently. However, this benefit is limited to a narrow set of cases because for most scientific content (e.g, H₂O, O₂, etc) the plain-text pipeline already reads correctly.

The v0 prototype uses the plain-text API because it delivers the words and numbers needed for speech without the complexity of parsing arbitrary Wikipedia HTML. Filtering raw HTML reliably is error-prone (risks ruining the current listening experience) and adds an ongoing maintenance burden for marginal benefit. We can revisit building an HTML-based extraction pipeline as we work towards scoping v1.

Following T426756#11944373, we integrated the NeMo text processing library into the TTS protottype since it handles a majority of the nuanced text normalization edge cases outlined in this task's description.

As shown in the output below, the pipeline now correctly interprets and expands: decimals, percentages, temperature units, measurements, dates, currency, ordinals, time, contextual abbreviations and number ranges:

>>> from wiki_tts.text import clean_spoken_text, init_nemo
>>> init_nemo()
 NeMo-text-processing :: INFO     :: Post processing graph was restored from /tmp/wiki-tts-nemo-grammars/en_tn_post_processing.far.
 NeMo-text-processing :: INFO     :: ClassifyFst.fst was restored from /tmp/wiki-tts-nemo-grammars/en_tn_True_deterministic_cased_nemo_whitelist.tsv_tokenize.far.
 NeMo-text-processing :: INFO     :: VerbalizeFinalFst graph was restored from /tmp/wiki-tts-nemo-grammars/en_tn_True_deterministic_verbalizer.far.
>>> 
>>> # Decimals: e.g. 3.14 -> "three point one four"
>>> clean_spoken_text("3.14")
'three point one four'
>>> 
>>> # Percentages: e.g. 50% -> "fifty percent"
>>> clean_spoken_text("50%")
'fifty percent'
>>> 
>>> # Temperature units: e.g. 3 °C -> "three degrees celsius", 4.2 °F -> "four point two degrees fahrenheit"
>>> clean_spoken_text("3 °C")
'three degrees Celsius'
>>> 
>>> # Measurements: e.g. 1 kg -> "one kilogram", 10 kg -> "ten kilograms"
>>> clean_spoken_text("1 kg")
'one kilogram'
>>> clean_spoken_text("10 kg")
'ten kilograms'
>>> 
>>> # Dates: e.g. 2025-01-15 -> "January fifteenth twenty twenty-five"
>>> clean_spoken_text("2025-01-15")
'january fifteenth twenty twenty five'
>>> 
>>> # Currency: e.g. $99.99 -> "ninety-nine dollars and ninety-nine cents"
>>> clean_spoken_text("$99.99")
'ninety nine dollars ninety nine cents'
>>> clean_spoken_text("$1")
'one dollar'
>>> 
>>> # Ordinals: e.g. 1st -> "first", 2nd -> "second"
>>> clean_spoken_text("1st")
'first'
>>> clean_spoken_text("2nd")
'second'
>>> clean_spoken_text("3rd")
'third'
>>> 
>>> # Time: e.g. 3:30pm -> "three thirty PM"
>>> clean_spoken_text("3:30pm")
'three thirty PM'
>>> clean_spoken_text("3:30am")
'three thirty AM'
>>> clean_spoken_text("1530 hours")
'fifteen thirty hours'
>>> 
>>> 
>>> # Abbreviations: e.g. "St." -> "Saint" or "Street" (disambiguated by context)
>>> clean_spoken_text("123 Main St.")
'one twenty three Main Street'
>>> clean_spoken_text("St. Patrick's Day")
"St. Patrick's Day"
>>> 
>>> # Number ranges: e.g. 100–900 -> "one hundred to nine hundred"
>>> clean_spoken_text("100–900")
'one hundred to nine hundred'
>>>

We have also added a nemo_whitelist.tsv. Without it, NeMo would treat unrecognised domain-specific vocabulary words like "UNESCO" as regular words, causing the TTS service to read them as "an eh sko" rather than "yoo neh sko". The whitelist preserves these terms as-is so the TTS service handles pronunciation correctly. The best part is that this list can be expanded whenever we notice new custom/wikipedia-specific/domain-specific terms that aren't being spoken well.

>>> from wiki_tts.text import clean_spoken_text, init_nemo
>>> 
>>> init_nemo() 
 NeMo-text-processing :: INFO     :: Post processing graph was restored from /tmp/wiki-tts-nemo-grammars/en_tn_post_processing.far.
 NeMo-text-processing :: INFO     :: ClassifyFst.fst was restored from /tmp/wiki-tts-nemo-grammars/en_tn_True_deterministic_cased_nemo_whitelist.tsv_tokenize.far.
 NeMo-text-processing :: INFO     :: VerbalizeFinalFst graph was restored from /tmp/wiki-tts-nemo-grammars/en_tn_True_deterministic_verbalizer.far.
>>> 
>>> # Custom/Wikipedia-specific terms: Domain vocabulary not handled by general-purpose text normalization
>>> clean_spoken_text("NASA launched a mission.")
'NASA launched a mission.'
>>> clean_spoken_text("UNESCO declared a world heritage site.")
'yoo neh sko declared a world heritage site.'
>>> clean_spoken_text("DNA and RNA are nucleic acids.")
'DNA and RNA are nucleic acids.'
>>> clean_spoken_text("AI technology is advancing.")
'AI technology is advancing.'

We found another text normalization edge case while working on T427488: Add word-level timestamps in TTS prototype.

This can be reproduced by visiting the Earth article and looking at the Symbol section. This section has the following content:

The standard astronomical symbols of Earth are a quartered circle, 🜨,[33] representing the four corners of the world, and a globus cruciger, ♁.

But the Wikipedia plain-text API shows the text below with orphaned punctuations (i.e , , and , .) left behind by stripped markup/symbols: https://en.wikipedia.org/w/api.php?action=query&titles=Earth&prop=extracts&explaintext=1&format=json

The standard astronomical symbols of Earth are a quartered circle, , representing the four corners of the world, and a globus cruciger, .

We fixed this by stripping the orphaned punctuations left behind in clean_spoken_text(). Below is the normalized result that the TTS service now receives and reads:

The standard astronomical symbols of Earth are a quartered circle, representing the four corners of the world, and a globus cruciger.