Page MenuHomePhabricator

TTS Section Generator Phase 4: a ~50-article batch pilot producing the measured numbers pack for the DE intake and the maxReplicas capacity decision
Closed, ResolvedPublic

Description

Following Phase 3 (T432308), the generator is deployed to staging with the NeMo cache baked into the image, artifact sinks behind one interface, the timeout chain provisioned at 600 s end to end, and the normalization ruleset finalized (norm-2026.07.20). The Prep Pantry intake document we will bring to the Data Engineering team still carries calibration-derived estimates where it should carry measurements: storage and audio-hour projections derived from character counts and a chars-per-audio-second constant, throughput math derived from isvc benchmarks rather than the full path, skip rates measured only at scan level, and failure rates that are guesses. In this task, we shall run a pilot over ~50 Featured Articles and replace those estimates with measured v1 numbers.

  • The pilot driver: deliberately a script, not a pipeline. We shall write a small driver that draws ~50 articles from the committed corpus scan snapshot (corpus_scan.json), reusing its pinned revisions, so the run is reproducible against a fixed population: the Phase 3 rescan demonstrated that a seed alone does not reproduce a sample when the Featured Article category changes underneath it, and revision pinning makes the run immune to edits. The driver shall enumerate each article's sections via GET /sections, invoke POST /generate-section for every generatable section through the mesh ingress (the same path the DE pipeline will call, validated in Phase 3 with the corpus-max section), request the full artifact family (audio_opus, audio_mp3, captions_vtt, timestamps_json), and append every outcome (success with timings and sizes, deterministic skip with taxonomy code, transient failure with retry count) to a results log. Concurrency shall be bounded to the isvc replica count so requests never queue at the inference service, since activator queue time counts against the Knative revision timeout; at the current single staging replica this is a serial run of several hours, and the driver shall therefore be idempotently resumable by consulting its own results log (which is a restart convenience, not a queue). It shall have no queue, no scheduler, no index, and no event consumption: those are the DE pipeline's job, and the moment this script grows them we have rebuilt v0's Celery layer and un-drawn the boundary the intake document draws. The staging service shall keep its current inline sink; the driver shall decode responses and write artifacts to a local scratch tree under the canonical revision-scoped key layout, so the pilot requires no deployment changes.
  • The measured numbers pack. From the results log and the artifact tree, we shall produce a one-page pack containing: real per-section Opus and MP3 sizes (the codec comparison for Apps) with the corpus storage projection, replacing the calibration-derived estimate; the per-section latency distribution through the full fetch, normalize, synthesize, transcode path (p50/p90/p99/max), which isvc-only benchmarks cannot see; skip rates by taxonomy code and failure modes actually encountered, so DE's retry and dead-letter design starts from observed reality; sections-per-article and audio-hours-per-article statistics that firm up the corpus-size estimate (~81k generatable sections per the Phase 3 scans) and the pod-day math; measured chars-per-audio-second and effective RTF at pilot scale, replacing the five-section calibration from the timeout spike; and the effective per-pod throughput of the full path, from which we shall derive a maxReplicas recommendation for the isvc (corpus drain time as a function of replica count, measured directly at the full-alignment tier and derived for the audio-only tier from the isvc's measured RTF ratio).
  • The memory envelope decision. The pilot shall record the generator pod's memory.peak across the run, completing the Phase 3 finding that the corpus tail sizes the memory envelope: the 3 GiB staging limit was set from a measurement inflated by an in-cgroup driver, and this run's external driver yields the clean peak from which we shall set the permanent limit.
  • The listening spot-check. We shall listen to a sample of the generated audio across article types (history, science, biography, music), since normalization regressions are audible rather than visible in logs: number and unit expansion, abbreviation handling, foreign names, and any non-spoken content that survived extraction. The checklist shall be seeded with the caption and normalization observations catalogued during Phase 3 (zero-duration trailing words at segment boundaries, punctuation carried into captions, letter-digit designations, abbreviation edge cases). Defects found shall be filed (and fixed with regression tests where they are extraction or normalization bugs, following the Phase 3 pattern); systematic quality questions shall be recorded for the voice/normalization evaluation rather than patched ad hoc.
  • Intake document update. We shall append the numbers pack to the Prep Pantry intake document and replace its estimate-marked rows (storage volume, audio hours, capacity envelope, per-section wall times) with the measured values, so the document DE reviews contains observed data from the actual compute function their pipelines would invoke.

Acceptance criteria: the pilot run completed over ~50 articles with a committed results log and summary script; the one-page measured numbers pack appended to the intake document with every estimate row replaced or explicitly reconfirmed; the maxReplicas recommendation written down with its derivation; the permanent generator memory limit set from the measured clean peak; the listening spot-check performed with findings filed; and the driver script committed as a reference client for the generator's contract (useful to DE as documentation, not as a component).

Scope of this task: the pilot run and its analysis only. Tracked separately:

  • The DE engagement itself: sharing the intake document and scheduling the design session are the follow-on step this task arms, not part of it.
  • The maxReplicas change: this task produces the recommendation; the capacity change is a team decision and a separate deployment patch.
  • Full-corpus generation: the ~7k-article drain is the DE batch pipeline's inaugural run, not a bigger pilot.
  • Voice/normalization quality evaluation: the listening spot-check surfaces issues; a systematic listening eval (voice selection, quantized-model comparison) remains separately tracked.
  • DE / Data Persistence / Traffic layers: unchanged ownership per the intake document.

Related Objects

Event Timeline

Change #1313163 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts-section-generator: Add Phase 4 pilot driver and numbers-pack summary script

https://gerrit.wikimedia.org/r/1313163

Change #1313168 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] docs: Refresh tts model-server and tts-section-generator READMEs

https://gerrit.wikimedia.org/r/1313168

Change #1313163 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts-section-generator: Add Phase 4 pilot driver and numbers-pack summary script

https://gerrit.wikimedia.org/r/1313163

Change #1313168 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] docs: Refresh tts model-server and tts-section-generator READMEs

https://gerrit.wikimedia.org/r/1313168

Change #1313729 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] docs: Document why tts-section-generator is a separate service from the TTS model-server

https://gerrit.wikimedia.org/r/1313729

Change #1313729 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] docs: Document why tts-section-generator is a separate service from the TTS model-server

https://gerrit.wikimedia.org/r/1313729

Change #1313884 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts-section-generator: Fix pilot regression: interlinear strip + × digit-context guard

https://gerrit.wikimedia.org/r/1313884

Change #1313884 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts-section-generator: Fix pilot regression: interlinear strip + × digit-context guard

https://gerrit.wikimedia.org/r/1313884

Change #1313903 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts-section-generator: Add pilot run artifacts and regression test suite

https://gerrit.wikimedia.org/r/1313903

Change #1313903 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts-section-generator: Add pilot run artifacts and regression test suite

https://gerrit.wikimedia.org/r/1313903

I've done a listening spot-check and most audios sound great. Below are some of the findings we need to fix:

1.https://en.wikipedia.org/wiki/Barnard%27s_Star#Description

  • "90 km/s" is read as as "ninety kilometers per S" instead of "ninety kilometers per second"
  • "is −110 km/s" is read as "is minus one hundred ten K M slash S" instead of "is minus one hundred ten kilometers per second"
  • "of 142.6±0.2 km/s" is read as "of one hundred forty-two six plus or minus zero two kilometers per S" instead of "of one hundred forty-two point six plus or minus zero point two kilometers per second"
  • "about 8.5," is read as "eight dot five comma" instead of "about eight point five,"

2.https://en.wikipedia.org/wiki/Qatna#The_campaigns_of_%C5%A0uppiluliuma_I

  • most of it sounds great. not sure whether non-English names/words are pronounced well

3.https://en.wikipedia.org/wiki/Water_fluoridation#Target_level

  • "of 1.5 mg/L" is read as "of one point five M G slash L" instead of "of one point five milligrams per liter"
  • "6 mg/day" is read as "six M G slash day" instead of "six milligrams per day"
  • surprisingly "0.05 mg/kg" is read correctly as "zero point zero five milligrams per kilogram"

4.https://en.wikipedia.org/wiki/Attack_on_Sydney_Harbour#M-24

  • "A$1.1 million" is read as "a dollar one million" instead of "one point one million Australian dollars"

5.https://en.wikipedia.org/wiki/Lightning_(Final_Fantasy) lead section

  • "Lightning (Japanese: ライトニング" is read as "Lightning Japanese Japanese letter Japanese letter Japanese letter Japanese letter Japanese letter Japanese letter".

Change #1313981 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts-section-generator: Add listening-pass normalization rules (ruleset 2026.07.23)

https://gerrit.wikimedia.org/r/1313981

Change #1313981 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts-section-generator: Add listening-pass normalization rules (ruleset 2026.07.23)

https://gerrit.wikimedia.org/r/1313981

Change #1314008 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: update tts-section-generator image to server that adds listening-pass normalization rules (ruleset 2026.07.23)

https://gerrit.wikimedia.org/r/1314008

Change #1314008 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update tts-section-generator image to server that adds listening-pass normalization rules (ruleset 2026.07.23)

https://gerrit.wikimedia.org/r/1314008

Deployed the merged patch (rMLIS86f00c632) to staging and exercised the normalization path directly and via the u-518 re-run.

Normalization spot-check (8 cases, all with NeMo enabled):

CaseInputOutputVerdict
Slash units90 km/sninety kilometers per secondfixed
Unicode minus−110 km/sminus one hundred and ten kilometers per secondfixed
Plus-minus142.6±0.2 km/sone hundred and forty two point six plus or minus zero point two kilometers per secondfixed (the 8.5, later in the same test sentence remains affected; see residual issue below)
mg/L, mg/day1.5 mg/L, 6 mg/daymilligrams per liter, milligrams per dayfixed
Currency prefixA$1.1 millionone point one million Australian dollarsfixed
CJK strip + romanizationライトニング, RaitoninguRaitoningu kept, kana strippedfixed
Time notation (regression guard)12:30, 3:1twelve thirty, three: oneno regression
CJK brackets 「」「ファイナルファンタジー」stripped entirelyfixed

u-518 re-run (the pilot's only failure detailed in P95036):

  • generation_version: kokoro-v1.0+af_heart+norm-2026.07.23-nemo-98d86449
  • 4 segments (was 7 before the fix), 72.9 s of audio, 203 words
  • Apparatus leakage: NONE (zero Old Norse transliteration tokens in the output; first words are the English prose: "Runestone U five hundred and eighteen...")
  • The interlinear strip (sections.py) and the × digit-context guard (text.py) both hold. The section that crashed synthesis three times during the pilot now generates cleanly.

100-article corpus rescan (seed 43, under ruleset 2026.07.23):

  • 1,116 generatable sections from 100 articles; 166 skipped-short; 0 article failures
  • p50: 2,096 chars (~140 s audio), p95: 5,343 chars (~356 s audio), max: 12,223 chars (~815 s audio)
  • 232 sections (20.8%) exceed an estimated 60 s of synthesis wall time; 24 (2.2%) exceed 120 s; none exceed 300 s
  • The 600 s timeout ceiling remains adequate with headroom; the long tail is manageable for batch scheduling. The drop in generatable sections versus the previous scan (1,158 to 1,116) is the interlinear strip correctly emptying transliteration pseudo-sections into the below-minimum skip.

One residual normalization issue (root-caused, deliberately deferred):

8.5, followed by a lowercase word is read as eight dot five comma. The glued 8.5, token isn't matched by any deterministic grammar, so it degrades to symbol-by-symbol reading. This is likely an upstream issue in the pinned nemo_text_processing version: the decimal grammar classifies the number portion but the trailing comma isn't handled as punctuation within the same token. Pre-NeMo workarounds are strictly worse than the bug: detaching the comma manufactures a stray punctuation token the verbalizer speaks aloud (the original CI failure), and stripping the comma removes a clause-boundary pause cue from the audio corpus-wide. Deferred rather than patched because the real fix is upstream. The upgrade does have a knock-on effect worth tracking: generation_version currently embeds the engine name (nemo) but not its version (see version.py under norm-{RULESET}-nemo-{whitelist_hash}), so an engine version bump that alters output for identical input would not be reflected in the version string. That gap should be closed in the upgrade patch. Will file a follow-up task.

Closing the pilot ran, every estimate in the intake document is now a measurement, and the compute function survived first contact with the real corpus.

The run. 50 Featured Articles (seeded, pinned revisions from the committed corpus scan snapshot), driven from deploy2003 through the mesh ingress, the exact path the DE pipeline will call: 588/589 sections generated in 8.3 hours at concurrency 1, full artifact family per section, zero transient failures, 85 deterministic skips (all text_below_minimum), one deterministic failure (root-caused below). The full run record is committed under pilot/: the results log (one JSON record per section outcome, from which every number below is independently re-derivable), the numbers pack, and the raw memory reading. The driver and summary script are committed as the reference client for the generator's contract.

The numbers (full pack: pilot/numbers_pack.md). Latency through the complete fetch, normalize, synthesize, align, transcode path: p50 43.5 s / p90 93.9 s / p99 165.6 s / max 223.6 s per section; effective end-to-end RTF 0.339 at the worst-case tier (full alignment, both codecs); calibration 16.2 chars per audio-second. Artifact sizes: Opus 587 KB mean per section, MP3 889 KB (the Apps codec choice is a ~25 GB decision at corpus scale). Corpus extrapolation: 11.8 generatable sections/article, ~82k sections, ~3,366 audio-hours, ~48 GB Opus, ~48 pod-days full-alignment drain (~39 audio-only). maxReplicas recommendation: 4 for backfill windows (~12-day corpus drain at a 32 CPU burst footprint; 8 replicas halves the calendar for double the footprint, 1-2 replicas stretches past three weeks), scale-to-1 in steady state (edit-driven writes are far below 1/s). The meta-result is estimate convergence across the project: corpus sections went 140k (first intake draft) to 85k (scan) to 82k (pilot); audio-hours 4,000 to 3,400 to 3,366; storage 55 to 48 GB. Each measurement pass moved the numbers less.

The memory envelope (closes the Phase 3 finding). Clean external-driver memory.peak across the run: 1.26 GiB (pod RESTARTS 0, so the reading is trustworthy), retroactively explaining the Phase 3 OOM at 1 Gi with precision. Recommended permanent limit: 2 Gi (peak x 1.5); staging stays at 3 Gi until the values patch lands.

The failure was worth more than the successes. The single failed section (Greece runestones, u-518, three identical synthesis_error attempts) root-caused to Kokoro's 510-phoneme context limit (P95036): interlinear-gloss transliteration apparatus (word-by-word Old Norse markup) survived section extraction, and one 400-char segment of it phonemized past the model's capacity, crashing kokoro-onnx with an IndexError that surfaced as a retryable 5xx. Three fixes fell out: (1) structural, the interlinear container class is now stripped like refbegin/citation (sections.py), so gloss apparatus never reaches the voice under any heading; the section now generates cleanly (4 segments, 72.9 s, zero apparatus tokens, verified above); (2) contract, an isvc-side phoneme-count guard plus a new deterministic taxonomy code text_not_synthesizable will be in a follow up patch so this failure class becomes a recorded skip instead of wasted retries, ours and DE's; (3) collateral, the investigation caught a regression in our own 2026.07.20 ruleset (global × replacement colliding with runic word separators), fixed with a digit-context guard.

The listening spot-check (findings and verification in the two comments above): a deliberately adversarial 13-file sample across article types; overall verdict "most audios sound great," with five defect classes found and four fixed in ruleset 2026.07.23 (rMLIS86f00c632, text.py): slash units, Unicode minus, plus-minus detachment, currency prefixes, and non-Latin script stripping with romanization kept, which is v1's explicit stance pending multilingual model-servers. The fifth is root-caused to an upstream nemo_text_processing 1.1.0 bug fixed upstream in 1.2.0; the pin bump plus embedding the engine version in generation_version will be tracked in a follow up patch. Remaining minor observations (letter-digit designations, "No. 37", ratio colons, zero-duration trailing words at segment boundaries, unit-symbol whitelist candidates) are logged for the systematic voice/normalization evaluation, not patched ad hoc.

Intake document updated. Every estimate row in the Prep Pantry intake now carries a measured or extrapolation-labeled value with its source; the reliability row (588/589, zero transient failures in 8.2 hours, one root-caused deterministic failure) and the capacity envelope are pilot-derived. The document DE reviews describes the compute function their pipelines would actually invoke, verified end to end through the path they would call it on.

Change #1314347 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts-section-generator: Refresh corpus_scan.json following new normalization rules (ruleset 2026.07.23)

https://gerrit.wikimedia.org/r/1314347

Change #1314347 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts-section-generator: Refresh corpus_scan.json following new normalization rules (ruleset 2026.07.23)

https://gerrit.wikimedia.org/r/1314347