Page MenuHomePhabricator

Migrate TTS inference logic from v0 to v1 (without orchestration, storage, and serving layers)
Closed, ResolvedPublic

Description

Following the TTS service v1 planning meeting (T419288#12049854), the ML team owns the inference layer, the Data Engineering team owns the orchestration layer, Data Persistence owns storage, and Traffic owns serving. In this task, we shall implement the inference layer: a custom KServe model-server, hosted on LiftWing, that turns pre-normalized article text into spoken audio with word-level timestamps. It is the v1 successor to the inference half of the v0 Toolforge prototype (T419288#11924778, T424378#12068767).

Scope of this task: the model-server only. The orchestrator, storage, and serving layers are out of scope here and tracked separately. For this to make sense, here is where the rest of v1 lives.

  • Orchestrator: The stateful glue this server depends on: resolves title -> page_id -> revision, fetches + cleans article text (v0's wikipedia_utils.py / text.py), runs _split_text, checks the index for existing audio, calls this model-server per section, writes results to storage, dedupes concurrent generation (v0's Redis lock pattern), and drives edit-triggered regeneration. Successor to v0's routes.py + submit_articles.py.
  • Storage: Where audio + timestamps live, behind an S3 interface (Swift-backed for the PoC, scoped to ~7k Featured Articles, relocatable to Ceph). Replaces v0's local audio_output/. Includes the revision-scoped key schema and the index (LAC / page properties / TBD) that answers "does audio exist for this revision, and where?"
  • Serving / delivery: How the mobile app retrieves audio, including CDN / Varnish caching and the cache-invalidation / purge path for vandalism and stale revisions. Replaces v0's FastAPI file responses.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Kevin Baziraoperations/dnsmaster+3 -0
Kevin Baziraoperations/deployment-chartsmaster+1 -1
Kevin Baziraoperations/deployment-chartsmaster+27 -0
Kevin Baziramachinelearning/liftwing/inference-servicesmain+7 K -8 K
Kevin Baziraoperations/deployment-chartsmaster+3 -3
Kevin Baziraoperations/deployment-chartsmaster+3 -3
Kevin Baziramachinelearning/liftwing/inference-servicesmain+57 -9
Kevin Baziraoperations/deployment-chartsmaster+1 -0
Klausmanlabs/privatemaster+5 -0
Kevin Baziraoperations/deployment-chartsmaster+2 -2
Kevin Baziramachinelearning/liftwing/inference-servicesmain+173 -28
Kevin Baziramachinelearning/liftwing/inference-servicesmain+122 -10
Kevin Baziramachinelearning/liftwing/inference-servicesmain+83 -8
Kevin Baziraoperations/deployment-chartsmaster+7 -3
Kevin Baziramachinelearning/liftwing/inference-servicesmain+35 -0
Kevin Baziraoperations/deployment-chartsmaster+5 -1
Kevin Baziramachinelearning/liftwing/inference-servicesmain+17 -4
Kevin Baziramachinelearning/liftwing/inference-servicesmain+40 -1
Kevin Baziraoperations/deployment-chartsmaster+1 -1
Kevin Baziramachinelearning/liftwing/inference-servicesmain+28 -2
Kevin Baziraoperations/deployment-chartsmaster+18 -0
Kevin Baziramachinelearning/liftwing/inference-servicesmain+55 -0
Kevin Baziraintegration/configmaster+15 -0
Kevin Baziramachinelearning/liftwing/inference-servicesmain+175 -34
Kevin Baziramachinelearning/liftwing/inference-servicesmain+841 -0
Kevin Baziramachinelearning/liftwing/inference-servicesmain+749 -0
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1307674 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts: containerize model-server unit tests

https://gerrit.wikimedia.org/r/1307674

Change #1307680 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[integration/config@master] inference-services: Add CI pipeline jobs for tts model-server

https://gerrit.wikimedia.org/r/1307680

Change #1307680 merged by jenkins-bot:

[integration/config@master] inference-services: Add CI pipeline jobs for tts model-server

https://gerrit.wikimedia.org/r/1307680

Change #1307674 merged by Kevin Bazira:

[machinelearning/liftwing/inference-services@main] tts: containerize model-server unit tests

https://gerrit.wikimedia.org/r/1307674

Change #1308012 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy tts isvc in experimental ns

https://gerrit.wikimedia.org/r/1308012

Change #1308012 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy tts isvc in experimental ns

https://gerrit.wikimedia.org/r/1308012

The TTS model-server has been deployed in the LiftWing experimental namespace. It is currently available through an internal endpoint that can only be accessed by tools that run within the WMF infrastructure (e.g deploy2002, stat1008, etc):

# pod running in experimental ns
$ kube_env experimental ml-staging-codfw
$ kubectl get pods
NAME                                                              READY   STATUS    RESTARTS   AGE
tts-predictor-00001-deployment-798db56665-m4ssb                   3/3     Running   0          41s

# query tts isvc
$ curl -X POST "https://inference-staging.svc.codfw.wmnet:30443/v1/models/tts:predict" \
-H  "Host: tts.experimental.wikimedia.org" \
-H "Content-Type: application/json" \
-d '{
    "segments": [
        {"text": "Hello world.", "voice": "af_heart"}
    ]
}'

{
    "audio_b64": "wI2LN1SD3bdxhoi45...qQ85qMNgOZJ1cDk=",
    "sample_rate": 24000,
    "duration_ms": 1045.3,
    "timestamps": [
        {
            "word": "Hello",
            "start_ms": 80.0,
            "end_ms": 280.0
        },
        {
            "word": "world.",
            "start_ms": 420.0,
            "end_ms": 720.0
        }
    ]
}

Change #1308631 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts: rebuild Kokoro ONNX session with full graph optimization

https://gerrit.wikimedia.org/r/1308631

Change #1308631 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts: set explicit intra_op_num_threads on Kokoro's ONNX session

https://gerrit.wikimedia.org/r/1308631

Change #1309157 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy TTS image that sets explicit intra_op_num_threads on Kokoro's ONNX session

https://gerrit.wikimedia.org/r/1309157

Change #1309157 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy TTS image that sets explicit intra_op_num_threads on Kokoro's ONNX session

https://gerrit.wikimedia.org/r/1309157

Change #1309557 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts: add per-request timing, RTF logging, and startup warm-up

https://gerrit.wikimedia.org/r/1309557

Change #1309557 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts: add per-request timing, RTF logging, and startup warm-up

https://gerrit.wikimedia.org/r/1309557

Change #1309605 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts: make Wav2Vec2 aligner threads env-tunable (W2V2_THREADS)

https://gerrit.wikimedia.org/r/1309605

Change #1309605 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts: make Wav2Vec2 aligner threads env-tunable (W2V2_THREADS)

https://gerrit.wikimedia.org/r/1309605

Change #1309616 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] tts: ml-services: deploy latest TTS isvc model-server

https://gerrit.wikimedia.org/r/1309616

Change #1309616 merged by jenkins-bot:

[operations/deployment-charts@master] tts: ml-services: deploy latest TTS isvc model-server

https://gerrit.wikimedia.org/r/1309616

Change #1309645 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts: serialize predict() since pipeline is not thread-safe

https://gerrit.wikimedia.org/r/1309645

Change #1309645 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts: serialize predict() since pipeline is not thread-safe

https://gerrit.wikimedia.org/r/1309645

Change #1309660 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy latest TTS isvc that serializes predict() since pipeline is not thread-safe

https://gerrit.wikimedia.org/r/1309660

Change #1309660 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy latest TTS isvc that serializes predict() since pipeline is not thread-safe

https://gerrit.wikimedia.org/r/1309660

Change #1309672 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts: overlap alignment with next-chunk synthesis

https://gerrit.wikimedia.org/r/1309672

Change #1309672 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts: overlap alignment with next-chunk synthesis

https://gerrit.wikimedia.org/r/1309672

Change #1309677 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts: add int16 PCM response encoding (new default), halving payload

https://gerrit.wikimedia.org/r/1309677

Change #1309677 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts: add int16 PCM response encoding (new default), halving payload

https://gerrit.wikimedia.org/r/1309677

Change #1309686 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts: add timestamps request parameter (full | proportional | none)

https://gerrit.wikimedia.org/r/1309686

Change #1309686 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts: add timestamps request parameter (full | proportional | none)

https://gerrit.wikimedia.org/r/1309686

Change #1309695 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy latest TTS isvc model-server

https://gerrit.wikimedia.org/r/1309695

Change #1309695 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy latest TTS isvc model-server

https://gerrit.wikimedia.org/r/1309695

Following the initial deployment in T430536#12093268, we profiled and optimized this isvc as shown below:

Key bottleneck: kokoro-onnx defaulted intra_op_num_threads=0, sizing the thread pool from the host's 96 cores instead of the pod's 8-CPU cgroup quota, causing ~12x oversubscription. Fixed by building the session explicitly via Kokoro.from_session with tunable KOKORO_THREADS / W2V2_THREADS.

Thread-safety bug found and fixed before consumers hit it: concurrent predict() calls corrupt output (identical input -> 14.5–26.3s audio across 4 calls; serialized -> exactly 16.81s each). Suspected espeak phonemizer global state. Mitigated with a per-pod predict() lock + containerConcurrency: 1.

Optimizations shipped:

  • Alignment of chunk i overlaps synthesis of chunk i+1 (alignment time mostly hidden)
  • Responses default to int16 PCM (payload halved; pure quantization error vs float32)
  • timestamps request parameter: full (CTC alignment), proportional (char-weighted, near-zero cost), or none (audio-only, fastest)

Results (staging): RTF 3.8 -> 0.27 (full timestamps) / 0.22 (audio-only). End-to-end for "Hello world.": 7.4s -> 0.59s (full) / 0.47s (none). Timestamps identical to the original deployment.

API contract note for consumers: default encoding is now int16 (previously float32; pcm_f32le available on request). timestamps field controls alignment
cost.

Here is the original benchmark request (before) and the optimization results (after):

before (7.369s with timestamps)
$ time curl -X POST "https://inference-staging.svc.codfw.wmnet:30443/v1/models/tts:predict" \
-H  "Host: tts.experimental.wikimedia.org" \
-H "Content-Type: application/json" \
-d '{
    "segments": [
        {"text": "Hello world.", "voice": "af_heart"}
    ]
}'

{"audio_b64":"wI2LN1SD3bdxhoi45...VHfTlYqQ85qMNgOZJ1cDk=","sample_rate":24000,"duration_ms":1045.3,"timestamps":[{"word":"Hello","start_ms":80.0,"end_ms":280.0},{"word":"world.","start_ms":420.0,"end_ms":720.0}]}

real	0m7.369s
user	0m0.012s
sys	0m0.004s
after (0.588s with timestamps / 0.473s without timestamps)
$ time curl -X POST "https://inference-staging.svc.codfw.wmnet:30443/v1/models/tts:predict" \
-H  "Host: tts.experimental.wikimedia.org" \
-H "Content-Type: application/json" \
-d '{
    "segments": [
        {"text": "Hello world.", "voice": "af_heart"}
    ],
    "timestamps": "full"
}'

{"audio_b64":"AAAAAP7//////wAAAAAAAAAA...GAAYABwAIAAkACAAJAAgABwAHAAQABwAHAA==","encoding":"pcm_s16le","timestamps_mode":"full","sample_rate":24000,"duration_ms":1045.3,"timestamps":[{"word":"Hello","start_ms":80.0,"end_ms":280.0},{"word":"world.","start_ms":420.0,"end_ms":720.0}]}

real	0m0.588s
user	0m0.044s
sys	0m0.004s
$
$
$ time curl -X POST "https://inference-staging.svc.codfw.wmnet:30443/v1/models/tts:predict" \
-H  "Host: tts.experimental.wikimedia.org" \
-H "Content-Type: application/json" \
-d '{
    "segments": [
        {"text": "Hello world.", "voice": "af_heart"}
    ],
    "timestamps": "none"
}'

{"audio_b64":"AAAAAP7//////wAAAAAAAAAA...GAAYABwAIAAkACAAJAAgABwAHAAQABwAHAA==","encoding":"pcm_s16le","timestamps_mode":"none","sample_rate":24000,"duration_ms":1045.3,"timestamps":[]}

real	0m0.473s
user	0m0.031s
sys	0m0.012s

Change #1311831 had a related patch set uploaded (by Klausman; author: Klausman):

[labs/private@master] hiera/deployment_server: Add pseudo-secrtes for new LW service tts-section-generator

https://gerrit.wikimedia.org/r/1311831

Change #1311831 merged by Klausman:

[labs/private@master] hiera/deployment_server: Add pseudo-secrets for new LW service tts-section-generator

https://gerrit.wikimedia.org/r/1311831

Change #1312527 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: Raise Knative revision timeout to 600s for TTS isvc

https://gerrit.wikimedia.org/r/1312527

Change #1312527 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Raise Knative revision timeout to 600s for TTS isvc

https://gerrit.wikimedia.org/r/1312527

Change #1312747 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts-section-generator: Translate sup/sub markup digits before text flattening

https://gerrit.wikimedia.org/r/1312747

Change #1312747 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts-section-generator: Translate sup/sub markup digits before text flattening

https://gerrit.wikimedia.org/r/1312747

Change #1312760 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy TTS image that translates sup/sub markup digits before text flattening

https://gerrit.wikimedia.org/r/1312760

Change #1312760 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy TTS image that translates sup/sub markup digits before text flattening

https://gerrit.wikimedia.org/r/1312760

Change #1312769 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: update both tts isvc and tts-section-generator images for text normalization

https://gerrit.wikimedia.org/r/1312769

Change #1312769 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update both tts isvc and tts-section-generator images for text normalization

https://gerrit.wikimedia.org/r/1312769

The v0 text normalization carry-over (T426756's superscript/subscript limitation) is now resolved and deployed in staging.

The v1 HTML-based extraction now translates <sup>/<sub> digit content to Unicode before flattening, so the existing normalization pipeline speaks it correctly. The citation-strip loop runs first, so <sup class="mw-ref">[3]</sup> is stripped before translation ever sees it. Footnote numbers never become exponents.

1.Text normalization code paths (via the production image's NeMo stack):

$ kubectl exec -it tts-section-generator-main-68d64d7b9c-w75vn -c tts-section-generator-main -- \
  python3 -c "
from tts_generator.text import clean_spoken_text, init_nemo
init_nemo()
# markup-derived scientific notation (the new capability)
print(clean_spoken_text('The mass is 5.97×10²⁴ kg.'))
# single superscript still works (no regression from reordering)
print(clean_spoken_text('An area of 30 km².'))
# subscripts read as plain digits (v0-validated behavior)
print(clean_spoken_text('H₂O and CO₂ are molecules.'))
"
 NeMo-text-processing :: INFO     :: Post processing graph was restored from /home/somebody/nemo-grammars/en_tn_post_processing.far.
 NeMo-text-processing :: INFO     :: ClassifyFst.fst was restored from /home/somebody/nemo-grammars/en_tn_True_deterministic_cased_nemo_whitelist.tsv_tokenize.far.
 NeMo-text-processing :: INFO     :: VerbalizeFinalFst graph was restored from /home/somebody/nemo-grammars/en_tn_True_deterministic_verbalizer.far.

The mass is five point nine seven times ten to the power of twenty four kilograms.
An area of thirty square kilometers.
H two O and CO two are molecules.

2.HTML extraction (markup -> Unicode translation before flattening):

$ kubectl exec -it tts-section-generator-main-68d64d7b9c-w75vn -c tts-section-generator-main -- \
  python3 -c "
from tts_generator.sections import extract_sections
html = '''<html><body><section data-mw-section-id=\"0\">
<p>The mass is 5.97×10<sup>24</sup> kg, an area of 1 m<sup>2</sup>,
water is H<sub>2</sub>O, the 4<sup>th</sup> planet<sup class=\"mw-ref\">[3]</sup>.</p>
</section></body></html>'''
print(extract_sections(html)[0].raw_text)
"

The mass is 5.97×10²⁴ kg, an area of 1 m², water is H₂O, the 4th planet.

3.Contract-level API test using Earth (page_id=9228, rev_id=1362915217):

generation_version with new ruleset
$ curl -s "https://tts-section-generator.k8s-ml-staging.discovery.wmnet:31443/sections?wiki_id=enwiki&page_id=9228&rev_id=1362915217" | python3 -m json.tool > sections_new.txt
$ cat sections_new.txt
{
    "wiki_id": "enwiki",
    "page_id": 9228,
    "rev_id": 1362915217,
    "revision_timestamp": "2026-07-07T00:02:54Z",
    "generation_version": "kokoro-v1.0+af_heart+norm-2026.07.20-nemo-98d86449",
    "sections": [
        {
            "section_id": "lead",
            "title": "Lead",
            "level": 1,
            "generatable": true,
            "char_count": 4124,
            "content_sha256": "457971c72fa3de06f9d9a44f4e1db345e59f5c6e5a92df0459376560f94f15fe"
        },
        {
            "section_id": "etymology",
            "title": "Etymology",
            "level": 2,
            "generatable": true,
            "char_count": 2749,
            "content_sha256": "444a0358044f00d514d7d2f4d283bc0499ab4b351d3de9760e58b3e0647b6a35"
        },
        {
            "section_id": "natural-history",
            "title": "Natural history",
            "level": 2,
            "generatable": false,
            "char_count": 0,
            "skip_reason": "text_below_minimum"
        },
        {
            "section_id": "formation",
            "title": "Formation",
            "level": 3,
            "generatable": true,
            "char_count": 1336,
            "content_sha256": "c443baad3f582bfd0fbfba87c2dc1bd0de868b10d74c6287948c2fadbcd1a032"
        },
        {
            "section_id": "after-formation",
            "title": "After formation",
            "level": 3,
            "generatable": true,
            "char_count": 2918,
            "content_sha256": "d46867c01bd5577c71f2ddb9b7189bf637b69f40d6f2fde0caeb1674805247ff"
        },
        {
            "section_id": "origin-of-life-and-evolution",
            "title": "Origin of life and evolution",
            "level": 3,
            "generatable": true,
            "char_count": 2487,
            "content_sha256": "341e85b084330edd6e92150a5f09af1bfebf5949a1f861034a06d38f528a88c5"
        },
        {
            "section_id": "future",
            "title": "Future",
            "level": 3,
            "generatable": true,
            "char_count": 1969,
            "content_sha256": "f8a78d2f28d064e6390ab2ed139d63e2857fbf45a8fe085e9e65bf9b28bd960d"
        },
        {
            "section_id": "composition-and-structure",
            "title": "Composition and structure",
            "level": 2,
            "generatable": true,
            "char_count": 651,
            "content_sha256": "b642cef4f7af4f4106e52ab21fed34a20808ad86152bee25616c89595cf44069"
        },
        {
            "section_id": "chemical-composition",
            "title": "Chemical composition",
            "level": 3,
            "generatable": true,
            "char_count": 1070,
            "content_sha256": "ed32ee5a52a1600105ce52667403cf23ee771e65328d76f4fd177e6f75da647d"
        },
        {
            "section_id": "internal-structure",
            "title": "Internal structure",
            "level": 3,
            "generatable": true,
            "char_count": 1490,
            "content_sha256": "c7fc1ec3dbdeeffe0ac880c189ead44fff087dd18d508bb18c3be39eb05b8425"
        },
        {
            "section_id": "internal-heat",
            "title": "Internal heat",
            "level": 3,
            "generatable": true,
            "char_count": 1567,
            "content_sha256": "93cc2bfa161ee450a04c935dba98ee7dc9e99990b0beb4c7c84b93381fe7ba50"
        },
        {
            "section_id": "tectonic-plates",
            "title": "Tectonic plates",
            "level": 3,
            "generatable": true,
            "char_count": 2237,
            "content_sha256": "08b86f38b63cb2de8277157090f4165340ae1b380345734d40f20f7d9f90c60f"
        },
        {
            "section_id": "crust",
            "title": "Crust",
            "level": 3,
            "generatable": true,
            "char_count": 824,
            "content_sha256": "fbc4fba2dab735d96e55914d1fb7fbb09edf38c884d7371982ac68692083c0ca"
        },
        {
            "section_id": "surface",
            "title": "Surface",
            "level": 3,
            "generatable": true,
            "char_count": 2687,
            "content_sha256": "12d41fb244e2a6c78212d76eccd7fa7cd98e2e47565806b90b2e1f9f34eed739"
        },
        {
            "section_id": "surface-topography",
            "title": "Surface topography",
            "level": 3,
            "generatable": true,
            "char_count": 609,
            "content_sha256": "b51b59d753e53aaa0466034a1f7b0ac98220dd8d61275f2bde0c2d07d95c9964"
        },
        {
            "section_id": "shape",
            "title": "Shape",
            "level": 3,
            "generatable": true,
            "char_count": 1397,
            "content_sha256": "fccb53cd0f7665fba18b0e7b452658415a65429b38293b5c681efd4e1e1093ff"
        },
        {
            "section_id": "gravity-and-magnetic-field",
            "title": "Gravity and magnetic field",
            "level": 2,
            "generatable": false,
            "char_count": 0,
            "skip_reason": "text_below_minimum"
        },
        {
            "section_id": "magnetic-field",
            "title": "Magnetic field",
            "level": 3,
            "generatable": true,
            "char_count": 2326,
            "content_sha256": "c20c99f4a029ed3581fc8428724cd12215092dceed532986f3f1424a5a81b2d7"
        },
        {
            "section_id": "gravitational-field",
            "title": "Gravitational field",
            "level": 3,
            "generatable": true,
            "char_count": 837,
            "content_sha256": "bc5f67a9221566b057766d3636049c43745fa9f84cd095c696ff6109e06c6007"
        },
        {
            "section_id": "moon-and-orbital-space",
            "title": "Moon and orbital space",
            "level": 2,
            "generatable": false,
            "char_count": 0,
            "skip_reason": "text_below_minimum"
        },
        {
            "section_id": "moon",
            "title": "Moon",
            "level": 3,
            "generatable": true,
            "char_count": 3167,
            "content_sha256": "2f9361fded0dd7113930ef9b79d247f1155af3c4b6612d79256dab6bed6bbbb4"
        },
        {
            "section_id": "asteroids-and-artificial-satellites",
            "title": "Asteroids and artificial satellites",
            "level": 3,
            "generatable": true,
            "char_count": 996,
            "content_sha256": "7dedda8fc738bebcb2f7fb01c93446776c95375082e19c2f351475aec1dde3cc"
        },
        {
            "section_id": "orbit-and-rotation",
            "title": "Orbit and rotation",
            "level": 2,
            "generatable": false,
            "char_count": 0,
            "skip_reason": "text_below_minimum"
        },
        {
            "section_id": "rotation",
            "title": "Rotation",
            "level": 3,
            "generatable": true,
            "char_count": 1520,
            "content_sha256": "762648be353c692fa825c523f9632c6e3690b4ec2b4697dddcb6ed72ec6c2bd6"
        },
        {
            "section_id": "orbit-and-location",
            "title": "Orbit and location",
            "level": 3,
            "generatable": true,
            "char_count": 2049,
            "content_sha256": "42d93415408b371944bfe6196f22c7a364c936b3326aeb784897728fd86b848a"
        },
        {
            "section_id": "axial-tilt-and-seasons",
            "title": "Axial tilt and seasons",
            "level": 3,
            "generatable": true,
            "char_count": 3629,
            "content_sha256": "50a46f3eb3025054b85003c0957b454d34a567dcb63855657eddf2206ef636f8"
        },
        {
            "section_id": "atmosphere-and-climate",
            "title": "Atmosphere and climate",
            "level": 2,
            "generatable": true,
            "char_count": 2096,
            "content_sha256": "a7454927f42cb52e776d4a227a2a7f5f41a545829ddd914f8cedfad7708cbb41"
        },
        {
            "section_id": "exosphere",
            "title": "Exosphere",
            "level": 3,
            "generatable": true,
            "char_count": 1228,
            "content_sha256": "f4a8497eacab445c1f4be8f0af56fc1c08e68d95a413408c32e751bc16fa2f86"
        },
        {
            "section_id": "upper-atmosphere",
            "title": "Upper atmosphere",
            "level": 3,
            "generatable": true,
            "char_count": 554,
            "content_sha256": "b2808742dafdc7f7ec101b93d256eb1928d8996b77111942d28db6c67fa9f834"
        },
        {
            "section_id": "troposphere",
            "title": "Troposphere",
            "level": 3,
            "generatable": true,
            "char_count": 635,
            "content_sha256": "07dbadb1fa2e56ba841af8ba127934849ea170470e6deac72a78d5e6bf29d59e"
        },
        {
            "section_id": "weather-and-climate",
            "title": "Weather and climate",
            "level": 3,
            "generatable": true,
            "char_count": 3082,
            "content_sha256": "3aa18311ac5d6d8f51fb65690d9e3c5f200990737d0c960c7f601b560d79b65f"
        },
        {
            "section_id": "hydrosphere",
            "title": "Hydrosphere",
            "level": 2,
            "generatable": true,
            "char_count": 3214,
            "content_sha256": "bd5c6758e6e2beeafb8f7b73adcbcc59218e46a0fa1993654ef43120db808294"
        },
        {
            "section_id": "biosphere",
            "title": "Biosphere",
            "level": 2,
            "generatable": true,
            "char_count": 1396,
            "content_sha256": "40caa3ca0ffc0146502be7eb53dd8b0997d52f0ddb3dc89ee5c67b873604278f"
        },
        {
            "section_id": "challenges-for-life-on-earth",
            "title": "Challenges for life on Earth",
            "level": 3,
            "generatable": true,
            "char_count": 937,
            "content_sha256": "77aefd30e07a8783021925f918d9dfdb4ea4fabe64cd3358240815af608053f7"
        },
        {
            "section_id": "human-geography",
            "title": "Human geography",
            "level": 2,
            "generatable": true,
            "char_count": 2216,
            "content_sha256": "3875e3d36976d4739076ab84cdd8028ae4ebb145d6395ec032948e7e4a1a0ee7"
        },
        {
            "section_id": "natural-resources-and-land-use",
            "title": "Natural resources and land use",
            "level": 3,
            "generatable": true,
            "char_count": 1656,
            "content_sha256": "de779234161bee412b8c0eeab616b1563573c1be86373d8f3ce8b7abfc460494"
        },
        {
            "section_id": "environmental-impact",
            "title": "Environmental impact",
            "level": 3,
            "generatable": true,
            "char_count": 1143,
            "content_sha256": "53542fcbdc2c37b6928ea3f3c69e26d844c2b402b2d28294ff74fed33b20e6ec"
        },
        {
            "section_id": "in-culture",
            "title": "In culture",
            "level": 2,
            "generatable": true,
            "char_count": 2140,
            "content_sha256": "68fed6b288e4cb6e1157c62314da43ffdc382d5d1bd389926593c0c547493431"
        }
    ]
}

Confirms generation_version: "kokoro-v1.0+af_heart+norm-2026.07.20-nemo-98d86449" i.e the new ruleset is live, all section hashes generated under the final v1 normalization rules.

4.Spoken-word check for Earth: Chemical composition:

$ curl -s -X POST "https://tts-section-generator.k8s-ml-staging.discovery.wmnet:31443/generate-section" \
  -H "Content-Type: application/json" \
  -d '{"wiki_id":"enwiki","page_id":9228,"rev_id":1362915217,"section_id":"chemical-composition","generation_config":{"artifacts":["timestamps_json"]}}' \
  | python3 -c "import json,sys; ts=json.load(sys.stdin)['artifacts'][0]['timestamps']; print(' '.join(w['word'] for w in ts))"

Earth's mass is approximately five point nine seven times ten to the power of twenty four kilograms (five point nine seven Rg). It is composed mostly of iron (thirty two point one percent by mass), oxygen (thirty point one percent), silicon (fifteen point one percent), magnesium (thirteen point nine percent), sulfur (two point nine percent), nickel (one point eight percent), calcium (one point five percent), and aluminum (one point four percent), with the remaining one point two percent consisting of trace amounts of other elements. Due to gravitational separation, the core is primarily composed of the denser elements: iron (eighty eight point eight percent), with smaller amounts of nickel (five point eight percent), sulfur (four point five percent), and less than one percent trace elements. The most common rock constituents of the crust are oxides. Over ninety nine percent of the crust is composed of various oxides of eleven elements, principally oxides containing silicon (the silicate minerals), aluminum, iron, calcium, magnesium, potassium, or sodium.

5.Below is the audio generated for: https://en.wikipedia.org/w/index.php?title=Earth&diff=1362915217#Chemical_composition

Change #1312979 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] tts-section-generator: Make corpus scanner in-pod-runnable and regenerate under norm-2026.07.20

https://gerrit.wikimedia.org/r/1312979

Change #1312979 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] tts-section-generator: Make corpus scanner in-pod-runnable and regenerate under norm-2026.07.20

https://gerrit.wikimedia.org/r/1312979

Change #1318573 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy tts isvc in prod

https://gerrit.wikimedia.org/r/1318573

Change #1318573 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy tts isvc in prod

https://gerrit.wikimedia.org/r/1318573

Change #1318668 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy tts-section-generator in prod

https://gerrit.wikimedia.org/r/1318668

Change #1318668 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy tts-section-generator in prod

https://gerrit.wikimedia.org/r/1318668

Change #1318685 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/dns@master] Add tts-section-generator CNAMEs to k8s-ingress-ml-serve

https://gerrit.wikimedia.org/r/1318685

The TTS isvc has been deployed in LiftWing production (eqiad). It can be accessed by tools that run within the WMF production network, as shown below.

No GPU resources provisioned for this initial deployment (Kokoro runs comfortably on CPU). GPUs will be added either when the serving footprint expands to larger TTS models or when this isvc needs a performance boost.

$ time curl -X POST "https://inference.discovery.wmnet:30443/v1/models/tts:predict" \
-H  "Host: tts.llm.wikimedia.org" \
-H "Content-Type: application/json" \
-d '{
    "segments": [
        {"text": "Hello world.", "voice": "af_heart"}
    ]
}'

{
    "audio_b64": "AAAAAP7//////wAAAAAAAAAA...AHAAcABgAGAAYABwAIAAkACAAJAAgABwAHAAQABwAHAA==",
    "encoding": "pcm_s16le",
    "timestamps_mode": "full",
    "sample_rate": 24000,
    "duration_ms": 1045.3,
    "timestamps": [
        {
            "word": "Hello",
            "start_ms": 80.0,
            "end_ms": 280.0
        },
        {
            "word": "world.",
            "start_ms": 420.0,
            "end_ms": 720.0
        }
    ]
}

real	0m0.638s
user	0m0.060s
sys	0m0.019s

Change #1318685 merged by Klausman:

[operations/dns@master] Add tts-section-generator CNAMEs to k8s-ingress-ml-serve

https://gerrit.wikimedia.org/r/1318685