Page MenuHomePhabricator

Q4 FY2025-26 Goal: Text-to-Speech V0 Prototype
Closed, ResolvedPublic

Description

Hypothesis

If we create a V0 text-to-speech service that generates section-level audio in offline batches, stores it in ordered, retrievable form, and exposes audio file URLs to clients through a low-latency API, then the Apps team will be able to validate at a high level whether presenting audio content to readers is appealing - and, if it is, learn where and how it's most useful.

Request details

Please respond to the following questions, and provide as much detail as possible for each.

Scoping details

  • Use case: Describe the user-facing experience(s) that this model will serve. Who is the intended audience? Where and how will the model outputs be surfaced to users? Please feel free to link to any demos, prototypes, design files, etc.

Readers are looking for low-friction ways to get information, and we are interested in hands-free audio features to meet that opportunity. Two use cases we envision: (1) As a reader, I can tap/click a button and have Wikipedia read an article to me, with navigation controls (including skip ahead/back [10s], skip to section, [0.75-2]x speed). (2) As a reader, I can speak a question into the in-article search have Wikipedia read back the answer to me as it jumps to the highlighted answer within the article.

  • Model purpose: What should the model do? What does it need to predict or generate?

Model should generate audio (speech) at varying speeds from article content as well as timestamps that will allow navigation.

  • Goal: What's the goal of this user experience? What patterns in user behavior do we want to impact? What metrics will let us know we're successful?

We want to help readers see that Wikipedia is an easy place to find information and thereby encourage them to come back more.
North star metric is 21d logged-out reader retention. We'd also look at usage metrics, e.g., click-through.

  • Prior art: How much of the UI for this experience has already been developed and/or tested? Are there any previous models or manually-created rules that we can learn from?

I think Apps previously did an early exploration with ElevenLabs on a version of this feature that didn't pass muster.

Prioritization details

  • Timing: When are you hoping to launch an experiment or feature using this model? How flexible is your timeline? Is there any other planned work that's blocked by this experiment or feature?

FY26-27, flexible and would love to partner to determine work-back timing together

  • KR impact: Which KRs are enabled by this project, and how critical is this project for moving the needle on those KRs?

OW3.1

Other comments

  • [Optional] Model requirements: If you have any specific concerns around model performance (latency, cost, etc.) or model output quality (likelihood of false positives, ability to detect all possible instances, etc.), please note them here.
  • [Optional] Is there anything else you'd like to share?

Related Objects

Event Timeline

added info in description above, thanks so much!

Cross-posting some thoughts that @Dbrant shared on Slack:

Here are some thoughts I had about how this might work architecturally:

  • All of the audio would be pregenerated and stored as static .mp3 files. i.e. we wouldn't want to do any kind of real-time or on-demand audio generation.
  • For every article, there would be an audio file per section of the article, so that it would be possible to jump to a section, and then seek within the audio of that section.
    • it would make it simple to navigate from one section to another, and would enable features that work better with per-section audio; I'm thinking of a playlist of lead-sections of the most popular articles etc.
    • it would also improve efficiency of audio regeneration: if we know the section where an edit was made, we only need to regenerate that section, instead of the whole article.
  • The audio for all articles would be re-generated on a rolling ~weekly basis, to account for new edits made to articles. I'm imagining a service that runs continuously and (a) provides an API endpoint that serves the .mp3 URLs based on article/section title, and (b) schedules audio regeneration for articles that are out of date.
  • Given the sizeable storage space requirements of such a system, we could consider limiting coverage to the top ~100 or 1000 most-visited articles, and scale from there.

Just jotting some requirements down to make sure we're all aligned.

Model Requirements
  • Natural, expressive, human-like speech with conversational flow
  • Multi-language coverage — suggest starting with English and expanding as needed
Some Candidate Models for initial exploration (will need to be enriched with more
Architecture & Infrastructure Considerations
  • Generate MP3 files per article section, store them (Ceph? Wikimedia Commons?)
  • REST interface needed for retrieval — need to investigate how Wikimedia Commons handles this
  • Audio and text must be indexed together for text-to-sound navigation
  • Schedule re-generation for sections that have changed (weekly rolling basis as mentioned above). After the experiment this would most likely need to be a stream
Proposed Initial Experiment Scope
  • Limit to top 100–1000 most-visited articles
  • English only to start
  • Generate per-section MP3s, storage to be decided (probably Ceph)
  • Build a small service with an endpoint to retrieve audio by article/section
Next Steps
  1. Investigate Ceph access patterns and whether Wikimedia Commons has existing infrastructure we can reuse
  2. Run quick PoC with the candidate models above to compare quality
  3. Estimate storage costs for 100–1000 articles × average sections
Things to figure out
  • storage: where will the mp3s be stored? on commons or in Ceph directly?
  • retrieval: how are they going to be retrieved. We'll likely need an article+section+revision index
Sucheta-Salgaonkar-WMF renamed this task from AI/ML Model Request: Text-to-Speech to Q4 FY2025-26 Goal: Text-to-Speech.Apr 27 2026, 2:08 PM
Sucheta-Salgaonkar-WMF added a project: Goal.
Sucheta-Salgaonkar-WMF updated the task description. (Show Details)

As we've been exploring ways to enable the Readers teams SWEs to test HuggingFace TTS models in T424378, we came across a couple of useful leaderboards that can help in selecting high-quality, well-tested models:

These are good starting points for identifying SOTA models with great speech quality.

Weekly Update:

  • There are no freely hosted TTS models available on the free tier HF Inference API (T424378#11859241)
  • Only 4 TTS models currently available on the paid providers HF Inference API (T424378#11860001). This allows SWEs to test a small set of models using the WMF HF token. However, the selection is limited.
  • Explored running TTS models locally (T424378#11861629, T424378#11865306). Enables testing a much wider range of models. Provides more control over voices, parameters, and evaluation.
  • Identified TTS leaderboards as useful resources for discovering high-quality models (T419288#11865561). Helpful for prioritizing newer, state-of-the-art models with better speech quality.

I'm sharing a draft of solution architecture we discussed in our ML Team Meeting for this problem. The approach is similar to the prototype solution developed by Kevin.
It could accommodate both batch jobs and real-time updates. End users would communicate with the backend via /generate endpoint to enqueue new TTS jobs or via /audio backend to retrieve path to the generated TTS files.

image.png (1,875×1,092 px, 166 KB)

Text chunking strategy for TTS usecase is typically sentences for faster first response, proper pauses and intonation at the end of each statement. if you cache the audio by hash(sentence)->audio chunk, we can have higher cache hit ratio in the case of new article revisions.

Change #1284645 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] feat: add qwen36-27b model server for Qwen 3.6 FP8 inference

https://gerrit.wikimedia.org/r/1284645

Weekly Update:

  • Followed up with Dmitry (Apps SWE) regarding TTS prototype requirements
  • Dmitry specified that the Apps team's key requirement for this initial phase is an API endpoint that serves .mp3 URLs based on Wikipedia article sections:
  • I set up a TTS prototype that fetches Wikipedia articles, cleans the text (removes citation brackets, edit tags, etc), splits content by section, and generates .mp3 files asynchronously.
  • Next step is hosting this prototype on toolforge so the Apps team can access it
    • started preparing for this by running resource benchmarks on ml-lab to understand how much resources will be required on toolforge (T424378#11903226)
    • requested for increased resources for this prototype on toolforge (T425804)

Weekly Update:

Wikipedia TTS prototype architecture.jpg (1,400×783 px, 83 KB)

Weekly Update:

  • Communicated the v0 release of the TTS prototype with the Apps team to test and provide feedback
  • Started iterating on the feedback provided to fix issues reported in v0
    • Limitation: v0 doesn't handle subscripts and superscripts well, as detailed in: T426756#11944373
    • Fixed text normalization edge cases, and the TTS prototype now correctly handles: decimals, percentages, temperature units, measurements, dates, currency, ordinals, time, contextual abbreviations, and number ranges. (T426756#11944379)
    • Added a customizable nemo_whitelist.tsv. This list can be expanded whenever we notice new custom/wikipedia-specific/domain-specific terms that aren't being spoken well. (T426756#11944396)
      • e.g previously “UNESCO” was spoken as “an eh sko”, now it is “yoo neh sko”
  • Discussed in ML<>Apps meeting, current limitations experienced in v0 (hosted on toolforge) to be fixed in v1 (hosted in LiftWing):
    • Currently toolforge provides only 24GB RAM and CPU-only compute. These resource constraints affect both generation speed and serving throughput for .mp3 files. LiftWing resources and GPU compute in prod will improve the performance.
    • Wikipedia's plain-text extract API strips formatting, which causes the subscript/superscript edge cases noted above. v1 will explore an HTML-based extraction pipeline to preserve structured content semantics (tables, math formulas, etc)
    • Generated .mp3 files are currently stored locally. We are consulting across teams for a more scalable media storage solution (Swift/Ceph)

Weekly Update:

  • Continued iterating on the feedback provided to fix issues reported in the TTS prototype
  • MLOps:
    • Added initialize_nemo_cache script that pre-compiles the NeMo grammar cache so the celery worker starts faster.
      • This avoids the k8s pod startup penalty (CrashLoopBackOff) from the 60+ second compilation timeout.
    • Added download_models script to replace manual wget commands.
      • It downloads all model files (Kokoro TTS, voice profiles, and Wav2Vec2 alignment) into the directory structure expected by config.py.

Weekly Update:

  • Continued iterating on the feedback provided to fix issues reported in the TTS prototype
    • Added word-level timestamps to TTS prototype (T427488)
      • Fixed text normalization edge case for orphaned punctuations. (T426756#11966668)
      • The TTS prototype now produces both section audio (.mp3) and caption files (.vtt) that can be used for the follow-along reading experience (T427488#11973196)
      • We have also added a demo of this feature in the TTS prototype UI. Now when you play a section's audio, the transcript below the audio player highlights each spoken word in real-time so that you can follow-along: ▶️ F85986028
      • NB: Before we added caption generation, the prototype was only generating .mp3 files in an already resource-constrained Toolforge environment (T419288#11948239). With .vtt now being generated alongside .mp3 using the same hosting resources, the overall audio generation speed and serving throughput for the pipeline are now slower. This is an expected limitation for the Toolforge prototype and will be resolved when we migrate the compute-heavy inference to LiftWing's dedicated GPU infrastructure in v1. (T427488#11975904)
  • MLOps:
    • Switched from running celery workers using --pool solo to --concurrency=1 with --max-tasks-per-child=5.
      • This forces the python process to gracefully recycle after processing 5 sections, actively flushing accumulated ONNX Runtime C++ memory and large audio arrays back to the OS. This resolves the intermittent OOM crashes.
    • To safely accommodate the heavier dual-model pipeline (Kokoro TTS + Wav2Vec2 aligner), we re-allocated our Toolforge quota.
      • Changed from 10 replicas at 2Gi RAM and 1CPU (T419288#11924778) to 5 replicas at 4Gi RAM and 2CPUs each. This gives each individual worker the necessary memory headroom to process large article sections, while maintaining our strict 20Gi total worker footprint (leaving a 4Gi buffer for the web server).
    • Added PYTHONUNBUFFERED=1 for real-time log streaming to k8s for faster debugging.

Weekly Update:

Following up here with some bullet points after the Architecture review meeting that took place last week. The Critical decisions section is something that I have added in the end and was not discussed in the meeting. Please add anything important I might have mistakenly omitted.

Decisions

  1. Commons is ruled out for storing the audio files. It retains full history permanently and can't truly overwrite or delete, which doesn't work for data that regenerates on every revision.
  2. The application will write to an S3 interface. This decouples the app from the underlying storage so the backend can be swapped by changing the endpoint.
  3. The proof of concept can use an S3 frontend with a Swift backend, scoped to Featured Articles (~7k), with the option to scale or relocate (e.g. to Ceph) later.

Action Items/Concerns

  1. Run the project through Prep pantry to gather and document requirements (Guilherme to support; the form should cover what Data Persistence needs to advise).
  2. Engage Data Persistence on the storage and indexing design, including whether LAC (Linked Artifact Cache), page properties, or something else serves as the index.
  3. Bring in additional SREs for storage solutions and SRE Traffic for CDN/delivery (Varnish caching impact).
  4. Talk to PMs about the access-constraint direction (e.g. limiting to logged-in users or app-only access for scraper protection).
  5. Build in a purge / force-regenerate mechanism for vandalism, ideally tied to MediaWiki's cache-purge capabilities.

Critical Decisions to enable an experiment

  • Which model are we targeting and which languages?
  • Even if we store the files in swift, we still need to decide how these will be served in the App. We need to look at the latency requirements for the experiment and engage with Data Persistence and Traffic to see how this can tackled if caching on CDN is required.
  • Will we go through Prep pantry for enabling the experiment or is this something that we’re aiming for the full rolllout? In any case, this is something that we should start asap.

Satus Update:

Over time, we've gathered the Apps team's requirements and built the TTS prototype, now live on Toolforge. It serves spoken Wikipedia articles with word-level captions. We've also worked on pre-generating a set of ~7,000 Featured Articles through a batch pipeline. Pre-generated articles are available in the audio library.

As we plan to scale this from a Toolforge prototype (v0) to production on LiftWing (v1), I've shared a report covering the API endpoints, user interface, and the limitations (hardware, storage, text extraction, and language support) we plan to address in v1: T424378#12068767

Thanks to everyone who helped with requirements gathering, model exploration, quota requests, and engaging with various stakeholders along the way.

v1 work is ongoing (T419288#12049854) and we'll share the tracking phab task once the new goal is ready.

Edit
v1 work is being tracked in: T430676

Sucheta-Salgaonkar-WMF renamed this task from Q4 FY2025-26 Goal: Text-to-Speech to Q4 FY2025-26 Goal: Text-to-Speech V0 Prototype.Jun 30 2026, 2:28 PM

Resolving this as the prototype is complete and demonstrated in https://wiki-tts.toolforge.org/.
Congrats for the great work everyone!
As mentioned above subsequent work for a v1 experiment is tracked in T430676: Q1 FY2026-27 Text-to-Speech V1 Experiment