Thanks for the notice, I didn't know the new xml files would be missing pages.

Thank you!

Thank you all for reporting the mismatch. We've fixed the issue, please let us know if the problem still persists. Thanks!

I should read the document first and verify the model output more thoroughly....

I pasted the code to a few llm and here are the same two suggestions made by these models:

1. buffer at enterprise.py line 55 is not cleared, it grows indefinitely. The fix is clear the buffer at the end of each iteration, or remove the buffer.

for page in pages:
    f.write(encoder.encode(page)  + b"\n")

2. remove asyncio.sleep(0.5) at spiders.py line 104, better to let Scrapy decide the pace itself by giving it an iterator. If must batch from database, increase chunk size dramatically (e.g., 5,000–10,000 items) so spend less time query-blocking the event loop.

Thank you so much! Could you provide a JSON file for example /latest/en.json:

[{
  "date": "20260501",
  "chunks": ["pages-NS0-00-16-36422.ndjson.zst"],
  "namespace": 0,
}]

otherwise I'll have to scrape the page to know which file to download.

I'm a little worried about creating too many files might be problematic for the file system consider more Wiktionaries and namespaces will be added in the future and also delete the root html folder could take some time. Do you think it's feasible to save pages to zim first then convert the data from zim to other formats as needed?

I think zim is a good option if don't need to be compatible with enterprise api, it uses zstd and pages can be searched. They have a python package: python-libzim

That's great news! Could you also add other Wiktionaries and namespaces later on?

At least the Wikimedia API document says WMCS is exempted from rate limiting, maybe worth a try.

Why can't you call Wiktionary's Parsoid API, isn't API limit exempted for Wikimedia Cloud Services? Another potential alternative is kiwix's zim file but unfortunately they removed "data-mw" attribute which is required data for my use case...

Maybe try enable SQLite WAL mode(PRAGMA journal_mode = WAL;) after adding pages? Or perhaps it's normal that generate a HTML page at first time is slow?

Could you share more about the community HTML snapshots? I'm very interested. I tried to read the enterprise code(wikimedia-enterprise) to find what's wrong with it but it feels very complicated. I'm not sure if it's calling "en.wiktionary.org/w/rest.php/v1/page/page/html" API many times or doing something else.

How can I download 2026-04-01 or 2026-03-01 snapshot files?

2026-04-21 snapshots missing many chunks, "enwiktionary_namespace_0" now has 48 chunks but it has 81 chunks in 2026-04-02. Other Wiktionary like es, fr, ru, de also missing many chunks.

There are more dump files need sync, for example:

• https://dumps.wikimedia.org/shwiktionary/20250801/
• https://dumps.wikimedia.org/mswiktionary/20250801/
• https://dumps.wikimedia.org/sqwikinews/20250801/
• https://dumps.wikimedia.org/nowikinews/20250801/
• https://dumps.wikimedia.org/punjabiwikimedia/20250801/

Can you please confirm?

Thanks! I downloaded the "pages-articles.xml.bz2" file and it seems to be good.

Did trwiktionary dump files also failed? https://dumps.wikimedia.org/trwiktionary/20250801/