Yes, I need to convert that specific task to run in a pod, I forgot about it. Thanks!

I notice the snakemake code downloads the deprecated xml file, they recommend MediaWiki Content File Exports now

@null just a heads-up, I did an import with the content file exports, but they're incomplete: T431723

both are now present

The content file export link is now available, the old location (https://dumps.wikimedia.org/enwiktionary/20260701/) is not.

@null I've added an index file to the snapshots: https://tools-static.wmflabs.org/digero/community_snapshot/index.json

Recently, the tool has been used to produce community HTML snapshots for enwiktionary (see T305407). These are quite storage intensive and there is no cleanup for the temp files, I'll add it.

In T305407#11928176, @null wrote:

I should read the document first and verify the model output more thoroughly....

It seems a bit wasteful to compress and decompress a single page?

In T305407#11928105, @null wrote:

I pasted the code to a few llm and here are the same two suggestions made by these models:

1. buffer at enterprise.py line 55 is not cleared, it grows indefinitely. The fix is clear the buffer at the end of each iteration, or remove the buffer.

for page in pages:
    f.write(encoder.encode(page)  + b"\n")

Here's a first snapshot of the English Wiktionary (~ 5.5 GB, containing Main, Appendix:, Thesaurus: and Reconstruction:):
https://tools-static.wmflabs.org/digero/20260501/community_snapshot/

In T305407#11920355, @null wrote:

I'm a little worried about creating too many files might be problematic for the file system consider more Wiktionaries and namespaces will be added in the future and also delete the root html folder could take some time. Do you think it's feasible to save pages to zim first then convert the data from zim to other formats as needed?

• I notice the snakemake code downloads the deprecated xml file, they recommend MediaWiki Content File Exports now

• zstd has been added to Python 3.14, could use the standard library if you are able to use 3.14. And if dictionary file is used in compress it's also needed in decompress.

The snapshot is being prepared (querying directly the API from WMCS). At the current rate it'll take 1.5–2 days to fetch all the content, which is still acceptable.

In T305407#11897430, @null wrote:

Why can't you call Wiktionary's Parsoid API, isn't API limit exempted for Wikimedia Cloud Services? Another potential alternative is kiwix's zim file but unfortunately they removed "data-mw" attribute which is required data for my use case...

In T305407#11895920, @Ottomata wrote:

This is a lot harder to detect, and the pages will stay outdated forever unless the page itself is updated

FWIW, this is a tricky problem to solve, even inside of WMF.

Here is one practical solution:
https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#Saneitizer_(background_repair_process)

The solution there isn't very detailed, but in summary:

slowly (over several weeks?) iterate from 0 to max page_id and re-fetch the HTML for each page.

This ensures that you will eventually some recentish HTML for every page, even if a page is missing from your dataset, or if a page hasn't received edits in a long time.

The idea is one I've outlined earlier in this ticket (T305407#9223782): we know that the XML dumps are reliable and complete, so they can be used as a starting point to generate the HTML output. A sqlite db is produced from the XML dump and dropped into a local (containerized) MediaWiki instance. Another script then makes API requests against this instance to get the HTML for each page. The advantage is that the instance is not live and will produce consistent data (and there won't be API limit issues).

@null
It looks like you can't. When querying the API from within the Wikimedia cloud, I get:

available snapshots: ['Snapshot[20260506, 46 chunks, 11259.051 MB]']

The 2026-04-30 snapshot has only 2427207 entries (compared to 8741772 in 2026-03-01). No point doing any further analysis.

In T305407#11858637, @null wrote:

2026-04-21 snapshots missing many chunks, "enwiktionary_namespace_0" now has 48 chunks but it has 81 chunks in 2026-04-02. Other Wiktionary like es, fr, ru, de also missing many chunks.

I also notice the duplicated pages are only inside the last few chunks. In 2026-04-02 enwiktionary snapshots, duplicated pages are in the last 3 chunks(78, 79, 80). In 2026-04-02 frwiktionary snapshots, duplicated pages are in the last 2 chunks(37, 38).

Thanks for the update! I've rerun the consistency checks on the 20260301 dumps (XML+HTML). There are still 228 pages entirely missing from the HTML dump, the oldest from 2014. Given the total amount of ~8.7 million pages 228 doesn't seem like much, but I think it's worth investigating why this happens.

If the bug cannot be fixed in the code, at least the dump files could be post-processed to filter out these duplicates prior to publishing. Right now, every user of these files has to do this cleanup themself.

The situation has improved somewhat, but there are still caching issues: if a page A transcludes template B, and B is updated (but not A), then the HTML dumps will still show the old content of B in A until A itself is updated.

@Fenakhay: I think we'll have to solve this ourselves for now. The reconstruction namespace is a lot smaller, it should be doable to produce dumps for them.

confirmed! thanks

Also, it looks like there's another bug where the SDK doesn't handle the request limit situation properly and keeps on retrying the requests in quick succession before seemingly getting rate-limited for the retries:

@creynolds Thanks for investigating. With those 1500 requests I was only able to download 35 of the 72 chunks of the Wiktionary dump. I suspect this can be explained by the SDK making several requests for each chunk. In my case I had it set to 5 MB (the default is 25MB), and each chunk is ~ 200 MB, so that makes (72 * 200) / 5 = 2880 requests. With the default transfer size of 25 MB this would just be ~ 576 requests and still in the free tier. Given that the chunk size is configurable it's strange to have these accounted for by number of requests sent. Maybe it would make more sense to cap by amount of data transferred instead.

How many free chunk requests are there? I'm now getting 429 responses on chunk downloads, and the API dashboard confusingly says: "0 / 0 Chunk requests left" (from a free account outside WMCS).

Confirmed, works now, thanks.

Will these requests work when sent from WMCS?

In T389542#10763021, @HShaikh wrote:

Hey @jberkel

yes accessing these through the Enterprise APIs does require a separate login for now and is a little more involved than how it was on the WMF dumps site. Hopefully the SDKs available on github make it easier to get started.

I was pointed to this ticket on T390839. I'd like to raise some concerns regarding the reduced accessibility and visibility of HTML dumps following the removal of the mirroring.

In T390839#10757036, @xcollazo wrote:

In T390839#10756813, @jberkel wrote:

This also affects the HTML dumps, right? There are none in https://dumps.wikimedia.org/other/enterprise_html/runs/20250401/ or https://dumps.wikimedia.org/other/enterprise_html/runs/20250420/.

The HTML dumps have been removed by the Enterprise team (see https://dumps.wikimedia.org/other/enterprise_html/ for alternatives).

This also affects the HTML dumps, right? There are none in https://dumps.wikimedia.org/other/enterprise_html/runs/20250401/ or https://dumps.wikimedia.org/other/enterprise_html/runs/20250420/.

Done some testing with the latest (20240701) dumps (allowing for some tolerance around the moment of dump generation):

That's good news. I've done some tests, and it's looking much better now. The XML dumps haven't been released yet (due to T365501), so there's no baseline to do more detailed testing.

Latest HTML enwikt dump (20240520) vs XML dump:

It's probably just the new content, with the baseline still being incomplete. I'll check with the XML dumps.

The HTML dumps are pretty much useless until T351712 is fixed.

Can anyone clarify though? It seems that the new sub-tasks are now stuck again.

It probably means the investigation has been "resolved". The main task is now T351712 + subtasks.

I'll add a command to automatically clear the tmp storage, that should help

I've deleted tmp and other unused stuff it's now down to 16GB, is that acceptable?

Could you explain a bit more what this means, please?

Latest enwikt dump is now at 9.6 GB, still some way to go to the 13GB of the 20230701 dump (also incomplete, but still useful as a baseline).

OK. I think it might be worth putting a disclaimer somewhere, perhaps on https://dumps.wikimedia.org/other/enterprise_html/, to warn users that the dumps are incomplete.

@REsquito-WMF thanks! So this means the next dumps will have more data, but will still be incomplete until this other bug is fixed?

@REsquito-WMF not sure if the changes were already in place, but the current enwiktionary NS0 dump is still at 3.5 GB (compared to 13 GB on 20230701).

Is there anything going to be done about this? The enterprise dumps have been in full failure mode for a few months now and are absolutely unusable. I really don't know how an obvious total failure of service can stay in triage hell for such a long time. I understand WMF resources are limited, but then at least let volunteers help out with this. My question about the code generating the dumps above is still unanswered. The transparency/communication on this whole issue has been miserable.

We don't really need to keep all the old dumps around, I've started the deletion of all dump files before 2023. Different files are needed different purposes: for the stats, and for the "wanted entries" on Wiktionary. After generating the dumps, all the data "lives" on Wiktionary, except for the raw data, which is hosted on ~tools.digero/www and shouldn't be deleted. Right now it uses about 1.3G.

@tstarling Thanks for unblocking this! 🙌

Some random guessing: perhaps the error handling code is borked, and it just finishes the dump and closes the file (without erroring the process)? But why then would so many repositories hit errors at the same time? All the 7-20 dumps seem to be affected, maybe some site-wide network/server problems which weren't handled properly?

In T305407#9223814, @Mitar wrote:

so why not simply base the HTML dumps off them?

This does not work. You cannot convert wikitext to HTML in offline mode. You need access to running Mediawiki instance to render macros, templates, etc. So rendering HTML is by definition a dynamic process and something which has not been available otherwise.

In T305407#9223796, @awight wrote:

In T305407#9223782, @jberkel wrote:

I've suggested this previously: the XML dumps have been around for a very long time, and compared to the HTML version, they are very reliable, so why not simply base the HTML dumps off them? Then you could easily compare the counts of the two files, it would also help users who consume both types of data.

When you say "base off them", are you suggesting that the HTML dumps be produced by iterating over the XML dumps and then fetching the HTML content for each row? This would be a cumbersome approach since it introduces an unnecessary extra dependency. The HTML content of the new dumps is not directly derived from the XML dumps, so I don't see much advantage to this approach. I agree that it would be nice to snapshot the wiki content before dumping but this isn't feasible given the way that HTML rendering requires random access to all articles and templates.

I've suggested this previously: the XML dumps have been around for a very long time, and compared to the HTML version, they are very reliable, so why not simply base the HTML dumps off them? Then you could easily compare the counts of the two files, it would also help users who consume both types of data.

Any idea why this would affect primarily non-wikipedia instances? Is the code which generates these dumps available somewhere?

Weirdly, there seems to be less variation in filesizes for Wikipedia dumps:

file sizes from the most recent enwikt HTML dumps (NS0):

Hasn't been fixed yet, data is still missing.

Ok, I hope this can be rolled out quickly, it can't get much worse than the current state

I just checked the latest dumps (2023-07-20), and it's now worse: there are around 2.5 million pages missing from the HTML dump (using the XML dump as a baseline).

Why was this already marked as resolved? New dumps haven't even been published yet, so it's impossible to verify.

Closing this, maybe it'll be useful for future reference. I haven't added documentation to wikitech, not sure where it should go.

I'll see if I can prebuilt the binaries and then just launch the commands without gradle to avoid this issue (so the locks are only held during building, not execution)

In T338770#8922207, @valerio.bozzolan wrote:

Have we tried declaring a different gradle home for each job?

https://github.com/gradle/gradle/issues/8750#issuecomment-605016788

There are ~150 entries missing from the HTML dump (compared to 2200 earlier):

It looks like the situation has improved with the latest dump (20230601, enwikt):

looks like the files have finally been synced to toolforge!

In T335761#8905844, @Hokwelum wrote:

still in progress?

Yes please :-) The rsync is still in progress!

The rsync, which copies the files over to the nfs share accessible to toolforge, is still in progress.

Looks like the data was copied successfully this time! I've downloaded the enwiktionary-NS0 dump and the checksum matches.

In T335761#8886872, @Protsack.stephan wrote:

Looking into potential fixes and trying to figure out the best way to handle this.

It might be the case that we are just serving the checksum of the previous dump.
Meaning: we are grabbing the checksum before the upload has finished.

@ArielGlenn if the API side isn't fixed until the June run would it be possible to ignore the checksums and copy the files regardless? We've been dump-less for 2 months now…