Weirdly, there seems to be less variation in filesizes for Wikipedia dumps:

wikipedia

Any idea why this would affect primarily non-wikipedia instances? Is the code which generates these dumps available somewhere?

More suspicious file sizes:
19G /mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/enterprise_html/runs/20230720/dewiki-NS0-20230720-ENTERPRISE-HTML.json.tar.gz
32G /mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/enterprise_html/runs/20230920/dewiki-NS0-20230920-ENTERPRISE-HTML.json.tar.gz

The 2023-07-20 file seems to have been 700k or so rows short. The 2023-09-20 file seems to be truncated, I keep running into a malformed stream error while processing.

Some random guessing: perhaps the error handling code is borked, and it just finishes the dump and closes the file (without erroring the process)? But why then would so many repositories hit errors at the same time? All the 7-20 dumps seem to be affected, maybe some site-wide network/server problems which weren't handled properly?

Is there anything going to be done about this? The enterprise dumps have been in full failure mode for a few months now and are absolutely unusable. I really don't know how an obvious total failure of service can stay in triage hell for such a long time. I understand WMF resources are limited, but then at least let volunteers help out with this. My question about the code generating the dumps above is still unanswered. The transparency/communication on this whole issue has been miserable.

If there's no will to maintain usable dumps from the WMF side the community will have to build alternative systems.

Hello @jberkel! Thanks for your feedback. We understand the frustration that can arise from delayed responses, and please know that your concerns have not gone unnoticed. Our team is fully aware of the impact this delay has had, and we are committed to rectifying the situation as promptly as possible.

While we cannot guarantee an immediate resolution, I want to assure you that the matter is currently at the top of our agenda. We have marked it as an 'expedited' topic to be tackled with the utmost priority. We appreciate your understanding and patience as we work on the ticket.

Thank you for your continued interest in using these database dumps.

Hello,

The team continues to work on this issue, we have detected autoscaling issues that we have addressed and continue to dig deeper into other potential causes, after a root cause analysis.

We will post more updates as we go along with the research.

Thank you

Hello,

We have made a change in the last 2 weeks and are analysing the results in order to figure out if there's less discrepancies, if you find any please let us know.

We also continue to look into improvements of our snapshot process.

Thank you

@REsquito-WMF not sure if the changes were already in place, but the current enwiktionary NS0 dump is still at 3.5 GB (compared to 13 GB on 20230701).

@jberkel Happy new year.

We returned to work and we made a change in a configuration, it should be updated tomorrow.

Thank you.

Hi,

The change we made had a great impact:

Before: 1290827 total pages

Current: 5812947 total pages

Expected: 7921988 total pages

the missing pages are related to a bug we are tracking here: https://phabricator.wikimedia.org/T351712

In that sense, we are going to be tracking the rest of the work there

@REsquito-WMF thanks! So this means the next dumps will have more data, but will still be incomplete until this other bug is fixed?

Correct @jberkel

OK. I think it might be worth putting a disclaimer somewhere, perhaps on https://dumps.wikimedia.org/other/enterprise_html/, to warn users that the dumps are incomplete.

Latest enwikt dump is now at 9.6 GB, still some way to go to the 13GB of the 20230701 dump (also incomplete, but still useful as a baseline).

I'm wondering what's the deal with the Closed as Unknown Status here, haven't seen this before and I'm unsure about its meaning.

So this has been resolved? 13GB of the 20230701 dump was so large why? Because it contained duplicate documents? Otherwise it is unclear why it is just 9.6 GB now.

It probably means the investigation has been "resolved". The main task is now T351712 + subtasks.

Can anyone clarify though? It seems that the new sub-tasks are now stuck again.

This last change seems reasonable? Size is increasing now?

It's probably just the new content, with the baseline still being incomplete. I'll check with the XML dumps.

Latest HTML enwikt dump (20240520) vs XML dump:

• 1.883.645 pages missing completely from the HTML dump
• 4843 pages out of date (present in HTML dump but not matching the XML revision id)

Histogram of the time skew, in months:

[  0.0,   2.0) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 2429
[  2.0,   4.0) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇ 954
[  4.0,   6.0) ┤▇▇▇ 224
[  6.0,   8.0) ┤▇▇ 131
[  8.0,  10.0) ┤▇▇▇ 224
[ 10.0,  12.0) ┤▇ 71
[ 12.0,  14.0) ┤ 7
[ 14.0,  16.0) ┤ 17
[ 16.0,  18.0) ┤▇ 86
[ 18.0,  20.0) ┤ 32
[ 20.0,  22.0) ┤▇ 46
[ 22.0,  24.0) ┤▇ 39

Basically this means almost 25% of the data is missing or outdated, and the outdated revisions often contain vandalism. I'm not sure why these dumps are even produced, it's a total waste of time, bandwidth, and CPU cycles.

Hi All,

Quick update, we have finished the work.

Latest enwiktionary file is now 15Gb.

Wait for the next public dump coming, or get it from enterprise.

We did a few tests and duplicates, deleted and missed are all below 1% at the time of testing.

Thanks

That's good news. I've done some tests, and it's looking much better now. The XML dumps haven't been released yet (due to T365501), so there's no baseline to do more detailed testing.

Done some testing with the latest (20240701) dumps (allowing for some tolerance around the moment of dump generation):

• 57 pages missing from the HTML dump
• 176 pages out-of-date (present, but not matching the XML revision id)

That means an error rate of ~ 0.003% which seems acceptable :)

It might still be worth investigating why some pages are completely missing, I've added them for reference:

In some cases it looks like they had been redirects that were converted into content pages.

Thanks @jberkel

Do you want to open a bug with this info?

If not we'll open it and have a look.

Thanks