Weirdly, there seems to be less variation in filesizes for Wikipedia dumps:

wikipedia

Any idea why this would affect primarily non-wikipedia instances? Is the code which generates these dumps available somewhere?

More suspicious file sizes:
19G /mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/enterprise_html/runs/20230720/dewiki-NS0-20230720-ENTERPRISE-HTML.json.tar.gz
32G /mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/enterprise_html/runs/20230920/dewiki-NS0-20230920-ENTERPRISE-HTML.json.tar.gz

The 2023-07-20 file seems to have been 700k or so rows short. The 2023-09-20 file seems to be truncated, I keep running into a malformed stream error while processing.

Some random guessing: perhaps the error handling code is borked, and it just finishes the dump and closes the file (without erroring the process)? But why then would so many repositories hit errors at the same time? All the 7-20 dumps seem to be affected, maybe some site-wide network/server problems which weren't handled properly?

Is there anything going to be done about this? The enterprise dumps have been in full failure mode for a few months now and are absolutely unusable. I really don't know how an obvious total failure of service can stay in triage hell for such a long time. I understand WMF resources are limited, but then at least let volunteers help out with this. My question about the code generating the dumps above is still unanswered. The transparency/communication on this whole issue has been miserable.

If there's no will to maintain usable dumps from the WMF side the community will have to build alternative systems.

Hello @jberkel! Thanks for your feedback. We understand the frustration that can arise from delayed responses, and please know that your concerns have not gone unnoticed. Our team is fully aware of the impact this delay has had, and we are committed to rectifying the situation as promptly as possible.

While we cannot guarantee an immediate resolution, I want to assure you that the matter is currently at the top of our agenda. We have marked it as an 'expedited' topic to be tackled with the utmost priority. We appreciate your understanding and patience as we work on the ticket.

Thank you for your continued interest in using these database dumps.

Hello,

The team continues to work on this issue, we have detected autoscaling issues that we have addressed and continue to dig deeper into other potential causes, after a root cause analysis.

We will post more updates as we go along with the research.

Thank you

Hello,

We have made a change in the last 2 weeks and are analysing the results in order to figure out if there's less discrepancies, if you find any please let us know.

We also continue to look into improvements of our snapshot process.

Thank you

@REsquito-WMF not sure if the changes were already in place, but the current enwiktionary NS0 dump is still at 3.5 GB (compared to 13 GB on 20230701).

@jberkel Happy new year.

We returned to work and we made a change in a configuration, it should be updated tomorrow.

Thank you.

Hi,

The change we made had a great impact:

Before: 1290827 total pages

Current: 5812947 total pages

Expected: 7921988 total pages

the missing pages are related to a bug we are tracking here: https://phabricator.wikimedia.org/T351712

In that sense, we are going to be tracking the rest of the work there

@REsquito-WMF thanks! So this means the next dumps will have more data, but will still be incomplete until this other bug is fixed?

Correct @jberkel

OK. I think it might be worth putting a disclaimer somewhere, perhaps on https://dumps.wikimedia.org/other/enterprise_html/, to warn users that the dumps are incomplete.

Latest enwikt dump is now at 9.6 GB, still some way to go to the 13GB of the 20230701 dump (also incomplete, but still useful as a baseline).

I'm wondering what's the deal with the Closed as Unknown Status here, haven't seen this before and I'm unsure about its meaning.

So this has been resolved? 13GB of the 20230701 dump was so large why? Because it contained duplicate documents? Otherwise it is unclear why it is just 9.6 GB now.

It probably means the investigation has been "resolved". The main task is now T351712 + subtasks.

Can anyone clarify though? It seems that the new sub-tasks are now stuck again.