Maniphest T399119

bzip2 wikidata dump incomplete
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Melderick
	Jul 9 2025, 7:07 PM

Tags

Referenced Files

None

Subscribers

Lucas_Werkmeister_WMDE

Description

Hello,

The bzip2 wikidata dump for this week is here : https://dumps.wikimedia.org/wikidatawiki/entities/20250707/
Comparing its size with previous week bzip2 dump from here : https://dumps.wikimedia.org/wikidatawiki/entities/20250630/
We can see that the size is way too short :

wikidata-20250707-all.json.bz2 09-Jul-2025 04:26 4913992760
wikidata-20250630-all.json.bz2 02-Jul-2025 08:28 96958600959

Whereas the gzip dumps are about the same size between both weeks.
The only conclusion is that this week bzip2 is incomplete.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Use 'set -o pipefail' instead of 'set -e' in wikibase scripts	operations/dumps	master	+7 -6

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Increase the memory for wikibase->wikidata dumps	repos/data-engineering/airflow-dags!1582	btullis	bump_wikibase_dump_resources	main

Customize query in GitLab

Related Objects

Mentioned In: T400383: Recent wikibase RDF dumps on Airflow have failed
T398756: No wikidata dumps this week (20250630)
Mentioned Here: T399077: wikidata-20250707-all.json.gz is corrupted

Event Timeline

Melderick created this task.Jul 9 2025, 7:07 PM

Melderick mentioned this in T398756: No wikidata dumps this week (20250630).

Seems like a similar issue to T399077: wikidata-20250707-all.json.gz is corrupted.

karapayneWMDE added a project: Wikibase Reuse Team.Jul 10 2025, 8:21 AM

As @xcollazo mentioned, these issues are indeed the same. I'll post progress reports on T399077. Note that I have backfilled the data to https://dumps.wikimedia.org/other/wikibase/wikidatawiki/20250707/ so you should be able to use these files while we address the root cause.

brouberol changed the task status from Open to In Progress.Jul 10 2025, 10:46 AM

brouberol claimed this task.

brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.

brouberol added a project: Data-Platform-SRE (2025.07.05 - 2025.07.25).

Silvan_WMDE merged a task: T399544: Incomplete Wikidata latest-all.json.bz2 dump (July 2025).Jul 15 2025, 10:14 AM

Silvan_WMDE added a subscriber: Lucas_Werkmeister_WMDE.

Change #1172682 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/dumps@master] Use 'set -o pipefail' instead of 'set -e' in wikibase scripts

https://gerrit.wikimedia.org/r/1172682

Change #1172682 merged by Btullis:

[operations/dumps@master] Use 'set -o pipefail' instead of 'set -e' in wikibase scripts

https://gerrit.wikimedia.org/r/1172682

Maintenance_bot removed a project: Patch-For-Review.Jul 25 2025, 5:30 PM

BTullis mentioned this in T400383: Recent wikibase RDF dumps on Airflow have failed.Jul 28 2025, 11:16 AM

I think that this should now be fixed. I'll move this to waiting, while we monitor the latest dumps.

BTullis triaged this task as High priority.Jul 28 2025, 11:20 AM

BTullis edited projects, added Data-Platform-SRE (2025.07.26 - 2025.08.15); removed Data-Platform-SRE (2025.07.05 - 2025.07.25).Jul 28 2025, 4:45 PM

BTullis moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.

BTullis edited projects, added Data-Platform-SRE (2025.08.16 - 2025.09.05); removed Data-Platform-SRE (2025.07.26 - 2025.08.15).Aug 20 2025, 10:31 AM

BTullis moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.

BTullis closed this task as Resolved.Aug 20 2025, 10:43 AM

BTullis moved this task from Blocked/Waiting to Done on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.Sep 5 2025, 12:45 PM

Gehel added a project: Essential-Work.Sep 5 2025, 3:05 PM