Page MenuHomePhabricator
Create Task
Maniphest T403882

Wikidata N-Triples RDF dumps empty, broken since at least 25 July 2025
Closed, ResolvedPublicBUG REPORT
Actions

Description

The last working truthy dump I could download was wikidata-20250723-truthy-BETA.nt.bz2 - so this issue exists for a while.
By now, all working truthy dumps have been evicted.

See this listing of RDF dumps at https://dumps.wikimedia.org/wikidatawiki/entities/ (retrieved 2025-09-06):

20250727/                                          27-Jul-2025 16:14                   -
20250728/                                          31-Jul-2025 01:07                   -
20250730/                                          30-Jul-2025 23:58                   -
20250801/                                          02-Aug-2025 00:10                   -
20250804/                                          07-Aug-2025 03:45                   -
20250806/                                          06-Aug-2025 23:35                   -
20250808/                                          08-Aug-2025 23:35                   -
20250811/                                          14-Aug-2025 06:33                   -
20250813/                                          13-Aug-2025 23:39                   -
20250815/                                          15-Aug-2025 23:52                   -
20250818/                                          18-Aug-2025 03:16                   -
20250819/                                          22-Aug-2025 02:25                   -
20250820/                                          22-Aug-2025 02:07                   -
20250822/                                          22-Aug-2025 23:37                   -
20250825/                                          28-Aug-2025 03:03                   -
20250827/                                          27-Aug-2025 23:36                   -
20250829/                                          29-Aug-2025 23:35                   -
20250901/                                          04-Sep-2025 02:32                   -
20250903/                                          03-Sep-2025 23:40                   -
20250905/                                          05-Sep-2025 23:34                   -
dcatap.rdf                                         27-Jun-2025 23:38               89497
latest-all.json.bz2                                03-Sep-2025 13:46         97958601385
latest-all.json.gz                                 03-Sep-2025 07:18        148808805062
latest-all.ttl.bz2                                 04-Sep-2025 02:22        119462651605
latest-all.ttl.gz                                  03-Sep-2025 23:29        146691916197
latest-lexemes.json.bz2                            03-Sep-2025 03:51           391420984
latest-lexemes.json.gz                             03-Sep-2025 03:50           531484359
latest-lexemes.ttl.bz2                             05-Sep-2025 23:34           572604384
latest-lexemes.ttl.gz                              05-Sep-2025 23:33           717836061

# Broken files according to size:
latest-truthy.nt.bz2                               25-Jul-2025 16:42                  45
latest-truthy.nt.gz                                25-Jul-2025 16:42                  44
latest-lexemes.nt.bz2                              05-Sep-2025 23:34                  14
latest-lexemes.nt.gz                               05-Sep-2025 23:33               39840
latest-all.nt.bz2                                  04-Sep-2025 02:32                  14
latest-all.nt.gz                                   03-Sep-2025 23:30               39840

There is issue https://phabricator.wikimedia.org/T162346 which might indicate that these dumps are not monitored yet.

Related Objects

Event Timeline

Aklakan created this task.Sep 6 2025, 10:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 6 2025, 10:20 AM
A_smart_kitten added a project: Dumps-Generation.Sep 6 2025, 11:35 AM
Aklakan updated the task description. (Show Details)Sep 7 2025, 5:58 PM
Aklakan updated the task description. (Show Details)
Jakob_WMDE added a project: Wikibase Reuse Team.Sep 8 2025, 12:43 PM
Jakob_WMDE subscribed.Sep 8 2025, 2:15 PM

We found that all the N-triples dumps have been missing since the dump generation infrastructure moved to Airflow because the tool that converts the dumps from .ttl to .nt has been missing. This is being looked into now and should hopefully be resolved soon.

Fnielsen subscribed.Sep 9 2025, 3:13 PM
Mahir256 renamed this task from Some published WikiData RDF Dumps are empty to Wikidata N-Triples RDF dumps empty, broken since at least 25 July 2025.Sep 15 2025, 7:56 PM
Mahir256 merged a task: T404637: Wikidata latest N-Triples dump are broken.
Mahir256 added subscribers: Tpt, ProgVal.
CodeReviewBot added a project: Patch-For-Review.Sep 16 2025, 12:06 PM

jakob opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1677

Fix wikidata truthy dump DAG

CodeReviewBot added a comment.Sep 16 2025, 12:33 PM

jakob opened https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/213

Add missing wikidata RDF dumps dependency "serdi"

CodeReviewBot added a comment.Sep 16 2025, 1:44 PM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1677

Fix wikidata truthy dump DAG

CodeReviewBot added a comment.Sep 16 2025, 2:48 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/213

Add missing wikidata RDF dumps dependency "serdi"

Stashbot added a comment.Sep 16 2025, 2:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-16T14:49:35Z] <dancy@deploy1003> Started scap sync-world: Testing for T403882

Stashbot added a comment.Sep 16 2025, 3:01 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-16T15:01:36Z] <dancy@deploy1003> Finished scap sync-world: Testing for T403882 (duration: 12m 01s)

Maintenance_bot removed a project: Patch-For-Review.Sep 16 2025, 3:31 PM
Jakob_WMDE claimed this task.Sep 17 2025, 10:31 AM
Jakob_WMDE moved this task from Incoming tickets outside WPP to Sprint 53 on the Wikibase Reuse Team board.
Jakob_WMDE edited projects, added Wikibase Reuse Team (Sprint 53); removed Wikibase Reuse Team.
Jakob_WMDE moved this task from To Do to Product Verification on the Wikibase Reuse Team (Sprint 53) board.
MrSampson subscribed.Sep 19 2025, 9:58 AM

When will the fixed dumps be available on the server?

Jakob_WMDE added a comment.Sep 19 2025, 10:42 AM
In T403882#11196732, @MrSampson wrote:

When will the fixed dumps be available on the server?

The latest truthy dump is already available here: https://dumps.wikimedia.org/other/wikibase/wikidatawiki/20250916/

The Lexeme N-Triples dump should be available tomorrow, and the full N-Triples dump will be available mid next week once the dump generation job starting on Monday is completed.

MrSampson added a comment.Sep 26 2025, 9:05 AM

Thanks! I see that something happened on the 24th, but it looks like the full N-Triples dump didn't finish.

Jakob_WMDE added a comment.Sep 29 2025, 8:29 AM

They're there now: https://dumps.wikimedia.org/wikidatawiki/entities/20250922/

They got successfully generated on the 22nd, but something went wrong in the process of copying them over to the dumps server. Everything should be running smoothly again from now on.

Jakob_WMDE moved this task from Product Verification to Done on the Wikibase Reuse Team (Sprint 53) board.Sep 29 2025, 9:16 AM
Jakob_WMDE mentioned this in T406044: Re-enable `set -e` in Wikibase entity dump scripts.Sep 30 2025, 3:21 PM
WMDE-leszek closed this task as Resolved.Oct 7 2025, 8:19 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits