Page MenuHomePhabricator
Create Task
Maniphest T227207

Wikibase JSON output (dumps, Special:EntityData) lacks qualifier hashes
Closed, ResolvedPublic
Actions

Description

Example from Q31 (first excerpt from the 20190617 dump, second excerpt from the 20190701):

"qualifiers":{"P585":[{"snaktype":"value","property":"P585","hash":"489481dca5f5660d9fe15875b4dc94a57389e4bd","datavalue":{"value":{"time":"+2014-01-01T00:00:00Z","timezone":0,"before":0,"after":0,"precision":11,"calendarmodel":"http:\/\/www.wikidata.org\/entity\/Q1985727"},"type":"time"},"datatype":"time"}]}
"qualifiers":{"P585":[{"snaktype":"value","property":"P585","datavalue":{"value":{"time":"+2014-01-01T00:00:00Z","timezone":0,"before":0,"after":0,"precision":11,"calendarmodel":"http:\/\/www.wikidata.org\/entity\/Q1985727"},"type":"time"},"datatype":"time"}]}

Details

SubjectRepoBranchLines +/-
DumpJsonTest: Use actual CompactEntitySerializermediawiki/extensions/Wikibasemaster+3 -8
Fix missing qualifier hashes in JSON outputmediawiki/extensions/Wikibasewmf/1.34.0-wmf.11+4 -1
Fix missing qualifier hashes in JSON outputmediawiki/extensions/Wikibasemaster+4 -1
Customize query in gerrit

Related Objects

Event Timeline

hoo created this task.Jul 3 2019, 4:30 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptJul 3 2019, 4:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Lydia_Pintscher triaged this task as Unbreak Now! priority.Jul 3 2019, 4:31 PM
Lydia_Pintscher added a project: Wikidata-Campsite.
Lydia_Pintscher subscribed.

Changing this to unbreak now to get it investigated. If we made this change it should have been announced as a breaking change.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptJul 3 2019, 4:31 PM
ArielGlenn subscribed.Jul 3 2019, 4:33 PM

I've moved the 20190701 json files to <samefilename>.not in that directory on the various hosts. Mirrors should reflect this change within 24 hours. Leaving the broken links so that folks with scripts get failure instead of reprocessing old files.

@hoo would you be wiling to send out a note to xmldatadumps-l and the wikidata mailing list explaining what's up?

ArielGlenn added a comment.Jul 3 2019, 4:34 PM

Also note the only window to get a fix in for the next run is tonight's late night UTC SWAT, tomorrow is no deploys (US holdiay, Fri is no deploys, Monday this job will have already started for the week.

TheDatum subscribed.Jul 3 2019, 4:42 PM
hoo added a comment.Jul 3 2019, 4:45 PM

This was caused by a1c5b7ed3eccef653a7041f42d65a09ff83bf6da

gerritbot added a comment.Jul 3 2019, 4:55 PM

Change 520496 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[mediawiki/extensions/Wikibase@master] Fix missing qualifier hashes in JSON output

https://gerrit.wikimedia.org/r/520496

gerritbot added a project: Patch-For-Review.Jul 3 2019, 4:55 PM
gerritbot added a comment.Jul 3 2019, 5:12 PM

Change 520501 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[mediawiki/extensions/Wikibase@master] DumpJsonTest: Use actual CompactEntitySerializer

https://gerrit.wikimedia.org/r/520501

Lea_Lacroix_WMDE subscribed.Jul 3 2019, 5:40 PM

Announcement made onwiki, to wikidata, wikidata-tech and xmldumps MLs, also on Telegram.

gerritbot added a comment.Jul 3 2019, 6:13 PM

Change 520496 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Fix missing qualifier hashes in JSON output

https://gerrit.wikimedia.org/r/520496

gerritbot added a comment.Jul 3 2019, 6:14 PM

Change 520509 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[mediawiki/extensions/Wikibase@wmf/1.34.0-wmf.11] Fix missing qualifier hashes in JSON output

https://gerrit.wikimedia.org/r/520509

gerritbot added a comment.Jul 3 2019, 6:46 PM

Change 520509 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.34.0-wmf.11] Fix missing qualifier hashes in JSON output

https://gerrit.wikimedia.org/r/520509

Stashbot added a comment.Jul 3 2019, 6:58 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-03T18:58:30Z] <jforrester@deploy1001> Synchronized php-1.34.0-wmf.11/extensions/Wikibase/data-access/src/GenericServices.php: T227207 Fix missing qualifier hashes in JSON output (duration: 00m 50s)

Jdforrester-WMF lowered the priority of this task from Unbreak Now! to High.Jul 3 2019, 6:59 PM
Jdforrester-WMF subscribed.

Deployed everywhere. Next dumps process will run on Monday 2019-07-08.

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.11; 2019-06-26).Jul 3 2019, 7:00 PM
Envlh subscribed.Jul 3 2019, 8:24 PM
In T227207#5304615, @Jdforrester-WMF wrote:

Deployed everywhere. Next dumps process will run on Monday 2019-07-08.

As this is the second week in a row that this dump is not generated (see T227207), is it possible to anticipate the next process?

hoo edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.Jul 3 2019, 10:24 PM
hoo moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Only thing left to do: Merge https://gerrit.wikimedia.org/r/520501 (makes the integration test cover the actual serialization code).

gerritbot added a comment.Jul 4 2019, 11:47 AM

Change 520501 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] DumpJsonTest: Use actual CompactEntitySerializer

https://gerrit.wikimedia.org/r/520501

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.13; 2019-07-09); removed MW-1.34-notes (1.34.0-wmf.11; 2019-06-26).Jul 4 2019, 12:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 4 2019, 12:11 PM
Stashbot added a comment.Jul 4 2019, 12:21 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-04T12:20:59Z] <hoo> Started a Wikidata JSON dump run (sudo -b -u dumpsgen /usr/local/bin/dumpwikidatajson.sh) on snapshot1008 (T227207)

hoo mentioned this in T227276: Check usages/ purpose of GenericServices::getCompactBaseDataModelSerializerFactory.Jul 4 2019, 12:55 PM
Lea_Lacroix_WMDE added a comment.Jul 4 2019, 1:08 PM

@Envlh We're working on generating one for this week. It should appear on Saturday.

hoo mentioned this in T227283: Create integration test for dump scripts using the scripts via CLI.Jul 4 2019, 1:32 PM
hoo moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

This is all done.

Follow up tasks: T227276 (nicer fix), T227283 (more integration tests).

• alaa_wmde subscribed.Jul 5 2019, 12:39 PM

Wanted to check this, but I'm getting 404 on latest json dumps https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.bz2 and https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz.. '20190704' folder is also empty (that means that we create an empty folder for latest. and move latest into it when generating the next dump?)

ArielGlenn added a comment.Jul 5 2019, 1:20 PM

The ones for this week will have a new date, the date on which they were started. The 'latest' links will point to it only when the dumps are complete and have passed some basic sanity checks in the script. No files are available to be monitored yet because they are still being written, and we don't rsync half-done files from our internal servers repeatedly out to the web server for performance reasons.

It might be nice to have the equivalent of the xml/sql dumps index.html file which shows some sort of progress, at least the amounts written. But the entity dumps are not set up by nature for that. Perhaps the wikidata folks would like to comment on this or other options, although extended discussion should probably move to a new task.

• alaa_wmde added a comment.Jul 5 2019, 1:23 PM

The ones for this week will have a new date, the date on which they were started. The 'latest' links will point to it only when the dumps are complete and have passed some basic sanity checks in the script. No files are available to be monitored yet because they are still being written, and we don't rsync half-done files from our internal servers repeatedly out to the web server for performance reasons.

@ArielGlenn thanks for the explanation. Then I'll wait for next dump to be ready to verify the fix.

It might be nice to have the equivalent of the xml/sql dumps index.html file which shows some sort of progress, at least the amounts written. But the entity dumps are not set up by nature for that. Perhaps the wikidata folks would like to comment on this or other options, although extended discussion should probably move to a new task.

Yeap that need to be discussed separately.. like the idea of showing some progress, noted and will try to capture it on a different task.

Envlh added a comment.Jul 7 2019, 8:52 AM

Thank you all! Using Wikidata Toolkit, I was able to load the dump generated on 2019-07-04 :)

Charlie_WMDE closed this task as Resolved.Jul 8 2019, 8:27 AM
Charlie_WMDE claimed this task.
Charlie_WMDE moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Charlie_WMDE subscribed.

\o/

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL