Page MenuHomePhabricator

Do not serialize empty containers (descriptions/aliases/sitelinks) as empty array []
Closed, ResolvedPublic5 Story Points

Description

Empty containers like the ones for descriptions/aliases should be serialized as empty object "{}" and not empty array "[]" in order to ease the deserialization process.

Examples:

We have checked other places and the API and dumps already use {}, this ticket only relates to Special:EntityData
(All of this is relevant for the announcement)

The change should be announced (as a breaking change) per https://www.wikidata.org/wiki/Wikidata:Stable_Interface_Policy
and thus we want to wrap the change in a config var so that we can choose when to announce and choose when to make the change.

Event Timeline

Tpt created this task.Jun 17 2016, 7:10 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJun 17 2016, 7:10 PM
Tpt updated the task description. (Show Details)Jun 17 2016, 8:08 PM
Ricordisamoa added a subscriber: Ricordisamoa.
Nikki added a subscriber: Nikki.Jul 24 2017, 11:32 AM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 24 2017, 11:32 AM

It looks like this affects the live JSON returned by Special:EntityData, but not the JSON found in the dumps – the dumps have "description":{}.

hoo added a subscriber: hoo.Apr 10 2018, 2:43 PM
NoInkling added a subscriber: NoInkling.

Here's a new example that's currently missing everything except claims, since the original was updated: https://www.wikidata.org/wiki/Special:EntityData/Q61519072.json

As I noted elsewhere, it might not be such a big deal for JavaScript clients since JS syntax/APIs that work on plain objects tend to also work on arrays, but in other languages this obviously isn't the case. Since it's kinda rare for an entity not to have a single label/description/alias/claim/sitelink, and this behaviour doesn't seem to be documented anywhere, it's not very likely that the person writing the client will think to account for this until it's too late.

Lydia_Pintscher moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.

We should probably announce this change, as clients expecting the old / wrong thing could break.

From: https://www.wikidata.org/wiki/Wikidata:Stable_Interface_Policy

Breaking change: a change to an API or data format that violates guarantees given or widely assumed before. Breaking changes include removal of API functions, parameters, or data fields and changes to the interpretation or format of parameters or data fields.

Addshore triaged this task as Low priority.Feb 12 2019, 2:01 PM
Addshore updated the task description. (Show Details)
Addshore updated the task description. (Show Details)Feb 26 2019, 3:50 PM
Addshore added a subscriber: Lea_Lacroix_WMDE.
Addshore updated the task description. (Show Details)Feb 26 2019, 3:52 PM
Addshore set the point value for this task to 5.
Addshore updated the task description. (Show Details)Feb 26 2019, 4:10 PM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptFeb 26 2019, 4:24 PM

Change 493096 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Add metadata to Special:EntityData output when config is set

https://gerrit.wikimedia.org/r/493096

This is one of the weirdest cases I've done. PHP doesn't distinguish between array and associative arrays when they are empty and there is no way to send this information around in php while in python they are completely different objects (lists and dictionaries). The reason that wbgetentity API module and the dumpers return {} instead of [] is because these containers are not passed around empty, they contain hidden elements that you can see when you change the formatting to xml (and for whatever reason, mediawiki needs to support xml output forever and beyond) but when they are turned to json, the hidden elements get stripped away but json knows it's a associative array.

What I did was that I just told ResultBuilder for Special:EntityData to add metadata (while they get removed in final output anyway). That would fix the issue we are dealing now. It's behind a config variable.

Change 493096 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add metadata to Special:EntityData output when config is set

https://gerrit.wikimedia.org/r/493096

Change 499999 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Add tmpSerializeEmptyListsAsObjects Wikibase repo config

https://gerrit.wikimedia.org/r/499999

Change 499999 merged by jenkins-bot:
[operations/mediawiki-config@master] Add tmpSerializeEmptyListsAsObjects Wikibase repo config

https://gerrit.wikimedia.org/r/499999

Mentioned in SAL (#wikimedia-operations) [2019-04-01T18:10:02Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:499999|Add tmpSerializeEmptyListsAsObjects Wikibase repo config (T138104)]] (duration: 00m 54s)

Hm, the config change should be deployed on Beta, but I’m still getting empty lists instead of empty objects there.

Hm, the config change should be deployed on Beta, but I’m still getting empty lists instead of empty objects there.

I had similar issue, make sure you cleared the cache, try a new Item you haven't seen in your life before.

I did, it was a different problem – the beta-scap-eqiad job hadn’t finished yet (I had only waited for beta-mediawiki-config-update-eqiad). I get empty objects now \o/

Here's the current process running, according to our stable interface policy:
✅ April 2nd - Announcement
✅ Test system available (min. 2 weeks before deployment)
⬜ April 30th - Deployment (4 weeks after announcement)

WMDE-leszek added a subscriber: WMDE-leszek.

Moving to "Stalled" column as I reckon it is a waiting time now

Hjfocs added a subscriber: Hjfocs.Apr 2 2019, 10:54 AM
Nirmos added a subscriber: Nirmos.Apr 8 2019, 10:07 PM

Change 502826 had a related patch set uploaded (by Pablo Grass (WMDE); owner: Jakob):
[wikibase/termbox@master] Replace empty arrays with empty objects in EntityInitializer

https://gerrit.wikimedia.org/r/502826

Change 502826 had a related patch set uploaded (by Jakob; owner: Jakob):
[wikibase/termbox@master] Replace empty arrays with empty objects in EntityInitializer

https://gerrit.wikimedia.org/r/502826

Change 502826 merged by jenkins-bot:
[wikibase/termbox@master] Replace empty arrays with empty objects in EntityInitializer

https://gerrit.wikimedia.org/r/502826

This is due to be enabled tomorrow (30.4.). Is anyone monitoring it?

I just realized this affects Commons as well (example entity data with empty list as descriptions). Does the stable interface policy apply there?

Change 507030 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Serialize empty lists as objects on Test Wikidata

https://gerrit.wikimedia.org/r/507030

Change 507031 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Serialize empty lists as objects on Wikidata

https://gerrit.wikimedia.org/r/507031

Change 507032 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Serialize empty lists as objects on Commons

https://gerrit.wikimedia.org/r/507032

The current stable interface policy seems to be specific to Wikidata only –

This Stable Interface Policy defines which guarantees are and are not given by the Wikidata development team regarding the stability of data formats and APIs provided by Wikibase as deployed on www.wikidata.org.

(emphasis added) – so for now let’s just deploy this bugfix to Commons at the same time, assuming that if anyone is already using Commons’ Special:EntityData, they’ll do it with the same code that they also use for Wikidata’s Special:EntityData, and which is prepared for the fixed output format. The future of the stable interface policy with regard to Commons will be discussed elsewhere, I am assured.

Change 507034 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Serialize empty lists as objects on Test Commons

https://gerrit.wikimedia.org/r/507034

All four config changes scheduled for tomorrow’s EU SWAT (13:00 CEST).

On second thought, I’ve moved the two test wiki changes to today’s Morning SWAT, so they can get a bit of testing in overnight. (I don’t think our announcement covered Test Wikidata.)

Change 507030 merged by jenkins-bot:
[operations/mediawiki-config@master] Serialize empty lists as objects on Test Wikidata

https://gerrit.wikimedia.org/r/507030

Change 507034 merged by jenkins-bot:
[operations/mediawiki-config@master] Serialize empty lists as objects on Test Commons

https://gerrit.wikimedia.org/r/507034

Mentioned in SAL (#wikimedia-operations) [2019-04-29T18:30:15Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Serialize empty lists as objects on Test Wikidata (T138104) (duration: 00m 53s)

Mentioned in SAL (#wikimedia-operations) [2019-04-29T18:37:16Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Serialize empty lists as objects on Test Commons (T138104) (duration: 00m 54s)

Change 507031 merged by jenkins-bot:
[operations/mediawiki-config@master] Serialize empty lists as objects on Wikidata

https://gerrit.wikimedia.org/r/507031

Mentioned in SAL (#wikimedia-operations) [2019-04-30T11:12:00Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:507031|Serialize empty lists as objects on Wikidata (T138104)]] (duration: 00m 55s)

Change 507032 merged by jenkins-bot:
[operations/mediawiki-config@master] Serialize empty lists as objects on Commons

https://gerrit.wikimedia.org/r/507032

Mentioned in SAL (#wikimedia-operations) [2019-04-30T11:15:48Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:507032|Serialize empty lists as objects on Commons (T138104)]] (duration: 00m 54s)

Should be done now (though I noticed a related issue in MediaInfo…)

Lucas_Werkmeister_WMDE closed this task as Resolved.Apr 30 2019, 5:08 PM

Since @Lea_Lacroix_WMDE sent another announcement to wikidata{,-tech}-l that this is now deployed, I’m going to call that sufficiently verified (and announced) and close this task :)