Page MenuHomePhabricator

Reconsider wikidata support in the REST API
Closed, ResolvedPublic

Description

Basically none of the current REST API end points in https://www.wikidata.org/api/rest_v1/ are actually useful currently. HTML contains a <pre> wrapping a large JSON blob. Nobody is using VisualEditor to edit this JSON blob. None of the derived content types like mobile sections or the page summary appear obviously useful. For the most part, wikidata information like item descriptions is consumed through individual project's REST APIs (such as the summary end point), rather than the wikidata REST API itself.

We also currently process all updates for these end points the same way as for any other wiki. This means a lot of unnecessary work and storage use.

So, lets revisit how we support WikiData in the REST API:

  • If we find no actual use cases for the current WikiData REST API, then I am proposing to remove it altogether, and drop the associated storage. This saves resources, and avoids exposing an API that isn't useful or meaningfully maintained.
    • We know that there are currently no uses for mobile sections or page summaries.
  • Consider which use cases the REST API could actually help with for WikiData, and set up a custom project config, similar to what we already do for wiktionaries.

Details

Related Gerrit Patches:
operations/puppet : productionRESTBase: Resurrect WikiData domains
mediawiki/services/change-propagation/deploy : masterConfig: Enable wikidata updates for all namespaces except 0 and 120
operations/puppet : productionRESTBase: Remove www.wikidata.org as well
mediawiki/services/change-propagation/deploy : masterIgnore all Wikidata domains for updates
operations/puppet : productionRESTBase: Remove Wikidata from the list of domains
mediawiki/services/change-propagation/deploy : masterIgnore updates from Wikidata

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2016, 8:19 PM
GWicke triaged this task as Medium priority.Oct 25 2016, 8:19 PM

According to sampled request log, the only requests to REST API on wikidata are made by ChangeProp and Mobile-Content-Service (see https://logstash.wikimedia.org/goto/155383470f2b3307f82b91b34ad29889 ) which suggests there's no real users of this API. SO, +1 for dropping it from me

+1 for dropping from me as well.

Pchelolo moved this task from next to doing on the Services board.EditedOct 26 2016, 10:11 PM
Pchelolo edited projects, added Services (doing); removed Services (next).

I think we've got general agreement on this. Some code changes needs to be done here:

I'm ok with it too, but would love some input from Wikidata folks before we proceed.

+1 for dropping it, I am not aware of any use case. I'd like to hear what @Lydia_Pintscher thinks, though.

Change 319551 had a related patch set uploaded (by Mobrovac):
Ignore updates from Wikidata

https://gerrit.wikimedia.org/r/319551

Change 319551 merged by Mobrovac:
Ignore updates from Wikidata

https://gerrit.wikimedia.org/r/319551

Change 319552 had a related patch set uploaded (by Mobrovac):
RESTBase: Remove Wikidata from the list of domains

https://gerrit.wikimedia.org/r/319552

Change-prop is not receiving WD updates any more. The only piece left is to remove the domain from RESTBase.

Change 319552 merged by Alexandros Kosiaris:
RESTBase: Remove Wikidata from the list of domains

https://gerrit.wikimedia.org/r/319552

Change 319560 had a related patch set uploaded (by Mobrovac):
RESTBase: Remove www.wikidata.org as well

https://gerrit.wikimedia.org/r/319560

Change 319567 had a related patch set uploaded (by Mobrovac):
Ignore all Wikidata domains for updates

https://gerrit.wikimedia.org/r/319567

Change 319567 merged by Mobrovac:
Ignore all Wikidata domains for updates

https://gerrit.wikimedia.org/r/319567

Change 319560 merged by Alexandros Kosiaris:
RESTBase: Remove www.wikidata.org as well

https://gerrit.wikimedia.org/r/319560

Does this also affect wikitext pages on Wikidata? Because that seems like not what we want.

Mentioned in SAL (#wikimedia-operations) [2016-11-03T14:00:05Z] <mobrovac> restbase rolling restart for T149114

Does this also affect wikitext pages on Wikidata?

It affects the whole domain.

Because that seems like not what we want.

Mind elaborating on that?

FYI, test.wd.org and www.wd.org have been removed and don't have the REST API any more.

We do have pages on Wikidata that should be able to use the VisualEditor like https://www.wikidata.org/wiki/Help:Contents
Daniel wasn't sure if this is prevented by the change.

Change 319645 had a related patch set uploaded (by Mobrovac):
RESTBase: Resurrect WikiData domains

https://gerrit.wikimedia.org/r/319645

Change 319645 merged by BBlack:
RESTBase: Resurrect WikiData domains

https://gerrit.wikimedia.org/r/319645

Change 319648 had a related patch set uploaded (by Ppchelko):
Config: Enable wikidata updates for all namespaces except 0 and 120

https://gerrit.wikimedia.org/r/319648

Change 319648 merged by Mobrovac:
Config: Enable wikidata updates for all namespaces except 0 and 120

https://gerrit.wikimedia.org/r/319648

Mentioned in SAL (#wikimedia-operations) [2016-11-03T19:23:48Z] <mobrovac> restbase restarting to re-include wikidata domains for T149114

mobrovac closed this task as Resolved.Nov 3 2016, 7:35 PM
mobrovac claimed this task.

The REST API is back on www.wikidata.org. I don' think there is more work to be done here, so closing.

Yurik added a subscriber: Yurik.Nov 29 2016, 4:57 PM

All graphs on Wikidata are now dead. Reported by users.
https://www.wikidata.org/wiki/Template:Graph:Chart

@Yurik are you implying that was using RESTBase to get item JSON (which was html-escaped)? Why? Anyway, you can just use Special:EntityData/Q1234.json to get the JSON. The response is cached in Varnish.

Yurik added a comment.Nov 29 2016, 5:06 PM

@daniel no, has nothing to do with items. https://www.wikidata.org/api/rest_v1/ no longer lists /graph/ as a supported service.