Feature request: WDQS WMAPI calls without continuation calls (one call per value of the input variables)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dipsacus_fullonum
	Jun 18 2020, 10:31 AM

Description

Feature request: When using MWAPI calls from Wikikdata Query Service, it would be nice to be able to disable all continuation calls but still make one API call for each case of different values of the input variables.

There is

bd:serviceParam wikibase:limit "once".

which limits MWAPI to only one API call but I miss something like

bd:serviceParam wikibase:limit "no continuation".

which I would like to use to make API calls for all values of the input variables but never generate continuation calls. That could for instance be used to fetch information about a set of Wikimedia pages using code like this:

SERVICE wikibase:mwapi
{
  bd:serviceParam wikibase:endpoint ?endpoint.
  bd:serviceParam wikibase:api "Generator".
  bd:serviceParam wikibase:limit "no continuation".
  bd:serviceParam mwapi:generator "allpages".
  bd:serviceParam mwapi:gaptitle ?title.
  bd:serviceParam mwapi:gaplimit "1".
 ?item wikibase:apiOutputItem mwapi:item.
 ?title wikibase:apiOutput mwapi:title.
 ?length wikibase:apiOutput "@length".
}

for some values of ?endpoint and ?title. Such a call is impossible now without disabling continuation calls because then it would try to get all pages from the used endpoints.

Related Objects

Mentioned Here: T255753: Wikidata Query Service MWAPI calls with bd:serviceParam wikibase:limit "once" execute 2 API calls instead of 1

Event Timeline

Dipsacus_fullonum created this task.Jun 18 2020, 10:31 AM

Restricted Application added a project: Wikidata. · View Herald TranscriptJun 18 2020, 10:31 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

The meaning of "once" should be exactly “no continuation” (notwithstanding T255753) – it should be a shortcut for wikibase:limitContinuations "0". (Due to T255753, it’s currently really a shortcut for wikibase:limitContinuations "1".) And this already seems to work for some extent?

SELECT * WHERE {
  VALUES ?endpoint { "en.wikipedia.org" "de.wikipedia.org" "fr.wikipedia.org" "it.wikipedia.org" }
  SERVICE wikibase:mwapi {
    bd:serviceParam wikibase:endpoint ?endpoint;
                    wikibase:api "Generator";
                    wikibase:limit "once";
                    mwapi:generator "allpages";
                    mwapi:gapfrom "Berlin";
                    mwapi:gaplimit "1".
    ?item wikibase:apiOutputItem mwapi:item.
    ?title wikibase:apiOutput mwapi:title.
    ?length wikibase:apiOutput "@length".
  }
}

endpoint	item	title	length
en.wikipedia.org	wd:Q64	Berlin	171163
en.wikipedia.org		Berlin's 2001 administrative reform	50
de.wikipedia.org	wd:Q64	Berlin	239769
fr.wikipedia.org	wd:Q64	Berlin	82379
it.wikipedia.org	wd:Q262009	Berlin	4065

One of them has an extra result, probably due to T255753, the rest look okay to me. (Though it’s curious that the other endpoints don’t have the extra result.)

With my fix for T255753, the results look fine:

endpoint	item	title	length
en.wikipedia.org	http://www.wikidata.org/entity/Q64	Berlin	171163
de.wikipedia.org	http://www.wikidata.org/entity/Q64	Berlin	239769
fr.wikipedia.org	http://www.wikidata.org/entity/Q64	Berlin	82379
it.wikipedia.org	http://www.wikidata.org/entity/Q262009	Berlin	4065

Aha, but some counters don’t get reset correctly:

SELECT * WHERE {
  VALUES ?endpoint { "en.wikipedia.org" "de.wikipedia.org" "fr.wikipedia.org" "it.wikipedia.org" }
  SERVICE wikibase:mwapi {
    bd:serviceParam wikibase:endpoint ?endpoint;
                    wikibase:api "Generator";
                    wikibase:limitContinuations "2";
                    mwapi:generator "allpages";
                    mwapi:gapfrom "Berlin";
                    mwapi:gaplimit "2".
    ?item wikibase:apiOutputItem mwapi:item.
    ?title wikibase:apiOutput mwapi:title.
    ?length wikibase:apiOutput "@length".
  }
}

I think each endpoint should have six results ((limitContinuations + 1) × gaplimit), but apparently past the first endpoint no more continuations are made:

endpoint	item	title	length
en.wikipedia.org	http://www.wikidata.org/entity/Q64	Berlin	171163
en.wikipedia.org		Berlin's 2001 administrative reform	50
en.wikipedia.org		Berlin's Academy of Sciences	67
en.wikipedia.org		Berlin's Egyptian Museum	39
en.wikipedia.org	http://www.wikidata.org/entity/Q23579210	Berlin's Environmental Zone	10515
en.wikipedia.org		Berlin's Gate	30
de.wikipedia.org	http://www.wikidata.org/entity/Q64	Berlin	239769
de.wikipedia.org	http://www.wikidata.org/entity/Q677207	Berlin, Berlin	31651
fr.wikipedia.org	http://www.wikidata.org/entity/Q64	Berlin	82379
fr.wikipedia.org	http://www.wikidata.org/entity/Q677207	Berlin, Berlin	9391
it.wikipedia.org	http://www.wikidata.org/entity/Q262009	Berlin	4065
it.wikipedia.org		Berlin, Berlin	18

https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Pagination says:

bd:serviceParam wikibase:limit "once" .

This makes the service to stop after the first API call and not use continuations.

wikibase:limitContinuations isn't mentioned in the manual, so I didn't know that it exists.

Yeah, mea culpa, I really need to update the documentation :/

Well, it seems that the requested feature already is here, but is undocumented:

bd:serviceParam wikibase:limitContinuations  "0".

Once the solution to T255753 is implemented, it will also be possible to use:

bd:serviceParam wikibase:limit "once" .

The latter is documented, but the documentation gives the impression that only one API is made even if the input variables have more than one value.

It also seems that the undocumented wikibase:limitContinuations sets a limit for the total number of continuation call across all values of the input variables. Consider this query:

SELECT ?num ?item ?title ?user ?timestamp ?comment
WHERE
{
  ?item wdt:P177 wd:Q739513. # Bridges crossing the river Gudenå
  BIND(CONCAT("Q", STRAFTER(STR(?item), "Q")) AS ?title)
  SERVICE wikibase:mwapi
  {
    bd:serviceParam wikibase:endpoint "www.wikidata.org".
    bd:serviceParam wikibase:api "Generator".
    bd:serviceParam wikibase:limitContinuations "20".
    bd:serviceParam mwapi:generator "revisions".
    bd:serviceParam mwapi:titles ?title.
    bd:serviceParam mwapi:prop "revisions".
    bd:serviceParam mwapi:rvprop "user|timestamp|comment".
    bd:serviceParam mwapi:grvdir "older".
    bd:serviceParam mwapi:grvlimit "1".
    ?user wikibase:apiOutput "revisions/rev/@user".
    ?timestamp wikibase:apiOutput "revisions/rev/@timestamp".
    ?comment wikibase:apiOutput "revisions/rev/@comment".
    ?num wikibase:apiOrdinal true.
  }
}

It list the revisions for 4 Wikidata items. The first item (Q12301310) has 16 revisions (1 start call + 15 continuation calls). The next item has 19 revisions but only 6 is in the result (1 start call + 5 continuation calls). For the rest is only one revision in the result. So it seems that

wikibase:limitContinuations "20"

is interpreted as max 20 (15 + 5) continuation in total for the whole query.

Gehel moved this task from Incoming to Feature Requests on the Wikidata-Query-Service board.Jun 24 2020, 12:33 PM

Gehel triaged this task as Medium priority.Sep 15 2020, 7:55 AM

Looks like this functions as expected, closing. Feel free to reopen if you disagree

Feature request: WDQS WMAPI calls without continuation calls (one call per value of the input variables)Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Feature request: WDQS WMAPI calls without continuation calls (one call per value of the input variables)
Closed, ResolvedPublic
Actions