Page MenuHomePhabricator

Deprecate WDQ in Template:Wikidata list and switch remaining lists
Closed, ResolvedPublic

Description

Template:Wikidata list (see links on https://www.wikidata.org/wiki/Q19860885 ) is the template used by Listeriabot to generate lists. It accepts old style wdq (wdq=) and sparql (sparql=). Deprecation warning should be added in the big projects when wdq= is set and a tracker category added. That way we can easily switch the remaining lists to the new syntax.

Goal of this task (is changed to) switching the remaining lists over to sparql.

Suggestions:

  • use https://tools.wmflabs.org/wdq2sparql/w2s.php
  • drop spaces in "claim [31:5]" for the tool to work: claim[31:5]
  • use SELECT DISTINCT instead of SELECT
  • replace | with \u007C in the template
  • add LIMIT 1000 where many results are expected (max is 5000)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 16 2016, 11:57 AM
Magnus added a subscriber: Magnus.Dec 16 2016, 3:55 PM

Rather than doing this for all the templates on all the wikis, this could be done centrally in ListeriaBot.

Multichill added a comment.EditedDec 16 2016, 4:08 PM

I added the warnings and some tracker categories on Wikidata, see https://www.wikidata.org/w/index.php?title=Template%3AWikidata_list&type=revision&diff=416999396&oldid=410001395

So it seems like reducing the old usage isn't too hard. Do you have some per wiki statistics @Magnus of wdq usage in lists per wiki? That probably helps to decide if it's worth the effort to do this centrally.

Actually, ListeriaBot automatically replaces "simple" WDQ with SPARQL upon editing. I could extend that to "all WDQ", with exceptions where the converter doesn't work. Some of the SPARQL is badly optimised, however, and will time-out, causing the list to become static (not edited anymore).

WDQ usage (wiki/pages):
wikidatawiki 559 (this one might change massively tomorrow after your updates)
dewiki 95
frwiki 76
enwiki 61
dawiki 25
itwiki 10
ndswiki 10
nlwiki 10
euwiki 9
svwiki 8
elwiki 8
cawiki 6
cywiki 4
ruwiki 4
fiwiki 4
eswiki 3
nowiki 2
commonswiki 2
ptwiki 2
etwiki 1
ltwiki 1

I also noticed that the query converter gives interesting output. Things like not using MINUS. I tend to also make the query a bit easier or better when converting. The number of pages is much lower than I expected on other wiki's. I guess that if you can just produce a list somewhere, we can just empty it out with a couple of volunteers.

It's now in the bot report (last column, linked to the per-wiki lists).

To be safe, maybe you want to add DISTINCT and LIMIT 5000 when converting

Lots queries are in userspace .. users probably don't mind if we improve them to keep working, but in general, I don't think we should attempt to edit them.

We are not aiming to change the visible result (only for the better, perhaps). And when I switch off WDQ support, they should be thankful that their lists keep working.

We are not aiming to change the visible result (only for the better, perhaps). And when I switch off WDQ support, they should be thankful that their lists keep working.

Exactly! Thanks for the overview. I just counted only 14 not failed WDQ lists.

Sure, but I don't think we necessarily need to do them all without the user ;)

Anyways, I left a short summary on frwiki

Esc3300 updated the task description. (Show Details)Dec 17 2016, 12:22 AM
Esc3300 added a comment.EditedDec 17 2016, 12:24 AM

something is broken in elwiki .. not sure what. Manual updates gives a message starting with:

Trying to update Χρήστης:Geraki/Θάνατοι_το_2016...
Warning: file_get_contents(): php_network_getaddresses: getaddrinfo failed: Name or service not known in /data/project/listeria/shared.inc on line 93
Warning: file_get_contents(https://elwiki.wikipedia.org/w/api.php?format=json&action=query&meta=siteinfo&siprop=namespaces%7Cnamespacealiases): failed to open stream: php_network_getaddresses: getaddrinfo failed: Name or service not known in /data/project/listeria/shared.inc on line 93

Aeroid has quite lot in their userspace, notably in dewiki.

Multichill renamed this task from Add deprecation warnings and tracking category for WDQ at Template:Wikidata list to Deprecate WDQ in Template:Wikidata list and switch remaining lists.Dec 17 2016, 11:25 AM
Multichill updated the task description. (Show Details)
Esc3300 updated the task description. (Show Details)Dec 17 2016, 1:27 PM

elwiki fixed, though the cause remains a mystery. Apparently, the language used is "elwiki", not "el".

Esc3300 added a comment.EditedDec 17 2016, 2:47 PM

Thanks. Somehow it seemed to work in daily bot mode, but not on user update.

@Magnus I think we done enough fixing. Let's start removing it everywhere? I removed it from Pywikibot today in T162585

What's the status here?

Magnus added a comment.Jul 4 2017, 9:36 AM

And I can't change that usage because I get caught in some spam filter. Any es admin around?

I think we can close this. There may be some lists left that were already broken before ..

I replaced the last usage on WIkidata, updated the template and documentation on both Wikidata and Enwp. I killed the tracker category. Switch the last eswp list and close this one?

Esc3300 closed this task as Resolved.Jul 5 2017, 10:20 AM
Magnus reopened this task as Open.Jul 6 2017, 9:17 AM

One left, but es.wp won't let me change it:
https://es.wikipedia.org/wiki/Usuario:Aeroid/Lista_de_los_distritos_de_la_Baja_Sajonia

You can use https://tools.wmflabs.org/wdq2sparql/w2s.php to convert the WDQ query to SPARQL.

XXN added a subscriber: XXN.Jul 6 2017, 10:15 AM

I succeeded to change it - diff, check if the new query (converted via wdq2sparql) is good enough.

I succeeded to change it - diff, check if the new query (converted via wdq2sparql) is good enough.

You need to change the parameter wdq= to sparql= (I can't do it, because I'm not autoconfirmed)

XXN added a comment.Jul 6 2017, 10:31 AM

@Mbch331, thanks, done.

Magnus closed this task as Resolved.Jul 6 2017, 10:37 AM
Magnus claimed this task.

All done now, thanks :-)