Same as T421295: unused images is broken

The issue is that cl_to was dropped from the wikireplicas mid-March, per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/2LYOH46IXCOPHGVYRULQPYFLUP5TE4VG/

In T421552#11760058, @LucasWerkmeister wrote:

In T421552#11760041, @JeanFred wrote:

At startup time, the Flask app imports a module which imports pywikibot ; but I would have thought only actual requests would trigger a pywikibot.Site initialization. Yet the logs seem to indicate that straight off the bat pywikibot does some throttling...

In Telegram, you wrote that the code looks like this:

self.site = pywikibot.Site(url=url)        
self.repo = self.site.data_repository()

At startup time, the Flask app imports a module which imports pywikibot ; but I would have thought only actual requests would trigger a pywikibot.Site initialization. Yet the logs seem to indicate that straight off the bat pywikibot does some throttling...

We are now using SPAQL compatible with both WDQS and QLever

I filed https://github.com/ad-freiburg/qlever/issues/2788 upstream

Is replacing

I think 69669c0 is good enough as a fix.

I switched the dashboard to use QLever, dashboard is now updating https://www.wikidata.org/wiki/Special:Diff/2472881133

Had excellent discussions with @Ainali and @Spinster on the topic, which really helped me to scope and prioritise features :)

Two solutions:

I know I encountered that before too, but I think it is resolved indeed. :)

Switched the dashboard to use QLever, works now :)

Will close as resolved then :)

Confirmed using pytest-socket

I added a case to handle that kind of errors more gracefully

I see that integraality suceeded on that page − all good now I think ? https://www.wikidata.org/w/index.php?title=Wikidata:WikiProject_Govdirectory/Albania/Coverage_of_first_level_agencies&action=history

Now succeeding when using QLever endpoint. Closing as resolved.

Loading discographies3.js while viewing https://www.wikidata.org/wiki/Q17544624 results in the following CSP errors in my browser's console:
Loading the stylesheet 'https://cdn.datatables.net/1.12.1/css/jquery.dataTables.css' violates the following Content Security Policy directive: [...]
Loading the script 'https://cdn.datatables.net/1.12.1/js/jquery.dataTables.js' violates the following Content Security Policy directive: [...]

Parameter documented at https://www.wikidata.org/wiki/Special:Diff/2469907080

Works, see example at https://www.wikidata.org/wiki/Wikidata:Tenth_Birthday/DACH_Culture_Contest/Lists/Culture/2

I’m pretty much done with this. one question: should the grouping order be preserved in the output, instead of being sorted by count ? Or be sorted by count, just like now?

Sounds like a good idea. Any suggestions on how to make the distinction? :)

hi @RobinIsadorable , thanks for using integraality and for reporting this. Can you provide more details here? Which dashboard were you trying to work on − can you provide a link?

Fairly sure this is due to e344ea80eb14

categorize-images is created by update-monuments since 4faee3445f96 (T319787).

This is now done, as present for Wikidata’s 13th birthday :)

One potential idea: using subqueries:

SELECT ?grouping ?higher_grouping ?grouping_link_value (COUNT(DISTINCT ?entity) as ?count) 
WITH {
  SELECT ?grouping (SAMPLE(?_higher_grouping) as ?higher_grouping) (COUNT(DISTINCT ?entity) as ?count) WHERE {
    ?entity wdt:P31 wd:Q3305213 .
    ?entity wdt:P195 ?grouping .
    OPTIONAL { ?grouping wdt:P17/wdt:P298 ?_higher_grouping }.
  }
  GROUP BY ?grouping ?higher_grouping
  HAVING (?count >= 100)
  LIMIT 1000
} AS %items
WHERE {
  INCLUDE %items.
  OPTIONAL {
    ?grouping rdfs:label ?labelMUL.
    FILTER(lang(?labelMUL)='mul')
  }.
  OPTIONAL {
    ?grouping rdfs:label ?labelEN.
    FILTER(lang(?labelEN)='en')
  }.
  BIND(COALESCE(?labelEN, ?labelMUL) AS ?grouping_link_value).
}
GROUP BY ?grouping ?higher_grouping ?grouping_link_value
ORDER BY DESC(?count)

does not time out.

Same thing happened to me today with https://commons.wikimedia.org/wiki/File:Au%C3%9Fenministerin_Beate_Meinl-Reisinger_empf%C3%A4ngt_eine_Delegation_der_Republik_Kongo_-_2025-06-25_-_MGR9335.jpg

I gave a try to Excalidraw and made https://commons.wikimedia.org/wiki/File:Integraality_architecture_diagram.png ; now added to https://wikitech.wikimedia.org/wiki/Tool:InteGraality

In T294892#10779177, @Addshore wrote:

How is it deployed? Primairly a web app thing?
Could a generic tool on toolforge for example also be made to work with any wikibase given the above considerations? (or at least, any wikibase that runs manifest?)

In T294892#10798339, @Addshore wrote:

I wonder where the whole wikibase-sparql thing in siteinfo came from? Not seen that before!

In T294892#10779177, @Addshore wrote:

Allow to pass the endpoint to pywikibot.data.sparql

Ideally all Wikibases would run wikibase manifest https://www.mediawiki.org/wiki/Extension:WikibaseManifest
This would provide an easy way for you to lookup wherre the endpoint is, and also what prefixes to use
Again an example on wikibase cloud https://addshore-alpha.wikibase.cloud/w/rest.php/wikibase-manifest/v0/manifest
See local_rdf_namespaces and external_services

I fixed the error handling, you should now more information when performing a manual update. I tried twice, and it’s WDQS times out (“Query timeout limit reached”) when running queries. The dashboard is trying to do something too expensive for WDQS it seems :-(

I’ll scope this to the problematic uncaught exception ; with 397c5e82 it should now be made clear to the user what is happening.

What is the meaning behind declining this @jon_amar-WMDE? Why shouldn’t I aim to support more Wikibases beyond Wikidata in my tool?

Thanks for reporting this. Do you have an example dashboard where this happens?

I had not commented here yet because I have commented elsewhere (some wikipage I think) before ; and others have made my points better than I would (in particular, @Dominicbm has very well put how tiring this periodic asking that goes nowhere is).

I have been noticing the issue for a while and found this ticket. Several of “my” Toolforge tools have duplicate entries:

Re-ran, it completed in ~400s. I guess the dataset is big and this is flirting with the timeouts.

@Palotabarat How did this happen − doing what, on which page etc?

And since qlever support Wikimedia Commons, this could unlock T294893: integraality for Structured Data on Commons?

The service was going in 403 forbidden (at least something different!) When I tried to stop/start, it started as a PHP service ; I then ran the command logged above in SAL ; and edited webservice.template to specify web: python3.9.