Handle timing out SPARQL endpoint in Pywikibot
Open, Needs TriagePublicBUG REPORT
Actions

Assigned To

None

Authored By

	Multichill
	Aug 9 2023, 9:46 PM

Description

I have some bots that are not doing their reporting. Looking at the logs I noticed I hit the default socket timeout of 45 seconds. The Wikidata SPARQL endpoint at https://query.wikidata.org/ has a server side time out of 60 seconds. That extra 15 seconds might just be enough.

Steps to replicate the issue (include links if applicable):

I added this to my user-config.py:

socket_timeout = 90

And ran a bot that does some heavy SPARQL queries ( https://github.com/multichill/toollabs/blob/master/bot/wikidata/painting_external-id_property_statistics.py ).

What happens?:

Not enough extra time so I'm hitting the server side time out:

  File "/home///pywikibot/pywikibot/comms/http.py", line 356, in error_handling_callback
    raise ServerError(
pywikibot.exceptions.ServerError: 500 Server Error: Internal Server Error
CRITICAL: Exiting due to uncaught exception ServerError: 500 Server Error: Internal Server Error

What should have happened instead?:

We should probably handle the 500 error as a recoverable error (just like the socket time out) and retry.

Software version (skip for WMF-hosted wikis like Wikipedia): Recent (but not latest) git version

Other information (browser name/version, screenshots, etc.):

Related Objects

Mentioned Here: T378788: TestCommonsQueryService.testLoginAndOauthPermisson failing with ServerError

Event Timeline

Multichill created this task.Aug 9 2023, 9:46 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptAug 9 2023, 9:46 PM

With the increased time out I was able to update https://www.wikidata.org/wiki/Wikidata:WikiProject_sum_of_all_paintings/Property_statistics ( https://github.com/multichill/toollabs/blob/master/bot/wikidata/painting_property_statistics.py )

Maintenance_bot added a project: Wikidata.Aug 9 2023, 10:29 PM

Xqt renamed this task from Handle timing out SPARQL endpint in Pywikibot to Handle timing out SPARQL endpoint in Pywikibot.Aug 10 2023, 7:11 AM

We should probably handle the 500 error as a recoverable error (just like the socket time out) and retry.

Maybe the WDQS maintainers will disagree, but to me, automatically retrying a query that timed out and already consumed a significant amount of resources in doing so doesn’t sound like a good idea.

In T343936#9083076, @Lucas_Werkmeister_WMDE wrote:

We should probably handle the 500 error as a recoverable error (just like the socket time out) and retry.

Maybe the WDQS maintainers will disagree, but to me, automatically retrying a query that timed out and already consumed a significant amount of resources in doing so doesn’t sound like a good idea.

That's current behavior. Queries often fail and on second try you do get valid data. Don't forget that the query service maintainers themselves implemented it like this in Pywikibot, see https://github.com/wikimedia/pywikibot/blame/master/pywikibot/data/sparql.py

Gehel moved this task from Incoming to Watching / Waiting on the Wikidata-Query-Service board.Aug 14 2023, 3:14 PM

matej_suchanek moved this task from Backlog to Data loading problems on the Pywikibot-Wikidata board.Feb 16 2024, 12:55 PM

Solved for Pywikibot with patch for T378788 by implementing wait cycles as suggested.

Xqt moved this task from Data loading problems to Upstream on the Pywikibot-Wikidata board.Nov 5 2024, 4:41 PM

Xqt moved this task from Backlog to Upstream issues on the Pywikibot board.

Handle timing out SPARQL endpoint in PywikibotOpen, Needs TriagePublicBUG REPORTActions

Description

Related Objects

Event Timeline

Handle timing out SPARQL endpoint in Pywikibot
Open, Needs TriagePublicBUG REPORT
Actions