(If anyone is interested in reproducing my tests, I threw this onto a branch https://github.com/JeanFred/inteGraality/tree/jaeger − docker-compose up -d and visiting http://localhost:5000/queries?page=User:Jean-Fr%C3%A9d%C3%A9ric/Red_Pandas&property=P50&grouping=Q3359737 should be enough (the Jaeger UI will be on localhost:16686 )

Do you have some pywikibot-only test case?

Something I forgot to mention which may be relevant: I was using packaged pywikibot 2019.11.28.

The logs of that run are P10293.

In T244030#5842178, @zhuyifei1999 wrote:

Do you have some pywikibot-only test case?

At its core the app does:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import logging

import pywikibot

page_title = "User:Jean-Frédéric/Red Pandas"

logging.info("Creating site object...")
site = pywikibot.Site('en', 'wikipedia')

logging.info("Getting data repo...")
repo = site.data_repository()

logging.info("Getting Page...")
page = pywikibot.Page(repo, page_title)

logging.info("Getting templates...")
all_templates_with_params = page.templatesWithParams()

print(all_templates_with_params)

I ran this with the old pywikibot − P10294. with the more recent 3.0.20200111, it’s quite faster P10295

So this is https://www.mediawiki.org/wiki/Manual:Maxlag_parameter I guess in theory you could increase it to make it more aggressive, but I think it's better to get the lag fixed.

In T244030#5842328, @zhuyifei1999 wrote:

So this is https://www.mediawiki.org/wiki/Manualst:Maxlag_parameter I guess in theory you could increase it to make it more aggressive, but I think it's better to get the lag fixed.

I was under the impression that the purpose of Maxlag was to limit editing (which makes sense, as more edits just make the lag even worse) ; but here all I’m doing is read-access. Or am I missing something here?

So you mean read accesses should ignore maxlag? hmm

The query it is doing at first is https://www.wikidata.org/w/api.php?action=query&meta=siteinfo%7Cuserinfo&siprop=namespaces%7Cnamespacealiases%7Cgeneral&continue=&uiprop=blockinfo%7Chasmsg&maxlag=5&format=json MediaWiki does respect maxlag even in a read.

Mentioned in SAL (#wikimedia-cloud) [2020-02-03T16:57:19Z] <wm-bot> <jeanfred> Deploy latest from Git master: 3b2867f, 10f67fd, 2e6a669 (T244030), 279c92b, de6308e

In T244030#5842411, @zhuyifei1999 wrote:

So you mean read accesses should ignore maxlag? hmm
MediaWiki does respect maxlag even in a read.

I guess if the MediaWiki API itself does it, then there’s not much to do. I guess my expectations are off, but I do find it a bit crazy to do a pywikibot._code_fam_from_url('https://www.wikidata.org/wiki/Q1') and be throttled for over 20 minutes before getting back ('wikidata', Family("wikidata")) :-)

Trying to chart a path here:

1. Waiting for the WDQS lag / Wikidata maxlag somehow gets under control T243701: Wikidata maxlag repeatedly over 5s since Jan 20, 2020 (primarily caused by the query service)
2. Waiting for the Wikidata API to not factor WDQS lag for read-only queries T243701#5834792
3. Implementing some caching so that reading pages are less painful: dashboards updates would purge the config cache, but looking glass would use it.

Number 3 could be quite good to do.

In T244030#5842377, @JeanFred wrote:

In T244030#5842328, @zhuyifei1999 wrote:

So this is https://www.mediawiki.org/wiki/Manualst:Maxlag_parameter I guess in theory you could increase it to make it more aggressive, but I think it's better to get the lag fixed.

I was under the impression that the purpose of Maxlag was to limit editing (which makes sense, as more edits just make the lag even worse) ; but here all I’m doing is read-access. Or am I missing something here?

maxlag is a nice way for non-interactive bots to automatically back off if the server is lagged. If this is an interactive tool that humans are waiting on, set maxlag=0 and stop caring about it.

@Dvorapa @Xqt Would you consider removing the limit for accessing the data, since this doesn't have an impact on the WDQS lag? This would be a huge relief for people using tools based on Pywikibot that can be blocked when the WDQS lag is high. Thanks a lot :)

@Lea_Lacroix_WMDE I'm really sorry, but I don't understand your request. Could you read the description of T242081 (there is an explanation, how Pywikibot works and why it does not work with Wikidata currently) and specify, what "limit" do you mean and in which way should we "remove" it? :)

@Lea_Lacroix_WMDE Or better could you please change Manual:Maxlag parameter according to a new policy (we can then adapt Pywikibot's behavior according to changes in manual)

@Dvorapa Hey, What we meant is not a change in the policy, it's about when there are "read" API requests happening. Is that that the case that pywikibot tries to use/enforce maxlag when there are read (wbgetentities, search, query, etc.) API requests? I recommend stop doing it as reads won't affect the replication lag problem (they happen on replicas not the master).

In T244030#5865281, @Ladsgroup wrote:

@Dvorapa Hey, What we meant is not a change in the policy, it's about when there are "read" API requests happening. Is that that the case that pywikibot tries to use/enforce maxlag when there are read (wbgetentities, search, query, etc.) API requests? I recommend stop doing it as reads won't affect the replication lag problem (they happen on replicas not the master).

I see, but this seems like a significant change to how Pywikibot works with API, applying maxlag only to write requests, not read. @Xqt?

In T244030#5865281, @Ladsgroup wrote:

@Dvorapa Hey, What we meant is not a change in the policy, it's about when there are "read" API requests happening. Is that that the case that pywikibot tries to use/enforce maxlag when there are read (wbgetentities, search, query, etc.) API requests? I recommend stop doing it as reads won't affect the replication lag problem (they happen on replicas not the master).

Also the policy/manual/APi etiquette does not distinguish between read/write request when talking about maxlag. From my point of view, if server is flooded by write requests, flooding it with read requests even if write requests being delayed will not help server much.

Currently the bot works as follows (for read and write request):

• submit a query with the given maxlag parameter (default is 5) and check it's result
• if there is no maxlag error code, process in usual way
• if there is a maxlag error code then do:

1. wait for many seconds returned by the retry_after header value returned by the http request
2. unfortunately this retry_after value is always 5 seconds and does not do what one expect, see T210606 and T172293. Therefore read the current lag time from error info. Multiply this value by the number of retries (upto 15 by default) and increase the wait time as showin in the table of T210606 but not longer than 120 seconds
3. After a maximum retries throw a TimeoutError exception and stop the bot working

What can be done?

• give an adequate maxlag time on server side if it isn't already
• solve T172293 and give a better retry_after value to instruct the bot how long it should wait for next retry
• we can use the current lag for a better estimate for the wait time instead of the retry_after value
• we can use an increasing wait time by using the current lag instead of the calculated time shown in the table above
• double the maxlag parameter for read requests
• with increasing wait time we can also increase the maxlag parameter by 1 second (maybe limited to 9) for each try. For example:

ot something like that. This would decrease client requests a lot and not retry every 5 seconds but on the other hand increase the change of beeing successful at any time after some wait cycles.

In T244030#5842260, @JeanFred wrote:

I ran this with the old pywikibot − P10294. with the more recent 3.0.20200111, it’s quite faster P10295

There is no difference in the wait cycle behaviour of these versions; they always waited for 5 seconds due to retry_after value returned from request http header. There is a change in the current master branch which increases the wait time slightly but raises a TimeoutError after a number of retries given by max_retries parameter (in [user-]config.py)