preload_sites.py script is too slow
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Xqt
	Oct 15 2023, 1:22 PM

Description

Steps to replicate the issue:

run pwb preload_sites -worker:25

What happens?:
The script needs up to 2 minutes to be completed and collect all sites

What should have happened instead?:
The script should terminate within 20 seconds. The reason for this lameness is that userinfo like [1] is called for each site and this api call is not cached.
[1] https://en.wikipedia.org/w/api.php?action=query&meta=userinfo&uiprop=blockinfo%7Cgroups%7Chasmsg%7Cratelimits%7Crights&formatversion=2&maxlag=5&format=json

Software version:
Pywikibot 8.5.0 - 8.0.4

This issue was introduced with rPWBC891a720

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
cleanup: preload_sites script was removed	pywikibot/core	master	+2 -8
pywikibot.scripts: Remove preload_sites.py	pywikibot/core	master	+17 -162
Revert "[IMPR] use CachedRequest for userinfo requests"	pywikibot/core	master	+5 -15
[IMPR] use CachedRequest for userinfo requests	pywikibot/core	master	+35 -12

Customize query in gerrit

Related Objects

Mentioned In: T365942: pywiibot-tests isn't able to login on wikipedia:tests recently
Mentioned Here: T365942: pywiibot-tests isn't able to login on wikipedia:tests recently
T350272: Invoking scripts with -help performs an unnecessary connection to the remote wiki
rPWBC891a720c09e6: [bugfix] No longer lazy load cookies

Event Timeline

Xqt created this task.Oct 15 2023, 1:22 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptOct 15 2023, 1:22 PM

Xqt triaged this task as High priority.Oct 15 2023, 1:23 PM

Xqt added a project: Regression.Oct 15 2023, 2:44 PM

See also T350272

Hi @Xqt what tests would you recommend to ensure the change is not causing regression?

I have the code change: https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1033715 where userinfo is being cached and read based on family name and username matches.

While testing, I noticed cookies are added to http.cookie_jar when the api call is actually made to retrieve userinfo but I am unsure if this is the cookie that is meant to be loaded in preload_sites script.

Hi @ericpien: I think preload_sites script is a good measurement. After collecting the siteinfo (and after Pywikibot 8.1 userinfo) the second call must much more faster. Thank you for your patch but I would suggest to use the already implemented CachedRequest to cache the userinfo which is already used for siteinfo. The advantage is that parallel running task will also benefit from it. At userinfo method the cached request can be implemented like

if not hasattr(self, '_userinfo'):
    uirequest = self._request(
        expiry=1,
        parameters=dict(
            action='query',
            meta='userinfo',
            uiprop='blockinfo|hasmsg|groups|rights|ratelimits',
            formatversion=2,
        )
    )
    uidata = uirequest.submit()

The 2nd run needs only ~30 seconds with this patch compared to 8 minutes without it. Maybe we could add some runtime test like this to our test matrix.

Change #1034059 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [IMPR] use CachedRequest for userinfo requests

https://gerrit.wikimedia.org/r/1034059

gerritbot added a project: Patch-For-Review.May 20 2024, 11:51 AM

The regression was introduced with Pywikibot 8.1 (see above). Previous runs were 30 times faster for the first time and 300 times for the second. The patch above fastens up the second call but the api calls seems sequentiell instead of simultaneously which would be expected with concurrent.futures.

Pretty sure we need muxh more workers here. With 5000 workers I get all sites within 30 seconds.

Xqt claimed this task.May 20 2024, 2:12 PM

Xqt closed this task as Resolved.May 25 2024, 3:48 PM

Change #1034059 merged by jenkins-bot:

[pywikibot/core@master] [IMPR] use CachedRequest for userinfo requests

https://gerrit.wikimedia.org/r/1034059

Maintenance_bot removed a project: Patch-For-Review.May 25 2024, 4:30 PM

Xqt mentioned this in T365942: pywiibot-tests isn't able to login on wikipedia:tests recently.May 26 2024, 11:53 AM

Change #1035864 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] Revert "[IMPR] use CachedRequest for userinfo requests"

https://gerrit.wikimedia.org/r/1035864

gerritbot added a project: Patch-For-Review.May 26 2024, 11:58 AM