Page MenuHomePhabricator

CachedRequest class occasionally misses the cache
Closed, ResolvedPublic

Description

While trying to debug another issue I found out that occasionally an already cached Request is fetched again through API.

After looking into it turned out that sometimes the CachedRequest._uniquedescriptionstr is:

APISite("en", "wpbeta")LoginStatus(NOT_LOGGED_IN)[('action', 'query'), ('continue', ''), ('format', 'json'), ('maxlag', '15'), ('meta', 'userinfo|siteinfo'), ('siprop', 'namespaces|namespacealiases|general'), ('uiprop', 'blockinfo|hasmsg')]

But other times:

APISite("en", "wpbeta")LoginStatus(NOT_LOGGED_IN)[('action', 'query'), ('continue', ''), ('format', 'json'), ('maxlag', '15'), ('meta', 'userinfo|siteinfo'), ('siprop', 'extensions'), ('uiprop', 'blockinfo|hasmsg')]

Notice the change of order in ('meta', 'userinfo|siteinfo') and ('meta', 'userinfo|siteinfo').

Because of this, Request._load_cache function which relies on CachedRequest._create_file_name and CachedRequest._uniquedescriptionstr , looks for the wrong cached filename and misses the cache.

'userinfo|siteinfo' parameters seem to be added via Request._add_defaults function.

_add_defaults uses set to make sure parameters are not redundant, but the problem is that order of elements in a set is not preserved...

After using using a deterministic workaround instead of set the issue was resolved. I'm going to propose a change accordingly.

Event Timeline

Dalba renamed this task from Cache misses in CachedRequest class to CachedRequest class occasional misses the cache.Mar 24 2017, 8:41 AM
Dalba renamed this task from CachedRequest class occasional misses the cache to CachedRequest class occasionally misses the cache.

Change 344584 had a related patch set uploaded (by Dalba):
[pywikibot/core@master] api.py: Preserve the order of parameters in Request._add_defaults

https://gerrit.wikimedia.org/r/344584

Dalba triaged this task as High priority.Mar 24 2017, 8:58 AM

Change 344584 merged by jenkins-bot:
[pywikibot/core@master] Request._add_defaults: Make sure the order of parameters does not change

https://gerrit.wikimedia.org/r/344584

Thanks to Xqt and Mpaa for reviewing.