Page MenuHomePhabricator

`user_add_claim_unless_exists` using the `inspect` module is very slow
Closed, ResolvedPublicBUG REPORT

Description

Background: I'm the author of a bot (https://github.com/konstin/github-wikidata-bot) that updates the software versions in wikidata pages. The initial processed set is 16k projects, less after filtering for various requirements for updating and a no-edit fast paths.

Since the bot may edit in parallel with others, it needs to use user_add_claim_unless_exists to create claims for versions (without it, i get errors for edit conflicts). I've noticed that the bot spends a large chunk of its runtime in method of the inspect for deserializing JSON. The call chain is roughly user_add_claim_unless_exists -> fromJSON -> __getattr__ -> inspect.signature. I was surprised by this, as usually the signatures shouldn't really change during runtime.

This may also be something I can improve on the bot's side, but I'm not really what the intended usage is.

image.png (1,445×906 px, 219 KB)

image.png (1,445×906 px, 205 KB)

Event Timeline

matej_suchanek subscribed.

DataSite.get_repo_for_entity_type calls DataSite.entity_source. This method is not implemented, hence it ends up in BaseSite.__getattr__ with the only purpose:

Delegate undefined methods calls to the Family object.

.. versionchanged:: 9.0
   Only delegate to public Family methods which have ``code`` as
   first parameter.

This is where inspect kicks in.

Perhaps there could be some caching layer.

Perhaps there could be some caching layer.

Maybe a conceptional change would be more appropriate e,g, having a set or dict within the Family class which holds methods which can be delegated. Instead inspecting the Family methods a simple lookup would do the job. Sample:

class Family:
    delegated_methods = {
        'foo',
        'bar',
        'baz',
    }
Xqt changed the task status from Open to In Progress.Jan 3 2026, 1:31 PM
Xqt claimed this task.

Change #1222701 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [IMPR] BaseSite.__getattr__: Use direct __code__ inspection for performance

https://gerrit.wikimedia.org/r/1222701

Change #1222701 merged by jenkins-bot:

[pywikibot/core@master] [IMPR] BaseSite.__getattr__: Use direct __code__ inspection for performance

https://gerrit.wikimedia.org/r/1222701