Page MenuHomePhabricator

LinkAlwaysKnownLookup should batch file lookup via RepoGroup::findFiles
Open, Needs TriagePublic

Description

In T408666: GlobalUserPage causes RequestTimeoutException due to unbatched queries in onTitleIsAlwaysKnown hook, we introduced LinkAlwaysKnownLookup, which is capable of evaluating the result of Title::isAlwaysKnown in batches. Currently, lookups of file existence are implemented using RepoGroup::findFile, which looks up one file at a time. However, RepoGroup::findFiles also exists, which is likely to be more efficient.

It seems that LocalRepo::findFiles implementation attempts to use a single SQL query, but ForeignRepo::findFiles just calls findFile in a loop (so for checks of file existence on Commons, this would actually NOT result in any performance improvement it seems). However, using the exposed batched logic makes sense, as it then means that whenever ForeignRepo::findFiles starts to actually batch things, LinkAlwaysKnownLookup would benefit from it instantly.

Event Timeline

@OSleger-WMF This is likely to be relevant for T430854 and friends you're working on.

Commons is using ForeignDBViaLBRepo which extends LocalRepo and using the same sqls. Also ForeignDBRepo extends LocalRepo.

Commons is using ForeignDBViaLBRepo which extends LocalRepo and using the same sqls. Also ForeignDBRepo extends LocalRepo.

Today I learned! Then what I wrote in the description is only relevant for Instant Commons (which is API based, IIRC), and in production it should significantly speed up things. Good to know :)