Page MenuHomePhabricator

Many Special pages that query user blocks make primary database connections on GET requests
Open, LowPublic

Description

A large proportion of the warnings in T267945: Increase in DBPerformance warnings since 1.36.0-wmf.13 are coming from special page GET requests that check user blocks.

Summary

When PermissionManager::checkUserBlock is called with PermissionManager::RIGOR_SECURE, the primary database is always used, due to the following line:

$useReplica = ( $rigor !== self::RIGOR_SECURE );

Many special pages are calling this method on GET requests, either by passing PermissionManager::RIGOR_SECURE directly, or by not passing anything and using the default rigor level.

These special pages can either execute on GET or POST, and they simply do the same checks on both. Example with Special:MovePage:

  • Displaying the Special:MovePage form is a GET request
  • Posting the form is a POST request
  • They both do the same block check
  • But the primary database should only be checked for the POST request

Details

About 500,000 warnings in the last 24 hours (at the time of writing) satisfy the following filters:

  • normalized_message.keyword: Expectation (masterConns <=) 0 by MediaWiki::main not met (actual: {actual}): {query}
  • http_method: GET
  • exception.trace: SpecialPage->run
  • exception.trace: MediaWiki\Permissions\PermissionManager->checkUserBlock

99% of these are related to account creation, with a stack trace like this:

from /srv/mediawiki/php-1.37.0-wmf.5/includes/libs/rdbms/TransactionProfiler.php(432)
#0 /srv/mediawiki/php-1.37.0-wmf.5/includes/libs/rdbms/TransactionProfiler.php(212): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, string, integer)
#1 /srv/mediawiki/php-1.37.0-wmf.5/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1010): Wikimedia\Rdbms\TransactionProfiler->recordConnection(string, string, boolean)
#2 /srv/mediawiki/php-1.37.0-wmf.5/includes/libs/rdbms/loadbalancer/LoadBalancer.php(964): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#3 /srv/mediawiki/php-1.37.0-wmf.5/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1130): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.5/includes/GlobalFunctions.php(2307): Wikimedia\Rdbms\LoadBalancer->getMaintenanceConnectionRef(integer, array, string)
#5 /srv/mediawiki/php-1.37.0-wmf.5/includes/block/DatabaseBlock.php(283): wfGetDB(integer)
#6 /srv/mediawiki/php-1.37.0-wmf.5/includes/block/DatabaseBlock.php(936): MediaWiki\Block\DatabaseBlock::newLoad(User, integer, boolean, string)
#7 /srv/mediawiki/php-1.37.0-wmf.5/includes/block/BlockManager.php(145): MediaWiki\Block\DatabaseBlock::newListFromTarget(User, string, boolean)
#8 /srv/mediawiki/php-1.37.0-wmf.5/includes/user/User.php(1641): MediaWiki\Block\BlockManager->getUserBlock(User, WebRequest, boolean, boolean)
#9 /srv/mediawiki/php-1.37.0-wmf.5/includes/user/User.php(1936): User->getBlockedStatus(boolean, boolean)
#10 /srv/mediawiki/php-1.37.0-wmf.5/includes/Permissions/PermissionManager.php(725): User->getBlock(boolean)
#11 /srv/mediawiki/php-1.37.0-wmf.5/includes/Permissions/PermissionManager.php(460): MediaWiki\Permissions\PermissionManager->checkUserBlock(string, User, array, string, boolean, Title)
#12 /srv/mediawiki/php-1.37.0-wmf.5/includes/Permissions/PermissionManager.php(310): MediaWiki\Permissions\PermissionManager->getPermissionErrorsInternal(string, User, Title, string)
#13 /srv/mediawiki/php-1.37.0-wmf.5/includes/Permissions/UserAuthority.php(254): MediaWiki\Permissions\PermissionManager->getPermissionErrors(string, User, Title, string)
#14 /srv/mediawiki/php-1.37.0-wmf.5/includes/Permissions/UserAuthority.php(226): MediaWiki\Permissions\UserAuthority->internalCan(string, string, Title, MediaWiki\Permissions\PermissionStatus)
#15 /srv/mediawiki/php-1.37.0-wmf.5/includes/user/User.php(4406): MediaWiki\Permissions\UserAuthority->authorizeWrite(string, Title, MediaWiki\Permissions\PermissionStatus)
#16 /srv/mediawiki/php-1.37.0-wmf.5/includes/auth/AuthManager.php(1057): User->authorizeWrite(string, Title, MediaWiki\Permissions\PermissionStatus)
#17 /srv/mediawiki/php-1.37.0-wmf.5/includes/specials/SpecialCreateAccount.php(75): MediaWiki\Auth\AuthManager->checkAccountCreatePermissions(User)
#18 /srv/mediawiki/php-1.37.0-wmf.5/includes/specialpage/LoginSignupSpecialPage.php(238): SpecialCreateAccount->checkPermissions()
#19 /srv/mediawiki/php-1.37.0-wmf.5/includes/specialpage/SpecialPage.php(646): LoginSignupSpecialPage->execute(NULL)
#20 /srv/mediawiki/php-1.37.0-wmf.5/includes/specialpage/SpecialPageFactory.php(1396): SpecialPage->run(NULL)
#21 /srv/mediawiki/php-1.37.0-wmf.5/includes/MediaWiki.php(313): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#22 /srv/mediawiki/php-1.37.0-wmf.5/includes/MediaWiki.php(916): MediaWiki->performRequest()
#23 /srv/mediawiki/php-1.37.0-wmf.5/includes/MediaWiki.php(550): MediaWiki->main()
#24 /srv/mediawiki/php-1.37.0-wmf.5/index.php(53): MediaWiki->run()
#25 /srv/mediawiki/php-1.37.0-wmf.5/index.php(46): wfIndexMain()
#26 /srv/mediawiki/w/index.php(3): require(string)
#27 {main}

SpecialCreateAccount::checkPermissions eventually calls UserAuthority::authorizeWrite, which passes PermissionManager::RIGOR_SECURE through to PermissionManager::getPermissionErrors.

The other 1% come from various different special pages calling PermissionManager methods (getPermissionErrors, userCan, etc) directly.

Possible solutions

Change PermissionManager::checkUserBlock

A quick solution that should fix all of these cases would be to change the above line from PermissionManager::checkUserBlock to something like:

$useReplica = $rigor !== self::RIGOR_SECURE && $user->getRequest()->getMethod() !== 'GET';

PermissionManager::checkUserBlock already checks properties of the WebRequest, so if checking the HTTP request method here is undesirable, it could presumably be fixed at the same time and in the same way as checking the IP address, etc. (perhaps as part of T231930).

Change the callers

We could pass the correct rigor level from the callers. This would involve:

  • Somehow calling through to UserAuthority::authorizeRead instead of UserAuthority::authorizeWrite from SpecialCreateAccount::checkPermissions
  • Passing the appropriate rigor level from the special pages that call PermissionManager methods directly

Both of the above

This might be the most correct thing to do, since callers should pass the correct rigor level and PermissionManager::checkUserBlock should arguably not allow connecting to the primary database on GET requests.

Event Timeline

Pchelolo subscribed.

Platform Engineering is doing a big refactor of the internals of PermissionManager where this will be addressed more thoroughly.

In the meantime, with Authority we now have a slightly different approach to rigor levels. Authority has 4 methods:

  • probablyCan === RIGOR_QUICK
  • definitelyCan === RIGOR_FULL
  • authorizeRead === RIGOR_FULL + side effects like throttling
  • authorizeWrite === RIGOR_SECURE + side effects like throttling.

So the create account special page, making the same check on both, needs to distinguish between just looking at the page and executing the page. In MovePage or MergeHistory we've just added multiple methods, e.g. 'probablyCanCreateAccount' vs 'authorizeCreateAccount'. WE probably should just do the same here.

Krinkle renamed this task from Many Special pages that check user blocks are making primary database connections on GET requests to Many Special pages that query user blocks make primary database connections on GET requests.May 20 2021, 5:39 PM
Krinkle moved this task from Limbo to Perf recommendation on the Performance-Team (Radar) board.

Change 693913 had a related patch set uploaded (by Tchanders; author: Tchanders):

[mediawiki/core@master] Introduce quick permissions checks for AuthManager

https://gerrit.wikimedia.org/r/693913

Krinkle triaged this task as High priority.Dec 20 2021, 8:19 PM
Krinkle moved this task from Later to Current: Other on the Sustainability (MediaWiki-MultiDC) board.

Change 693913 merged by jenkins-bot:

[mediawiki/core@master] auth: Introduce quick permissions checks for AuthManager

https://gerrit.wikimedia.org/r/693913

I am seeing this for MoveTranslatableBundleSpecialPage in the Language-team dashboard. Is there a way to fix it?

daniel lowered the priority of this task from High to Low.Jun 29 2022, 10:55 AM

I am seeing this for MoveTranslatableBundleSpecialPage in the Language-team dashboard. Is there a way to fix it?

I think this is probably T316129: MovePage sometimes connects to the primary database on a GET request

Change 928834 had a related patch set uploaded (by TK-999; author: TK-999):

[mediawiki/core@master] SpecialMovePage: Avoid TransactionProfiler warnings

https://gerrit.wikimedia.org/r/928834

Change 928834 merged by jenkins-bot:

[mediawiki/core@master] SpecialMovePage: Avoid TransactionProfiler warnings

https://gerrit.wikimedia.org/r/928834

Un-assigning myself - the work I did was to fix the problem centrally, but callers can still do this deliberately. Leaving open for feature owners to make fixes.