Fix problematic database master queries performed on HTTP GET/HEAD
Open, NormalPublic

Description

Most of MediaWiki was written with the assumption that MediaWiki and its data stores are collocated and connected via reliable, low-latency network links. Until recently, MediaWiki had minimal facilities for maintaining consistency and partition tolerance across wide-area network links. As a result, although the Wikimedia Foundation operates data centers in multiple locations, we only run MediaWiki in one location at any one time.

This has several practical consequences: first, we are not as fault-tolerant as we'd like to be. We have a secondary data center with enough capacity to serve our traffic in case our primary datacenter goes down, but it is in cold standby, meaning it takes some time (and some manual effort) to get it running. Second, site performance is poor for logged-in users that are geographically remote from Ashburn, Virginia, due to the time it takes to transmit and receive data across long-distance links. Thirdly, in some basic cases, like parsing pages, the the master database must be up, leading to a SPOF.

It's going to take a lot of work to fix this completely, but we are getting closer to being able to serve some traffic from secondary datacenter. Specifically, we would like to serve "reads" -- requests that don't require a master database connection -- from a secondary datacenter.

In order to serve reads from a different datacenter, we need to be able to predict which incoming requests will modify data, so that we can route them accordingly. We need to be able to make this determination at the edge -- i.e., the outermost layers of the infrastructure, so it cannot be complicated or slow.

The solution we have is to use the HTTP request method (T91820): GETs/HEADs are read-only, while POSTs are not. This was already true for most cases, but there is a long tail of actions with side-effects that are done via GET, such as purge, rollback, markpatrolled.

This task mostly involves fixing DBPerformance log warnings. Warnings can be dealt with be:
a) Changing DB master reads to use DB slaves
b) Moving the database updates to POST requests, the jobqueue, or at least to post-send updates via DeferredUpdates
c) Disabling warnings for a few exceptional cases like CentralAuth.

See +channel:DBPerformance on logstash.wikimedia.org

Most of these warnings are writes or master queries on HTTP GET requests, which would be cross DC in active-active setup for some user. Ideally we could eventually get these to zero.

Related Objects

StatusAssignedTask
Openaaron
Openaaron
Resolvedaude
DuplicateGilles
Duplicateaaron
Resolvedaaron
DuplicateNone
Resolvedhoo
DuplicateNone
DuplicateNone
Resolvedaaron
OpenNone
Resolvedaaron
DuplicateNone
Resolvedaaron
ResolvedNikerabbit
OpenNone
StalledNone
ResolvedKrinkle
ResolvedNikerabbit
Resolvedaaron
DuplicateNone
OpenNone
DuplicateNone
OpenNone
Resolvedaaron
OpenNone
Resolvedtstarling
ResolvedLadsgroup
ResolvedKrinkle
Duplicateaaron
Resolvedkostajh
ResolvedHuji
Openmatthiasmullie
OpenNone
OpenNone
There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 289810 had a related patch set uploaded (by Aaron Schulz):
Avoid master connection in recacheMessageCount()

https://gerrit.wikimedia.org/r/289810

Change 289811 had a related patch set uploaded (by Aaron Schulz):
Avoid DBPerformance warnings on PURGE/TRACE requests

https://gerrit.wikimedia.org/r/289811

Change 289810 merged by jenkins-bot:
Avoid master connection in recacheMessageCount()

https://gerrit.wikimedia.org/r/289810

Change 290865 had a related patch set uploaded (by Aaron Schulz):
Avoid master queries/writes in listUnattached()

https://gerrit.wikimedia.org/r/290865

Change 289811 merged by jenkins-bot:
Avoid DBPerformance warnings on PURGE/TRACE requests

https://gerrit.wikimedia.org/r/289811

Change 291109 had a related patch set uploaded (by Aaron Schulz):
Avoid master queries on HTTP GET in getVisibilitySettingsFromRow()

https://gerrit.wikimedia.org/r/291109

Change 291109 merged by jenkins-bot:
Avoid master queries on HTTP GET in getVisibilitySettingsFromRow()

https://gerrit.wikimedia.org/r/291109

Change 290865 merged by jenkins-bot:
Avoid master queries/writes in listUnattached()

https://gerrit.wikimedia.org/r/290865

Change 292522 had a related patch set uploaded (by Aaron Schulz):
Avoid DBPerformance warnings in writeToDatabase()

https://gerrit.wikimedia.org/r/292522

Change 292524 had a related patch set uploaded (by Aaron Schulz):
Avoid DBPerformance warnings in Category::initialize()

https://gerrit.wikimedia.org/r/292524

Change 292524 merged by jenkins-bot:
Avoid DBPerformance warnings in Category::initialize()

https://gerrit.wikimedia.org/r/292524

Change 292522 merged by jenkins-bot:
Avoid DBPerformance warnings in writeToDatabase()

https://gerrit.wikimedia.org/r/292522

aaron renamed this task from Fix various DB master warnings from dbperformance.log to Fix problematic master queries performed on HTTP GET.Jun 17 2016, 1:37 AM
aaron updated the task description. (Show Details)
aaron renamed this task from Fix problematic master queries performed on HTTP GET to Fix problematic master queries performed on HTTP GET/HEAD.Jun 17 2016, 1:50 AM
aaron updated the task description. (Show Details)
aaron renamed this task from Fix problematic master queries performed on HTTP GET/HEAD to Fix problematic database master queries performed on HTTP GET/HEAD.

Change 297577 had a related patch set uploaded (by Gergő Tisza):
Avoid DBPerformance log warnings in onUserLogout()

https://gerrit.wikimedia.org/r/297577

Change 297577 merged by jenkins-bot:
Avoid DBPerformance log warnings in onUserLogout()

https://gerrit.wikimedia.org/r/297577

Change 299880 had a related patch set uploaded (by Gergő Tisza):
Fix central logout

https://gerrit.wikimedia.org/r/299880

Change 299880 merged by jenkins-bot:
Fix central logout

https://gerrit.wikimedia.org/r/299880

Change 309861 had a related patch set uploaded (by Aaron Schulz):
Avoid page_touched update for HTTP GET action=purge requests

https://gerrit.wikimedia.org/r/309861

Change 309868 had a related patch set uploaded (by Aaron Schulz):
Make SessionBackend::save() update the user post-send

https://gerrit.wikimedia.org/r/309868

Change 309877 had a related patch set uploaded (by Aaron Schulz):
Make CentralAuthUser::register() trigger importLocalNames()

https://gerrit.wikimedia.org/r/309877

Change 309861 merged by jenkins-bot:
Avoid page_touched update for HTTP GET action=purge requests

https://gerrit.wikimedia.org/r/309861

Change 309868 merged by jenkins-bot:
Make SessionBackend::save() update the user post-send

https://gerrit.wikimedia.org/r/309868

Change 309877 merged by jenkins-bot:
Make CentralAuthHooks::onLocalUserCreated() trigger importLocalNames()

https://gerrit.wikimedia.org/r/309877

Change 348030 had a related patch set uploaded (by Aaron Schulz):
[mediawiki/extensions/CentralAuth@master] Make opportunistic password hash upgrades post-send

https://gerrit.wikimedia.org/r/348030

Change 348030 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@master] Make opportunistic password hash upgrades post-send

https://gerrit.wikimedia.org/r/348030

Change 348658 had a related patch set uploaded (by Aaron Schulz):
[mediawiki/extensions/CentralAuth@master] Avoid triggering master queries in ApiValidatePassword

https://gerrit.wikimedia.org/r/348658

Change 350969 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Avoid master queries in loadAndLazyInit() for miser mode

https://gerrit.wikimedia.org/r/350969

Change 350969 merged by jenkins-bot:
[mediawiki/core@master] Avoid master queries in loadAndLazyInit() for miser mode

https://gerrit.wikimedia.org/r/350969

Change 351790 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/extensions/CentralAuth@master] Add $flags parameter to renameInProgressOn()

https://gerrit.wikimedia.org/r/351790

Change 351790 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@master] Add $flags parameter to renameInProgressOn()

https://gerrit.wikimedia.org/r/351790

Change 353823 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/extensions/CentralAuth@master] Avoid master queries in beginSecondaryAuthentication()

https://gerrit.wikimedia.org/r/353823

Change 353824 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/extensions/PageTriage@master] Avoid DB_MASTER queries on HTTP GET in ArticleMetadata->getMetadata

https://gerrit.wikimedia.org/r/353824

Change 353825 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/extensions/CentralAuth@master] Avoid master queries in SpecialGlobalRenameProgress

https://gerrit.wikimedia.org/r/353825

Change 353827 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Avoid DB_MASTER queries in User::newSystemUser() when possible

https://gerrit.wikimedia.org/r/353827

Change 353829 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/extensions/Translate@master] Use TranslateUtils::getSafeReadDB() in loadAggregateGroups

https://gerrit.wikimedia.org/r/353829

Change 353824 merged by jenkins-bot:
[mediawiki/extensions/PageTriage@master] Avoid DB_MASTER queries on HTTP GET in ArticleMetadata->getMetadata

https://gerrit.wikimedia.org/r/353824

Change 353829 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Use TranslateUtils::getSafeReadDB() in loadAggregateGroups

https://gerrit.wikimedia.org/r/353829

Change 353825 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@master] Avoid master queries in SpecialGlobalRenameProgress

https://gerrit.wikimedia.org/r/353825

Change 353827 merged by jenkins-bot:
[mediawiki/core@master] Avoid DB_MASTER queries in User::newSystemUser() when possible

https://gerrit.wikimedia.org/r/353827

Seb35 added a subscriber: Seb35.Jun 3 2017, 6:48 AM

Change 353823 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@master] Avoid master queries in beginSecondaryAuthentication()

https://gerrit.wikimedia.org/r/353823

Change 348658 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@master] Avoid triggering master queries in ApiValidatePassword

https://gerrit.wikimedia.org/r/348658