Page MenuHomePhabricator
Create Task
Maniphest T332228

High log volume from rdbms lib triggering logstash throttler
Closed, ResolvedPublic
Actions

Description

These logs have high enough rate that they trigger the logstash throttler. T314434

includes/libs/rdbms/ChronologyProtector.php

found position data under a presumed clientId (T314434)

We had to filter these logs at the appserver level to mitigate kafka lag towards codfw T330205#8699489

includes/libs/rdbms/platform/SQLPlatform.php

Wikimedia\Rdbms\Platform\SQLPlatform::isWriteQuery fallback to regex

from /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/platform/SQLPlatform.php(1760)
#0 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/database/Database.php(934): Wikimedia\Rdbms\Platform\SQLPlatform->isWriteQuery(string, integer)
#1 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/database/DatabaseMysqlBase.php(197): Wikimedia\Rdbms\Database->executeQuery(string, string, integer, string)
#2 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/database/Database.php(1779): Wikimedia\Rdbms\DatabaseMysqlBase->doSelectDomain(Wikimedia\Rdbms\DatabaseDomain)
#3 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1011): Wikimedia\Rdbms\Database->selectDomain(Wikimedia\Rdbms\DatabaseDomain)
#4 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/loadbalancer/LoadBalancer.php(869): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(integer, Wikimedia\Rdbms\DatabaseDomain, integer)
#5 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/loadbalancer/LoadBalancer.php(861): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#6 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/database/DBConnRef.php(103): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#7 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/database/DBConnRef.php(117): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#8 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/database/DBConnRef.php(722): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#9 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/rdbms/database/Database.php(2963): Wikimedia\Rdbms\DBConnRef->getSessionLagStatus()
#10 /srv/mediawiki/php-1.40.0-wmf.27/extensions/CentralAuth/includes/User/CentralAuthUser.php(619): Wikimedia\Rdbms\Database::getCacheSetOptions(Wikimedia\Rdbms\DBConnRef)
#11 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/objectcache/wancache/WANObjectCache.php(1759): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->MediaWiki\Extension\CentralAuth\User\{closure}(boolean, integer, array, NULL, array)
#12 /srv/mediawiki/php-1.40.0-wmf.27/includes/libs/objectcache/wancache/WANObjectCache.php(1589): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#13 /srv/mediawiki/php-1.40.0-wmf.27/extensions/CentralAuth/includes/User/CentralAuthUser.php(639): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#14 /srv/mediawiki/php-1.40.0-wmf.27/extensions/CentralAuth/includes/User/CentralAuthUser.php(448): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->loadFromCache()
#15 /srv/mediawiki/php-1.40.0-wmf.27/extensions/CentralAuth/includes/User/CentralAuthUser.php(691): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->loadState()
#16 /srv/mediawiki/php-1.40.0-wmf.27/extensions/CentralAuth/includes/User/CentralAuthUser.php(784): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->getId()
#17 /srv/mediawiki/php-1.40.0-wmf.27/extensions/CentralAuth/includes/CentralAuthPrimaryAuthenticationProvider.php(425): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->exists()
#18 /srv/mediawiki/php-1.40.0-wmf.27/includes/auth/AuthManager.php(2344): MediaWiki\Extension\CentralAuth\CentralAuthPrimaryAuthenticationProvider->testUserExists(string, integer)
#19 /srv/mediawiki/php-1.40.0-wmf.27/includes/auth/AuthManager.php(1060): MediaWiki\Auth\AuthManager->userExists(string, integer)
#20 /srv/mediawiki/php-1.40.0-wmf.27/includes/api/ApiQueryUsers.php(280): MediaWiki\Auth\AuthManager->canCreateAccount(string)
#21 /srv/mediawiki/php-1.40.0-wmf.27/includes/api/ApiQuery.php(679): ApiQueryUsers->execute()
#22 /srv/mediawiki/php-1.40.0-wmf.27/includes/api/ApiMain.php(1903): ApiQuery->execute()
#23 /srv/mediawiki/php-1.40.0-wmf.27/includes/api/ApiMain.php(880): ApiMain->executeAction()
#24 /srv/mediawiki/php-1.40.0-wmf.27/includes/api/ApiMain.php(851): ApiMain->executeActionWithErrorHandling()
#25 /srv/mediawiki/php-1.40.0-wmf.27/api.php(91): ApiMain->execute()
#26 /srv/mediawiki/php-1.40.0-wmf.27/api.php(46): wfApiMain()
#27 /srv/mediawiki/w/api.php(3): require(string)
#28 {main}

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
logstash: sample high-volume rdbms lib loggingoperations/puppetproduction+15 -1
profile: remove SQLPlatform::isWriteQuery high log volume mitigationoperations/puppetproduction+0 -3
rdbms: Add missing QUERY_CHANGE_ flag to internal "USE" querymediawiki/corewmf/1.40.0-wmf.27+1 -1
rdbms: Add missing QUERY_CHANGE_ flag to internal "USE" querymediawiki/coreREL1_40+1 -1
rdbms: Add missing QUERY_CHANGE_ flag to internal "USE" querymediawiki/coremaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

colewhite created this task.Mar 15 2023, 9:03 PM
Restricted Application added a project: Performance-Team. · View Herald TranscriptMar 15 2023, 9:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
colewhite updated the task description. (Show Details)Mar 15 2023, 9:08 PM
colewhite updated the task description. (Show Details)
Umherirrender added subscribers: Krinkle, aaron, Umherirrender.Mar 15 2023, 9:39 PM

Wikimedia\Rdbms\Platform\SQLPlatform::isWriteQuery fallback to regex is new this week and from https://gerrit.wikimedia.org/r/c/mediawiki/core/+/894110

Krinkle triaged this task as High priority.Mar 16 2023, 6:20 AM
gerritbot added a comment.Mar 17 2023, 2:05 AM

Change 900521 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] rdbms: Add missing QUERY_CHANGE_ flag to internal "USE" query

https://gerrit.wikimedia.org/r/900521

gerritbot added a project: Patch-For-Review.Mar 17 2023, 2:05 AM
Krinkle claimed this task.Mar 17 2023, 2:05 AM
Krinkle moved this task from Untriaged to Rdbms: MySQL support on the MediaWiki-libs-Rdbms board.
gerritbot added a comment.Mar 17 2023, 9:58 AM

Change 900521 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Add missing QUERY_CHANGE_ flag to internal "USE" query

https://gerrit.wikimedia.org/r/900521

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.1; 2023-03-20).Mar 17 2023, 10:00 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 17 2023, 10:11 AM
gerritbot added a comment.Mar 17 2023, 1:57 PM

Change 900667 had a related patch set uploaded (by Jforrester; author: Krinkle):

[mediawiki/core@wmf/1.40.0-wmf.27] rdbms: Add missing QUERY_CHANGE_ flag to internal "USE" query

https://gerrit.wikimedia.org/r/900667

gerritbot added a project: Patch-For-Review.Mar 17 2023, 1:57 PM

Change 900668 had a related patch set uploaded (by Jforrester; author: Krinkle):

[mediawiki/core@REL1_40] rdbms: Add missing QUERY_CHANGE_ flag to internal "USE" query

https://gerrit.wikimedia.org/r/900668

gerritbot added a comment.Mar 17 2023, 10:12 PM

Change 900668 merged by jenkins-bot:

[mediawiki/core@REL1_40] rdbms: Add missing QUERY_CHANGE_ flag to internal "USE" query

https://gerrit.wikimedia.org/r/900668

ReleaseTaggerBot added a project: MW-1.40-notes.Mar 17 2023, 11:00 PM
gerritbot added a comment.Mar 20 2023, 11:12 AM

Change 900667 merged by jenkins-bot:

[mediawiki/core@wmf/1.40.0-wmf.27] rdbms: Add missing QUERY_CHANGE_ flag to internal "USE" query

https://gerrit.wikimedia.org/r/900667

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2023, 11:30 AM
Krinkle reassigned this task from Krinkle to colewhite.EditedMar 20 2023, 11:40 AM

@colewhite The above should fix 99% if not 100% of them. I'm leaving this open and assigned to you to remove the filter or confirm that it's not solved and then feel free to sign back over to me.

Based on https://grafana.wikimedia.org/d/VCK8-FpZz/cwhite-logstash?orgId=1&refresh=1m&viewPanel=46 at least, it seems like it's working :)

ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.27; 2023-03-13); removed MW-1.40-notes.Mar 20 2023, 12:00 PM
colewhite added a comment.Mar 20 2023, 3:39 PM

Looks like wmf.27 is rolled out. Lets try it.

gerritbot added a comment.Mar 20 2023, 3:42 PM

Change 900716 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: remove SQLPlatform::isWriteQuery high log volume mitigation

https://gerrit.wikimedia.org/r/900716

gerritbot added a project: Patch-For-Review.Mar 20 2023, 3:42 PM
gerritbot added a comment.Mar 20 2023, 7:25 PM

Change 900716 merged by Cwhite:

[operations/puppet@production] profile: remove SQLPlatform::isWriteQuery high log volume mitigation

https://gerrit.wikimedia.org/r/900716

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2023, 7:30 PM
gerritbot added a comment.Mar 20 2023, 8:39 PM

Change 900718 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: sample high-volume rdbms lib logging

https://gerrit.wikimedia.org/r/900718

gerritbot added a project: Patch-For-Review.Mar 20 2023, 8:39 PM
gerritbot added a comment.Mar 21 2023, 4:25 PM

Change 900718 merged by Cwhite:

[operations/puppet@production] logstash: sample high-volume rdbms lib logging

https://gerrit.wikimedia.org/r/900718

Maintenance_bot removed a project: Patch-For-Review.Mar 21 2023, 4:30 PM
colewhite closed this task as Resolved.Mar 21 2023, 5:42 PM

The log rate is no longer overwhelming codfw. I've added filters to reduce ingest load on the cluster in a trackable way. You can view the events/sec rate here.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits