Page MenuHomePhabricator
Create Task
Maniphest T374210

Error connecting to db1237 as user wikiadmin2023: :real_connect(): (HY000/2002): Connection timed out on labswiki
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
message
Error connecting to db1237 as user wikiadmin2023: :real_connect(): (HY000/2002): Connection timed out
trace
from /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/Database.php(1210)
#0 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DatabaseMySQL.php(136): Wikimedia\Rdbms\Database->newExceptionAfterConnectError(string)
#1 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/Database.php(265): Wikimedia\Rdbms\DatabaseMySQL->open(string, string, string, null, null, string)
#2 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1056): Wikimedia\Rdbms\Database->initConnection()
#3 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(915): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(int, Wikimedia\Rdbms\DatabaseDomain, array)
#4 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(789): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(int, Wikimedia\Rdbms\DatabaseDomain, int)
#5 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1867): Wikimedia\Rdbms\LoadBalancer->getServerConnection(int, string, int)
#6 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1729): Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(bool, int, array, null, array)
#7 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1559): WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#8 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1878): WANObjectCache->getWithSetCallback(string, int, Closure, array)
#9 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(375): Wikimedia\Rdbms\LoadBalancer->getLagTimes()
#10 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(510): Wikimedia\Rdbms\LoadBalancer->getRandomNonLagged(array)
#11 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(446): Wikimedia\Rdbms\LoadBalancer->pickReaderIndex(array)
#12 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(769): Wikimedia\Rdbms\LoadBalancer->getReaderIndex(string)
#13 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(111): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(int, array, string, int)
#14 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(125): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#15 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(338): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#16 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(764): Wikimedia\Rdbms\DBConnRef->selectField(array, string, array, string, array, array)
#17 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CheckUser/src/Services/CheckUserCentralIndexManager.php(88): Wikimedia\Rdbms\SelectQueryBuilder->fetchField()
#18 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CheckUser/src/Jobs/PruneCheckUserDataJob.php(64): MediaWiki\CheckUser\Services\CheckUserCentralIndexManager->purgeExpiredRows(string, string)
#19 /srv/mediawiki/php-1.43.0-wmf.21/includes/jobqueue/JobRunner.php(369): MediaWiki\CheckUser\Jobs\PruneCheckUserDataJob->run()
#20 /srv/mediawiki/php-1.43.0-wmf.21/includes/jobqueue/JobRunner.php(330): JobRunner->doExecuteJob(MediaWiki\CheckUser\Jobs\PruneCheckUserDataJob)
#21 /srv/mediawiki/php-1.43.0-wmf.21/includes/jobqueue/JobRunner.php(225): JobRunner->executeJob(MediaWiki\CheckUser\Jobs\PruneCheckUserDataJob)
#22 /srv/mediawiki/php-1.43.0-wmf.21/maintenance/runJobs.php(101): JobRunner->run(array)
#23 /srv/mediawiki/php-1.43.0-wmf.21/maintenance/includes/MaintenanceRunner.php(703): RunJobs->execute()
#24 /srv/mediawiki/php-1.43.0-wmf.21/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#25 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#26 {main}
Impact

Purging of CheckUser data is not occurring consistently for labswiki, which should be occurring to ensure that we are in-line with data retention policies. There have been 503 of these in the last 24 hours which also causing logstash spam.

Appears to start after wmf.21 was deployed to labswiki via the train.

Related Objects
Search...

Event Timeline

Dreamy_Jazz created this task.Sep 6 2024, 9:58 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 6 2024, 9:58 AM
Dreamy_Jazz updated the task description. (Show Details)Sep 6 2024, 10:00 AM
Dreamy_Jazz added a project: DBA.Sep 6 2024, 10:03 AM
Dreamy_Jazz added a parent task: T373640: 1.43.0-wmf.21 deployment blockers.Sep 6 2024, 10:05 AM
Dreamy_Jazz updated the task description. (Show Details)
Dreamy_Jazz updated the task description. (Show Details)
Ladsgroup subscribed.Sep 6 2024, 10:48 AM

In less than a month, wikitech will go inside production and this would be automatically resolved. Is it breaking an important functionality?

Ladsgroup moved this task from Triage to Pending comment on the DBA board.Sep 6 2024, 10:48 AM
Dreamy_Jazz added a comment.Sep 6 2024, 10:59 AM
This comment was removed by Dreamy_Jazz.
Dreamy_Jazz added a comment.Sep 6 2024, 11:06 AM

Looking at the stack trace again, I see this isn't actually failing to purge data from the local CheckUser tables but the new central tables on extension1. Therefore this is the same issue as seen in T366574.

Purging the central tables is important, but should not contain any specifically private data as we do not have temporary accounts enabled on labswiki. As such, I think it's not something that must be resolved within the month.

RhinosF1 subscribed.Sep 6 2024, 11:09 AM
Jdforrester-WMF subscribed.Sep 6 2024, 1:34 PM
In T374210#10125034, @Dreamy_Jazz wrote:

As such, I think it's not something that must be resolved within the month.

In this case, is this no longer a train-blocker?

Dreamy_Jazz added a comment.Sep 6 2024, 1:39 PM
In T374210#10125742, @Jdforrester-WMF wrote:
In T374210#10125034, @Dreamy_Jazz wrote:

As such, I think it's not something that must be resolved within the month.

In this case, is this no longer a train-blocker?

Yup, and I'll un-tag it as such.

Dreamy_Jazz removed a parent task: T373640: 1.43.0-wmf.21 deployment blockers.Sep 6 2024, 1:39 PM
akosiaris subscribed.Sep 6 2024, 1:39 PM
thcipriani moved this task from Untriaged to Sep 2024 on the Wikimedia-production-error board.Sep 6 2024, 4:30 PM
Dreamy_Jazz added a comment.EditedSep 10 2024, 2:51 PM

It appears because of the DB connection issues that no data is actually being purged from the local CheckUser tables because the query rolls back. This is a serious problem.

Dreamy_Jazz mentioned this in T374462: CheckUser data is not being purged for labswiki.Sep 10 2024, 2:54 PM
Dreamy_Jazz added a subtask: T374462: CheckUser data is not being purged for labswiki.
Dreamy_Jazz closed subtask T374462: CheckUser data is not being purged for labswiki as Resolved.Sep 10 2024, 4:36 PM

Fixed through T374462.

Dreamy_Jazz closed this task as Resolved.Sep 10 2024, 4:38 PM
Dreamy_Jazz claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits