Page MenuHomePhabricator

deployment-db10 databases are broken
Closed, ResolvedPublic

Description

After todays cloud incident (T329535), deployment-db10 came back without it's volume. The volume came back after some time, but it seems like there is some data corruption.

22:05:53 Wikimedia\Rdbms\DBQueryError from line 1534 of /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php: Error 1932: Table 'aawiki.revision' doesn't exist in engine
22:05:53 Function: MediaWiki\Revision\RevisionStore::fetchRevisionRowFromConds
22:05:53 Query: SELECT  rev_id,rev_page,rev_timestamp,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,rev_sha1,comment_rev_comment.comment_text AS `rev_comment_text`,comment_rev_comment.comment_data AS `rev_comment_data`,comment_rev_comment.comment_id AS `rev_comment_cid`,actor_rev_user.actor_user AS `rev_user`,actor_rev_user.actor_name AS `rev_user_text`,rev_actor,page_namespace,page_title,page_id,page_latest,page_is_redirect,page_len,user_name  FROM `revision` JOIN `comment` `comment_rev_comment` ON ((comment_rev_comment.comment_id = rev_comment_id)) JOIN `actor` `actor_rev_user` ON ((actor_rev_user.actor_id = rev_actor)) JOIN `page` ON ((page_id = rev_page)) LEFT JOIN `user` ON ((actor_rev_user.actor_user != 0) AND (user_id = actor_rev_user.actor_user))   WHERE page_namespace = 8 AND page_title = 'Gadgets-definition' AND (rev_id=page_latest)  LIMIT 1
22:05:53 
22:05:53 #0 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(1518): Wikimedia\Rdbms\Database->getQueryException('Table 'aawiki.r...', 1932, 'SELECT  rev_id,...', 'MediaWiki\\Revis...')
22:05:53 #1 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(1492): Wikimedia\Rdbms\Database->getQueryExceptionAndLog('Table 'aawiki.r...', 1932, 'SELECT  rev_id,...', 'MediaWiki\\Revis...')
22:05:53 #2 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(865): Wikimedia\Rdbms\Database->reportQueryError('Table 'aawiki.r...', 1932, 'SELECT  rev_id,...', 'MediaWiki\\Revis...', false)
22:05:53 #3 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(1629): Wikimedia\Rdbms\Database->query('SELECT  rev_id,...', 'MediaWiki\\Revis...', 32)
22:05:53 #4 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(1638): Wikimedia\Rdbms\Database->select(Array, Array, Array, 'MediaWiki\\Revis...', Array, Array)
22:05:53 #5 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/DBConnRef.php(119): Wikimedia\Rdbms\Database->selectRow(Array, Array, Array, 'MediaWiki\\Revis...', Array, Array)
22:05:53 #6 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/DBConnRef.php(357): Wikimedia\Rdbms\DBConnRef->__call('selectRow', Array)
22:05:53 #7 /srv/mediawiki-staging/php-master/includes/Revision/RevisionStore.php(2396): Wikimedia\Rdbms\DBConnRef->selectRow(Array, Array, Array, 'MediaWiki\\Revis...', Array, Array)
22:05:53 #8 /srv/mediawiki-staging/php-master/includes/Revision/RevisionStore.php(2340): MediaWiki\Revision\RevisionStore->fetchRevisionRowFromConds(Object(Wikimedia\Rdbms\DBConnRef), Array, 0, Array)
22:05:53 #9 /srv/mediawiki-staging/php-master/includes/Revision/RevisionStore.php(1284): MediaWiki\Revision\RevisionStore->loadRevisionFromConds(Object(Wikimedia\Rdbms\DBConnRef), Array, 0, Object(Title))
22:05:53 #10 /srv/mediawiki-staging/php-master/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(139): MediaWiki\Revision\RevisionStore->getRevisionByTitle(Object(Title))
22:05:53 #11 /srv/mediawiki-staging/php-master/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(114): MediaWiki\Extension\Gadgets\MediaWikiGadgetsDefinitionRepo->fetchStructuredList()
22:05:53 #12 /srv/mediawiki-staging/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1759): MediaWiki\Extension\Gadgets\MediaWikiGadgetsDefinitionRepo->MediaWiki\Extension\Gadgets\{closure}(false, 86400, Array, NULL, Array)
22:05:53 #13 /srv/mediawiki-staging/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1589): WANObjectCache->fetchOrRegenerate('aawiki:gadgets-...', 86400, Object(Closure), Array, Array)
22:05:53 #14 /srv/mediawiki-staging/php-master/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(118): WANObjectCache->getWithSetCallback('aawiki:gadgets-...', 86400, Object(Closure), Array)
22:05:53 #15 /srv/mediawiki-staging/php-master/includes/libs/objectcache/BagOStuff.php(216): MediaWiki\Extension\Gadgets\MediaWikiGadgetsDefinitionRepo->MediaWiki\Extension\Gadgets\{closure}(9)
22:05:53 #16 /srv/mediawiki-staging/php-master/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(121): BagOStuff->getWithSetCallback('aawiki:gadgets-...', 9, Object(Closure))
22:05:53 #17 /srv/mediawiki-staging/php-master/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(42): MediaWiki\Extension\Gadgets\MediaWikiGadgetsDefinitionRepo->loadGadgets()
22:05:53 #18 /srv/mediawiki-staging/php-master/extensions/Gadgets/includes/Hooks.php(228): MediaWiki\Extension\Gadgets\MediaWikiGadgetsDefinitionRepo->getGadgetIds()
22:05:53 #19 /srv/mediawiki-staging/php-master/includes/HookContainer/HookContainer.php(160): MediaWiki\Extension\Gadgets\Hooks->onResourceLoaderRegisterModules(Object(MediaWiki\ResourceLoader\ResourceLoader))
22:05:53 #20 /srv/mediawiki-staging/php-master/includes/ResourceLoader/HookRunner.php(46): MediaWiki\HookContainer\HookContainer->run('ResourceLoaderR...', Array, Array)
22:05:53 #21 /srv/mediawiki-staging/php-master/includes/ServiceWiring.php(1629): MediaWiki\ResourceLoader\HookRunner->onResourceLoaderRegisterModules(Object(MediaWiki\ResourceLoader\ResourceLoader))
22:05:53 #22 /srv/mediawiki-staging/php-master/vendor/wikimedia/services/src/ServiceContainer.php(447): Wikimedia\Services\ServiceContainer::{closure}(Object(MediaWiki\MediaWikiServices))
22:05:53 #23 /srv/mediawiki-staging/php-master/vendor/wikimedia/services/src/ServiceContainer.php(411): Wikimedia\Services\ServiceContainer->createService('ResourceLoader')
22:05:53 #24 /srv/mediawiki-staging/php-master/includes/MediaWikiServices.php(306): Wikimedia\Services\ServiceContainer->getService('ResourceLoader')
22:05:53 #25 /srv/mediawiki-staging/php-master/includes/MediaWikiServices.php(1604): MediaWiki\MediaWikiServices->getService('ResourceLoader')
22:05:53 #26 /srv/mediawiki-staging/php-master/maintenance/purgeMessageBlobStore.php(17): MediaWiki\MediaWikiServices->getResourceLoader()
22:05:53 #27 /srv/mediawiki-staging/php-master/maintenance/includes/MaintenanceRunner.php(578): PurgeMessageBlobStore->execute()
22:05:53 #28 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(99): MediaWiki\Maintenance\MaintenanceRunner->run()
22:05:53 #29 /srv/mediawiki-staging/php-master/maintenance/purgeMessageBlobStore.php(23): require_once('/srv/mediawiki-...')
22:05:53 #30 /srv/mediawiki-staging/multiversion/MWScript.php(118): require_once('/srv/mediawiki-...')
22:05:53 #31 {main}

Related Objects

StatusSubtypeAssignedTask
ResolvedNone
ResolvedZabe

Event Timeline

Change 888807 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] beta: Depool deployment-db10

https://gerrit.wikimedia.org/r/888807

Change 888807 merged by jenkins-bot:

[operations/mediawiki-config@master] beta: Depool deployment-db10

https://gerrit.wikimedia.org/r/888807

Mentioned in SAL (#wikimedia-releng) [2023-02-13T23:20:21Z] <zabe> created backup of all databases on deployment-db09 # T329577

I won't try recovering deployment-db10, I will just create deployment-db11 as a replacement.

Mentioned in SAL (#wikimedia-releng) [2023-02-13T23:35:45Z] <zabe> create deployment-db11 as g3.cores8.ram16.disk20 # T329577

Mentioned in SAL (#wikimedia-releng) [2023-02-13T23:48:06Z] <zabe> create volume db11 and attach to deployment-db11 # T329577

Change 888814 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] beta: Remove deployment-db10

https://gerrit.wikimedia.org/r/888814

Change 888814 merged by jenkins-bot:

[operations/mediawiki-config@master] beta: Remove deployment-db10

https://gerrit.wikimedia.org/r/888814

Mentioned in SAL (#wikimedia-releng) [2023-02-14T00:22:32Z] <zabe> install mariadb 10.6 via role::mariadb::beta on deployment-db11 # T329577

Mentioned in SAL (#wikimedia-releng) [2023-02-14T01:06:47Z] <zabe> rsync dump from deployment-db09 to deployment-db11 # T329577

Mentioned in SAL (#wikimedia-releng) [2023-02-14T01:14:45Z] <zabe> deployment-db11: import dump into mariadb # T329577

Mentioned in SAL (#wikimedia-releng) [2023-02-14T01:38:40Z] <zabe> create deployment-db12 as g3.cores8.ram16.disk20 # T329577

Mentioned in SAL (#wikimedia-releng) [2023-02-14T01:50:28Z] <zabe> create volume db12 and attach to deployment-db12 # T329577

Mentioned in SAL (#wikimedia-releng) [2023-02-14T02:09:54Z] <zabe> install mariadb 10.6 via role::mariadb::beta on deployment-db12 # T329577

Mentioned in SAL (#wikimedia-releng) [2023-02-14T02:16:18Z] <zabe> rsync dump from deployment-db09 to deployment-db12 # T329577

Mentioned in SAL (#wikimedia-releng) [2023-02-14T02:19:45Z] <zabe> deployment-db12: import dump into mariadb # T329577

Mentioned in SAL (#wikimedia-releng) [2023-02-14T11:38:02Z] <zabe> deployment-db11: start replicating from deployment-db09 # T329577

Change 889118 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] beta: Add deployment-db11 and deployment-db12

https://gerrit.wikimedia.org/r/889118

Change 889118 merged by jenkins-bot:

[operations/mediawiki-config@master] beta: Add deployment-db11 and deployment-db12

https://gerrit.wikimedia.org/r/889118

Change 889126 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] beta: Add deployment-db11 and deployment-db12 (part 2)

https://gerrit.wikimedia.org/r/889126

Change 889126 merged by jenkins-bot:

[operations/mediawiki-config@master] beta: Add deployment-db11 and deployment-db12 (part 2)

https://gerrit.wikimedia.org/r/889126

Change 889128 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] beta: Pool deployment-db11

https://gerrit.wikimedia.org/r/889128

Change 889128 merged by jenkins-bot:

[operations/mediawiki-config@master] beta: Pool deployment-db11

https://gerrit.wikimedia.org/r/889128

Mentioned in SAL (#wikimedia-releng) [2023-02-14T14:04:26Z] <zabe> delete deployment-db10 and volume db10 # T329577

Mentioned in SAL (#wikimedia-releng) [2023-02-14T15:02:47Z] <zabe> deployment-db12: start replicating from deployment-db09 # T329577

Change 889150 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] beta: Pool deployment-db12

https://gerrit.wikimedia.org/r/889150

Change 889150 merged by jenkins-bot:

[operations/mediawiki-config@master] beta: Pool deployment-db12

https://gerrit.wikimedia.org/r/889150

Zabe claimed this task.

deployment-db10 has been replaced