Page MenuHomePhabricator

mw1415 fatals due to serving responses from 1.39.0-wmf.10 (was DBQueryError: Unknown column page_restrictions)
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1054: Unknown column 'page_restrictions' in 'field list' (db1118)
Function: MediaWiki\Page\PageStore::getPageByNameViaLinkCache
Query: SELECT  page_id,page_namespace,page_title,page_is_redire
exception.trace
from /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1743)
#0 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1727): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1701): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1202): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1947): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(2048): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#5 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->selectRow(array, array, array, string, array, array)
#6 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(297): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(686): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#8 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/PageStore.php(199): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#9 /srv/mediawiki/php-1.39.0-wmf.10/includes/cache/LinkCache.php(482): MediaWiki\Page\PageStore->MediaWiki\Page\{closure}(Wikimedia\Rdbms\DBConnRef, integer, string, array)
#10 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1690): LinkCache->{closure}(boolean, integer, array, NULL, array)
#11 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1518): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#12 /srv/mediawiki/php-1.39.0-wmf.10/includes/cache/LinkCache.php(487): WANObjectCache->getWithSetCallback(string, integer, Closure)
#13 /srv/mediawiki/php-1.39.0-wmf.10/includes/cache/LinkCache.php(539): LinkCache->getGoodLinkRowInternal(TitleValue, Closure, integer)
#14 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/PageStore.php(203): LinkCache->getGoodLinkRow(integer, string, Closure, integer)
#15 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/PageStore.php(154): MediaWiki\Page\PageStore->getPageByNameViaLinkCache(integer, string, integer)
#16 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/PageStore.php(326): MediaWiki\Page\PageStore->getPageByName(integer, string, integer)
#17 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(4129): MediaWiki\Page\PageStore->getPageByReference(Title, integer)
#18 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(1101): Title->getFieldFromPageStore(string, integer)
#19 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(1118): Title->getContentModel()
#20 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(1575): Title->hasContentModel(string)
#21 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/includes/Config/WikiPageConfig.php(63): Title->isSiteJsonConfigPage()
#22 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/includes/Config/WikiPageConfig.php(86): GrowthExperiments\Config\WikiPageConfig->getConfigTitle()
#23 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/includes/Config/WikiPageConfig.php(153): GrowthExperiments\Config\WikiPageConfig->getConfigData(integer)
#24 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/includes/Config/GrowthExperimentsMultiConfig.php(101): GrowthExperiments\Config\WikiPageConfig->hasWithFlags(string, integer)
#25 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/includes/Config/GrowthExperimentsMultiConfig.php(84): GrowthExperiments\Config\GrowthExperimentsMultiConfig->getWithFlags(string)
#26 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/ServiceWiring.php(872): GrowthExperiments\Config\GrowthExperimentsMultiConfig->get(string)
#27 /srv/mediawiki/php-1.39.0-wmf.10/vendor/wikimedia/services/src/ServiceContainer.php(447): Wikimedia\Services\ServiceContainer::{closure}(MediaWiki\MediaWikiServices)
#28 /srv/mediawiki/php-1.39.0-wmf.10/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\Services\ServiceContainer->createService(string)
#29 /srv/mediawiki/php-1.39.0-wmf.10/includes/MediaWikiServices.php(301): Wikimedia\Services\ServiceContainer->getService(string)
#30 /srv/mediawiki/php-1.39.0-wmf.10/vendor/wikimedia/services/src/ServiceContainer.php(424): MediaWiki\MediaWikiServices->getService(string)
#31 /srv/mediawiki/php-1.39.0-wmf.10/vendor/wikimedia/object-factory/src/ObjectFactory.php(211): Wikimedia\Services\ServiceContainer->get(string)
#32 /srv/mediawiki/php-1.39.0-wmf.10/vendor/wikimedia/object-factory/src/ObjectFactory.php(152): Wikimedia\ObjectFactory\ObjectFactory::getObjectFromSpec(array, array)
#33 /srv/mediawiki/php-1.39.0-wmf.10/includes/HookContainer/HookContainer.php(473): Wikimedia\ObjectFactory\ObjectFactory->createObject(array)
#34 /srv/mediawiki/php-1.39.0-wmf.10/includes/HookContainer/HookContainer.php(156): MediaWiki\HookContainer\HookContainer->getHandlers(string, array)
#35 /srv/mediawiki/php-1.39.0-wmf.10/includes/HookContainer/HookRunner.php(2516): MediaWiki\HookContainer\HookContainer->run(string, array)
#36 /srv/mediawiki/php-1.39.0-wmf.10/includes/cache/MessageCache.php(951): MediaWiki\HookContainer\HookRunner->onMessageCache__get(string)
#37 /srv/mediawiki/php-1.39.0-wmf.10/includes/language/Message.php(1494): MessageCache->get(string, boolean, LanguageEn)
#38 /srv/mediawiki/php-1.39.0-wmf.10/includes/language/Message.php(971): Message->fetchMessage()
#39 /srv/mediawiki/php-1.39.0-wmf.10/includes/language/Message.php(1074): Message->format(string)
#40 /srv/mediawiki/php-1.39.0-wmf.10/includes/title/MalformedTitleException.php(55): Message->text()
#41 /srv/mediawiki/php-1.39.0-wmf.10/includes/title/MediaWikiTitleCodec.php(92): MalformedTitleException->__construct(string, string, array)
#42 /srv/mediawiki/php-1.39.0-wmf.10/includes/title/MediaWikiTitleCodec.php(399): MediaWikiTitleCodec::{closure}(string, string)
#43 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(3020): MediaWikiTitleCodec->splitTitleString(string, integer)
#44 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(477): Title->secureAndSplit(string)
#45 /srv/mediawiki/php-1.39.0-wmf.10/includes/MediaWiki.php(90): Title::newFromURL(string)
#46 /srv/mediawiki/php-1.39.0-wmf.10/includes/MediaWiki.php(163): MediaWiki->parseTitle()
#47 /srv/mediawiki/php-1.39.0-wmf.10/includes/MediaWiki.php(873): MediaWiki->getTitle()
#48 /srv/mediawiki/php-1.39.0-wmf.10/includes/MediaWiki.php(566): MediaWiki->main()
#49 /srv/mediawiki/php-1.39.0-wmf.10/index.php(50): MediaWiki->run()
#50 /srv/mediawiki/php-1.39.0-wmf.10/index.php(46): wfIndexMain()
#51 /srv/mediawiki/w/index.php(3): require(string)
#52 {main}
Notes

This is being reported by 1.39.0-wmf.10 even though the current train is moving from 1.39.0-wmf.14 to 1.39.0-wmf.15.

All from mw1415

Details

Request URL
https://en.wikipedia.org/

Event Timeline

dancy renamed this task from Wikimedia\Rdbms\DBQueryError: Error 1054: Unknown column 'page_restrictions' in 'field list' (dbXXXX)Function: MediaWiki\Page\PageStore::getPageByNameViaLinkCacheQuery: SELECT page_id,page_namespace,page_title,page_is_redire to mw1415: Wikimedia\Rdbms\DBQueryError: Error 1054: Unknown column 'page_restrictions' in 'field list' (dbXXXX)Function: MediaWiki\Page\PageStore::getPageByNameViaLinkCacheQuery: SELECT page_id,page_namespace,page_title,page_is_redire.Wed, Jun 8, 8:05 PM
dancy updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.

Ref T35334: Remove database column page.page_restrictions from MediaWiki core.

This column indeed has been dropped from production tables. 1.39.0-wmf.10 has come and gone and the issue here is that its code is still running. Supporting 1.39.0-wmf.10 at this point is not an option.

Krinkle renamed this task from mw1415: Wikimedia\Rdbms\DBQueryError: Error 1054: Unknown column 'page_restrictions' in 'field list' (dbXXXX)Function: MediaWiki\Page\PageStore::getPageByNameViaLinkCacheQuery: SELECT page_id,page_namespace,page_title,page_is_redire to mw1415 fatals due to serving responses from 1.39.0-wmf.10 (was DBQueryError: Unknown column page_restrictions).Wed, Jun 8, 8:18 PM

I looked at mw1415:/srv/mediawiki/wikiversions.json and all of its entries reference wmf.10. It's completely different than what's on the deploy server.

Krinkle added subscribers: Ladsgroup, Dzahn.

https://sal.toolforge.org/production?p=0&q=mw1415&d=

2022-05-09:

  • <ladsgroup@cumin1001> conftool action : set/pooled=inactive; selector: name=mw1415.eqiad.wmnet [production]
  • <ladsgroup@cumin1001> conftool action : set/pooled=no; selector: name=mw1415.eqiad.wmnet [production]

2022-05-06:

  • <dzahn@cumin2002> conftool action : set/pooled=no; selector: dc=eqiad,name=mw1415.eqiad.wmnet [production]
  • <dzahn@cumin2002> conftool action : set/pooled=no; selector: dc=codfw,name=mw1415.eqiad.wmnet

2022-03-31:

  • <duesen> created /var/run/php/use-config-schema on canary mw1415 to enable config schema loading (T304460)

/cc @Dzahn @Ladsgroup

Mentioned in SAL (#wikimedia-operations) [2022-06-08T20:42:27Z] <Krinkle> krinkle@mw1415: Run scap pull manually ref T310225

Mentioned in SAL (#wikimedia-operations) [2022-06-08T21:41:10Z] <mutante> repooled mw1415 after restarting apache and php-fpm, seeing all Icinga alerts recover etc T307755 T310225

mw1415 does not serve 500s anymore. T307755#7990623

What happened here is:

The machine died on May 5th.

Ticket was opened with dcops to check hardware. Server was set to pooled=inactive (to avoid warnings/errors for deployers who would not be able to scap to a broken machine. This takes the server out of "dsh groups" which means it's not getting scap updates.

dcops contacted Dell technician to fix it

Today Dell tech showed up and did fix it.

Monitoring noticed the host state change and was now serving 500s instead of being just unreachable. Started alerting.

serviceops happens to see 500s on canary server and it happens to be during deployment window. assumes new alert is caused by ongoing deployment, tells releng.

releng files T310225

IRC discussion starts

scap pull is run manually, server is set to pooled=false again, apache gets restarted, all icinga checks are rescheduled, monitoring recovers

server is set to pooled=yes again

hardware repair ticket closed

dancy claimed this task.

Thanks for the summary @Dzahn .