Page MenuHomePhabricator
Create Task
Maniphest T310225

mw1415 fatals due to serving responses from 1.39.0-wmf.10 (was DBQueryError: Unknown column page_restrictions)
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1054: Unknown column 'page_restrictions' in 'field list' (db1118)
Function: MediaWiki\Page\PageStore::getPageByNameViaLinkCache
Query: SELECT  page_id,page_namespace,page_title,page_is_redire
exception.trace
from /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1743)
#0 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1727): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1701): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1202): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(1947): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/Database.php(2048): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#5 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->selectRow(array, array, array, string, array, array)
#6 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(297): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(686): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#8 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/PageStore.php(199): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#9 /srv/mediawiki/php-1.39.0-wmf.10/includes/cache/LinkCache.php(482): MediaWiki\Page\PageStore->MediaWiki\Page\{closure}(Wikimedia\Rdbms\DBConnRef, integer, string, array)
#10 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1690): LinkCache->{closure}(boolean, integer, array, NULL, array)
#11 /srv/mediawiki/php-1.39.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1518): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#12 /srv/mediawiki/php-1.39.0-wmf.10/includes/cache/LinkCache.php(487): WANObjectCache->getWithSetCallback(string, integer, Closure)
#13 /srv/mediawiki/php-1.39.0-wmf.10/includes/cache/LinkCache.php(539): LinkCache->getGoodLinkRowInternal(TitleValue, Closure, integer)
#14 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/PageStore.php(203): LinkCache->getGoodLinkRow(integer, string, Closure, integer)
#15 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/PageStore.php(154): MediaWiki\Page\PageStore->getPageByNameViaLinkCache(integer, string, integer)
#16 /srv/mediawiki/php-1.39.0-wmf.10/includes/page/PageStore.php(326): MediaWiki\Page\PageStore->getPageByName(integer, string, integer)
#17 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(4129): MediaWiki\Page\PageStore->getPageByReference(Title, integer)
#18 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(1101): Title->getFieldFromPageStore(string, integer)
#19 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(1118): Title->getContentModel()
#20 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(1575): Title->hasContentModel(string)
#21 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/includes/Config/WikiPageConfig.php(63): Title->isSiteJsonConfigPage()
#22 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/includes/Config/WikiPageConfig.php(86): GrowthExperiments\Config\WikiPageConfig->getConfigTitle()
#23 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/includes/Config/WikiPageConfig.php(153): GrowthExperiments\Config\WikiPageConfig->getConfigData(integer)
#24 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/includes/Config/GrowthExperimentsMultiConfig.php(101): GrowthExperiments\Config\WikiPageConfig->hasWithFlags(string, integer)
#25 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/includes/Config/GrowthExperimentsMultiConfig.php(84): GrowthExperiments\Config\GrowthExperimentsMultiConfig->getWithFlags(string)
#26 /srv/mediawiki/php-1.39.0-wmf.10/extensions/GrowthExperiments/ServiceWiring.php(872): GrowthExperiments\Config\GrowthExperimentsMultiConfig->get(string)
#27 /srv/mediawiki/php-1.39.0-wmf.10/vendor/wikimedia/services/src/ServiceContainer.php(447): Wikimedia\Services\ServiceContainer::{closure}(MediaWiki\MediaWikiServices)
#28 /srv/mediawiki/php-1.39.0-wmf.10/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\Services\ServiceContainer->createService(string)
#29 /srv/mediawiki/php-1.39.0-wmf.10/includes/MediaWikiServices.php(301): Wikimedia\Services\ServiceContainer->getService(string)
#30 /srv/mediawiki/php-1.39.0-wmf.10/vendor/wikimedia/services/src/ServiceContainer.php(424): MediaWiki\MediaWikiServices->getService(string)
#31 /srv/mediawiki/php-1.39.0-wmf.10/vendor/wikimedia/object-factory/src/ObjectFactory.php(211): Wikimedia\Services\ServiceContainer->get(string)
#32 /srv/mediawiki/php-1.39.0-wmf.10/vendor/wikimedia/object-factory/src/ObjectFactory.php(152): Wikimedia\ObjectFactory\ObjectFactory::getObjectFromSpec(array, array)
#33 /srv/mediawiki/php-1.39.0-wmf.10/includes/HookContainer/HookContainer.php(473): Wikimedia\ObjectFactory\ObjectFactory->createObject(array)
#34 /srv/mediawiki/php-1.39.0-wmf.10/includes/HookContainer/HookContainer.php(156): MediaWiki\HookContainer\HookContainer->getHandlers(string, array)
#35 /srv/mediawiki/php-1.39.0-wmf.10/includes/HookContainer/HookRunner.php(2516): MediaWiki\HookContainer\HookContainer->run(string, array)
#36 /srv/mediawiki/php-1.39.0-wmf.10/includes/cache/MessageCache.php(951): MediaWiki\HookContainer\HookRunner->onMessageCache__get(string)
#37 /srv/mediawiki/php-1.39.0-wmf.10/includes/language/Message.php(1494): MessageCache->get(string, boolean, LanguageEn)
#38 /srv/mediawiki/php-1.39.0-wmf.10/includes/language/Message.php(971): Message->fetchMessage()
#39 /srv/mediawiki/php-1.39.0-wmf.10/includes/language/Message.php(1074): Message->format(string)
#40 /srv/mediawiki/php-1.39.0-wmf.10/includes/title/MalformedTitleException.php(55): Message->text()
#41 /srv/mediawiki/php-1.39.0-wmf.10/includes/title/MediaWikiTitleCodec.php(92): MalformedTitleException->__construct(string, string, array)
#42 /srv/mediawiki/php-1.39.0-wmf.10/includes/title/MediaWikiTitleCodec.php(399): MediaWikiTitleCodec::{closure}(string, string)
#43 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(3020): MediaWikiTitleCodec->splitTitleString(string, integer)
#44 /srv/mediawiki/php-1.39.0-wmf.10/includes/Title.php(477): Title->secureAndSplit(string)
#45 /srv/mediawiki/php-1.39.0-wmf.10/includes/MediaWiki.php(90): Title::newFromURL(string)
#46 /srv/mediawiki/php-1.39.0-wmf.10/includes/MediaWiki.php(163): MediaWiki->parseTitle()
#47 /srv/mediawiki/php-1.39.0-wmf.10/includes/MediaWiki.php(873): MediaWiki->getTitle()
#48 /srv/mediawiki/php-1.39.0-wmf.10/includes/MediaWiki.php(566): MediaWiki->main()
#49 /srv/mediawiki/php-1.39.0-wmf.10/index.php(50): MediaWiki->run()
#50 /srv/mediawiki/php-1.39.0-wmf.10/index.php(46): wfIndexMain()
#51 /srv/mediawiki/w/index.php(3): require(string)
#52 {main}
Notes

This is being reported by 1.39.0-wmf.10 even though the current train is moving from 1.39.0-wmf.14 to 1.39.0-wmf.15.

All from mw1415

Details

Request URL
https://en.wikipedia.org/

Related Objects

Event Timeline

dancy created this task.Jun 8 2022, 8:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 8 2022, 8:02 PM
dancy renamed this task from Wikimedia\Rdbms\DBQueryError: Error 1054: Unknown column 'page_restrictions' in 'field list' (dbXXXX)Function: MediaWiki\Page\PageStore::getPageByNameViaLinkCacheQuery: SELECT page_id,page_namespace,page_title,page_is_redire to mw1415: Wikimedia\Rdbms\DBQueryError: Error 1054: Unknown column 'page_restrictions' in 'field list' (dbXXXX)Function: MediaWiki\Page\PageStore::getPageByNameViaLinkCacheQuery: SELECT page_id,page_namespace,page_title,page_is_redire.Jun 8 2022, 8:05 PM
dancy updated the task description. (Show Details)
Krinkle added projects: MediaWiki-Core-Revision-backend, Platform Engineering.Jun 8 2022, 8:16 PM
Krinkle subscribed.

Ref T35334: Remove database column page.page_restrictions from MediaWiki core.

This column indeed has been dropped from production tables. 1.39.0-wmf.10 has come and gone and the issue here is that its code is still running. Supporting 1.39.0-wmf.10 at this point is not an option.

Krinkle renamed this task from mw1415: Wikimedia\Rdbms\DBQueryError: Error 1054: Unknown column 'page_restrictions' in 'field list' (dbXXXX)Function: MediaWiki\Page\PageStore::getPageByNameViaLinkCacheQuery: SELECT page_id,page_namespace,page_title,page_is_redire to mw1415 fatals due to serving responses from 1.39.0-wmf.10 (was DBQueryError: Unknown column page_restrictions).Jun 8 2022, 8:18 PM
Krinkle edited projects, added Deployments; removed Platform Engineering, MediaWiki-Core-Revision-backend.
dancy added a comment.Jun 8 2022, 8:20 PM

I looked at mw1415:/srv/mediawiki/wikiversions.json and all of its entries reference wmf.10. It's completely different than what's on the deploy server.

dancy updated the task description. (Show Details)Jun 8 2022, 8:21 PM
Krinkle added a project: serviceops.EditedJun 8 2022, 8:22 PM
Krinkle added subscribers: Ladsgroup, Dzahn.

https://sal.toolforge.org/production?p=0&q=mw1415&d=

2022-05-09:

  • <ladsgroup@cumin1001> conftool action : set/pooled=inactive; selector: name=mw1415.eqiad.wmnet [production]
  • <ladsgroup@cumin1001> conftool action : set/pooled=no; selector: name=mw1415.eqiad.wmnet [production]

2022-05-06:

  • <dzahn@cumin2002> conftool action : set/pooled=no; selector: dc=eqiad,name=mw1415.eqiad.wmnet [production]
  • <dzahn@cumin2002> conftool action : set/pooled=no; selector: dc=codfw,name=mw1415.eqiad.wmnet

2022-03-31:

  • <duesen> created /var/run/php/use-config-schema on canary mw1415 to enable config schema loading (T304460)

/cc @Dzahn @Ladsgroup

dancy added a subscriber: • dduvall.Jun 8 2022, 8:27 PM
Stashbot added a comment.Jun 8 2022, 8:42 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-08T20:42:27Z] <Krinkle> krinkle@mw1415: Run scap pull manually ref T310225

Zabe subscribed.Jun 8 2022, 9:10 PM
Stashbot mentioned this in T307755: mw1415 (canary appserver) is down, incl. mgmt.Jun 8 2022, 9:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-08T21:41:10Z] <mutante> repooled mw1415 after restarting apache and php-fpm, seeing all Icinga alerts recover etc T307755 T310225

Dzahn added a comment.EditedJun 8 2022, 9:46 PM

mw1415 does not serve 500s anymore. T307755#7990623

Dzahn added a comment.Jun 8 2022, 9:50 PM

What happened here is:

The machine died on May 5th.

Ticket was opened with dcops to check hardware. Server was set to pooled=inactive (to avoid warnings/errors for deployers who would not be able to scap to a broken machine. This takes the server out of "dsh groups" which means it's not getting scap updates.

dcops contacted Dell technician to fix it

Today Dell tech showed up and did fix it.

Monitoring noticed the host state change and was now serving 500s instead of being just unreachable. Started alerting.

serviceops happens to see 500s on canary server and it happens to be during deployment window. assumes new alert is caused by ongoing deployment, tells releng.

releng files T310225

IRC discussion starts

scap pull is run manually, server is set to pooled=false again, apache gets restarted, all icinga checks are rescheduled, monitoring recovers

server is set to pooled=yes again

hardware repair ticket closed

dancy closed this task as Resolved.Jun 8 2022, 10:39 PM
dancy claimed this task.

Thanks for the summary @Dzahn .

Krinkle mentioned this in T251942: Aggregate check_mw_versions alerts for each individual app server.Jun 9 2022, 12:52 AM
Krinkle moved this task from Untriaged to Jun 2022 on the Wikimedia-production-error board.Jun 23 2022, 4:01 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL