Page MenuHomePhabricator

Echo triggering CentralAuth "Can only obtain a centralauthtoken when using CentralAuth sessions" error
Closed, ResolvedPublic

Description

We're still getting:

2016-07-12 18:38:21 [V4U5HApAMDgAACyfYbEAAAAX] mw1221 commonswiki 1.28.0-wmf.8 exception ERROR: [V4U5HApAMDgAACyfYbEAAAAX] /w/api.php?action=query&format=json&origin=https%3A%2F%2Fes%2Ewikipedia%2Eorg&centralauthtoken=7b1aae48ebbcd1df01a7c0202a45c59e2cfeec6&meta=siteinfo&siprop=uploaddialog&formatversion=2   UsageException from line 1487 of /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiBase.php: Can only obtain a centralauthtoken when using CentralAuth sessions {"exception_id":"V4U5HApAMDgAACyfYbEAAAAX"} 
[Exception UsageException] (/srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiBase.php:1487) Can only obtain a centralauthtoken when using CentralAuth sessions
  #0 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/api/ApiCentralAuthToken.php(51): ApiBase->dieUsage(string, string)
  #1 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(1373): ApiCentralAuthToken->execute()
  #2 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(438): ApiMain->executeAction()
  #3 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(49): ApiMain->execute()
  #4 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(88): EchoForeignWikiRequest->getCentralAuthToken(User)
  #5 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(68): EchoForeignWikiRequest->getQueryParams(string)
  #6 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(29): EchoForeignWikiRequest->getRequestParams()
  #7 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(709): EchoForeignWikiRequest->execute()
  #8 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(740): MWEchoNotifUser->getForeignData()
  #9 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(487): MWEchoNotifUser->getForeignCount(string)
  #10 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/Hooks.php(1150): MWEchoNotifUser->resetNotificationCount()
  #11 /srv/mediawiki/php-1.28.0-wmf.8/includes/Hooks.php(195): EchoHooks::onUserSaveSettings(User)
  #12 /srv/mediawiki/php-1.28.0-wmf.8/includes/user/User.php(3952): Hooks::run(string, array)
  #13 /srv/mediawiki/php-1.28.0-wmf.8/includes/auth/AuthManager.php(1678): User->saveSettings()
  #14 /srv/mediawiki/php-1.28.0-wmf.8/includes/Setup.php(861): MediaWiki\Auth\AuthManager->autoCreateUser(User, string, boolean)
  #15 /srv/mediawiki/php-1.28.0-wmf.8/includes/WebStart.php(137): include(string)
  #16 /srv/mediawiki/php-1.28.0-wmf.8/api.php(38): include(string)
  #17 /srv/mediawiki/w/api.php(3): include(string)
  #18 {main}

I'm putting up a troubleshooting patch.

It does happen when there is a central ID (included more just to show there are multiple entry points).

2016-07-12 19:30:15 [V4VFRwpAEDIAALPruqIAAAAD] mw1285 commonswiki 1.28.0-wmf.8 Echo DEBUG: Exception when fetching CentralAuth token: wiki: commonswiki, userName: Laurenthomp, userId: 6018965, centralId: 47181900, exception: badsession: Can only obtain a centralauthtoken when using CentralAuth sessions {"wiki":"commonswiki","userName":"Laurenthomp","userId":6018965,"centralId":47181900} 
[Exception UsageException] (/srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiBase.php:1487) Can only obtain a centralauthtoken when using CentralAuth sessions
  #0 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/api/ApiCentralAuthToken.php(51): ApiBase->dieUsage(string, string)
  #1 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(1373): ApiCentralAuthToken->execute()
  #2 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(438): ApiMain->executeAction()
  #3 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(57): ApiMain->execute()
  #4 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(112): EchoForeignWikiRequest->getCentralAuthToken(User)
  #5 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(92): EchoForeignWikiRequest->getQueryParams(string)
  #6 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(29): EchoForeignWikiRequest->getRequestParams()
  #7 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(709): EchoForeignWikiRequest->execute()
  #8 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(740): MWEchoNotifUser->getForeignData()
  #9 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(487): MWEchoNotifUser->getForeignCount(string)
  #10 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/Hooks.php(1150): MWEchoNotifUser->resetNotificationCount()
  #11 /srv/mediawiki/php-1.28.0-wmf.8/includes/Hooks.php(195): EchoHooks::onUserSaveSettings(User)
  #12 /srv/mediawiki/php-1.28.0-wmf.8/includes/user/User.php(3952): Hooks::run(string, array)
  #13 /srv/mediawiki/php-1.28.0-wmf.8/includes/auth/AuthManager.php(1678): User->saveSettings()
  #14 /srv/mediawiki/php-1.28.0-wmf.8/includes/Setup.php(861): MediaWiki\Auth\AuthManager->autoCreateUser(User, string, boolean)
  #15 /srv/mediawiki/php-1.28.0-wmf.8/includes/WebStart.php(137): include(string)
  #16 /srv/mediawiki/php-1.28.0-wmf.8/api.php(38): include(string)
  #17 /srv/mediawiki/w/api.php(3): include(string)
  #18 {main}
2016-07-12 19:30:16 [V4VFRwpAEDIAALPruqIAAAAD] mw1285 commonswiki 1.28.0-wmf.8 Echo DEBUG: Exception when fetching CentralAuth token: wiki: commonswiki, userName: Laurenthomp, userId: 6018965, centralId: 47181900, exception: badsession: Can only obtain a centralauthtoken when using CentralAuth sessions {"wiki":"commonswiki","userName":"Laurenthomp","userId":6018965,"centralId":47181900} 
[Exception UsageException] (/srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiBase.php:1487) Can only obtain a centralauthtoken when using CentralAuth sessions
  #0 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/api/ApiCentralAuthToken.php(51): ApiBase->dieUsage(string, string)
  #1 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(1373): ApiCentralAuthToken->execute()
  #2 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(438): ApiMain->executeAction()
  #3 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(57): ApiMain->execute()
  #4 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(112): EchoForeignWikiRequest->getCentralAuthToken(User)
  #5 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(92): EchoForeignWikiRequest->getQueryParams(string)
  #6 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(29): EchoForeignWikiRequest->getRequestParams()
  #7 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(709): EchoForeignWikiRequest->execute()
  #8 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(740): MWEchoNotifUser->getForeignData()
  #9 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(487): MWEchoNotifUser->getForeignCount(string)
  #10 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/model/Notification.php(161): MWEchoNotifUser->resetNotificationCount(integer)
  #11 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/mapper/NotificationMapper.php(61): Closure$EchoNotification::insert#2()
  #12 /srv/mediawiki/php-1.28.0-wmf.8/includes/db/Database.php(2482): Closure$EchoNotificationMapper::insert()
  #13 /srv/mediawiki/php-1.28.0-wmf.8/includes/db/Database.php(2454): DatabaseBase->runOnTransactionIdleCallbacks()
  #14 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/mapper/NotificationMapper.php(64): DatabaseBase->onTransactionIdle(Closure$EchoNotificationMapper::insert;1964115705)
  #15 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/model/Notification.php(165): EchoNotificationMapper->insert(EchoNotification)
  #16 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/model/Notification.php(94): EchoNotification->insert()
  #17 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/Notifier.php(21): EchoNotification::create(array)
  #18 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/controller/NotificationController.php(277): EchoNotifier::notifyWithNotification(User, EchoEvent)
  #19 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/controller/NotificationController.php(104): EchoNotificationController::doNotification(EchoEvent, User, string)
  #20 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/model/Event.php(155): EchoNotificationController::notify(EchoEvent, boolean)
  #21 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/DiscussionParser.php(72): EchoEvent::create(array)
  #22 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/Hooks.php(474): EchoDiscussionParser::generateEventsForRevision(Revision)
  #23 /srv/mediawiki/php-1.28.0-wmf.8/includes/deferred/CallableUpdate.php(22): Closure$EchoHooks::onArticleSaved()
  #24 /srv/mediawiki/php-1.28.0-wmf.8/includes/deferred/DeferredUpdates.php(159): MWCallableUpdate->doUpdate()
  #25 /srv/mediawiki/php-1.28.0-wmf.8/includes/deferred/DeferredUpdates.php(88): DeferredUpdates::execute(array, string)
  #26 /srv/mediawiki/php-1.28.0-wmf.8/includes/MediaWiki.php(776): DeferredUpdates::doUpdates(string)
  #27 /srv/mediawiki/php-1.28.0-wmf.8/includes/MediaWiki.php(618): MediaWiki->restInPeace(string)
  #28 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
  #29 {main}
2016-07-12 19:31:16 [V4VFhApAMEAAACPlaaYAAABQ] mw1229 commonswiki 1.28.0-wmf.8 Echo DEBUG: Exception when fetching CentralAuth token: wiki: commonswiki, userName: Antonio wimoveis, userId: 6
018344, centralId: 47129486, exception: badsession: Can only obtain a centralauthtoken when using CentralAuth sessions {"wiki":"commonswiki","userName":"Antonio wimoveis","userId":6018344,"c
entralId":47129486} 
[Exception UsageException] (/srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiBase.php:1487) Can only obtain a centralauthtoken when using CentralAuth sessions
  #0 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/api/ApiCentralAuthToken.php(51): ApiBase->dieUsage(string, string)
  #1 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(1373): ApiCentralAuthToken->execute()
  #2 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(438): ApiMain->executeAction()
  #3 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(57): ApiMain->execute()
  #4 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(112): EchoForeignWikiRequest->getCentralAuthToken(User)
  #5 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(92): EchoForeignWikiRequest->getQueryParams(string)
  #6 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(29): EchoForeignWikiRequest->getRequestParams()
  #7 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(709): EchoForeignWikiRequest->execute()
  #8 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(770): MWEchoNotifUser->getForeignData()
  #9 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(492): MWEchoNotifUser->getForeignTimestamp(string)
  #10 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(365): MWEchoNotifUser->resetNotificationCount(integer)
  #11 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/api/ApiEchoMarkRead.php(22): MWEchoNotifUser->markRead(array)
  #12 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(1373): ApiEchoMarkRead->execute()
  #13 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(469): ApiMain->executeAction()
  #14 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(440): ApiMain->executeActionWithErrorHandling()
  #15 /srv/mediawiki/php-1.28.0-wmf.8/api.php(83): ApiMain->execute()
  #16 /srv/mediawiki/w/api.php(3): include(string)
  #17 {main}

Details

Related Gerrit Patches:
mediawiki/extensions/Echo : wmf/1.28.0-wmf.9CentralAuth: Bail if not fully initialized
mediawiki/extensions/Echo : wmf/1.28.0-wmf.8CentralAuth: Bail if not fully initialized
mediawiki/extensions/Echo : masterCentralAuth: Bail if not fully initialized
mediawiki/extensions/Echo : wmf/1.28.0-wmf.9Troubleshoot why Echo is still triggering CA failures
mediawiki/extensions/Echo : wmf/1.28.0-wmf.8Troubleshoot why Echo is still triggering CA failures
mediawiki/extensions/Echo : masterTroubleshoot why Echo is still triggering CA failures

Event Timeline

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJul 12 2016, 6:50 PM

Change 298551 had a related patch set uploaded (by Mattflaschen):
Troubleshoot why Echo is still triggering CA failures

https://gerrit.wikimedia.org/r/298551

Change 298551 merged by jenkins-bot:
Troubleshoot why Echo is still triggering CA failures

https://gerrit.wikimedia.org/r/298551

Change 298553 had a related patch set uploaded (by Mattflaschen):
Troubleshoot why Echo is still triggering CA failures

https://gerrit.wikimedia.org/r/298553

Change 298554 had a related patch set uploaded (by Mattflaschen):
Troubleshoot why Echo is still triggering CA failures

https://gerrit.wikimedia.org/r/298554

Change 298553 merged by jenkins-bot:
Troubleshoot why Echo is still triggering CA failures

https://gerrit.wikimedia.org/r/298553

Change 298554 merged by jenkins-bot:
Troubleshoot why Echo is still triggering CA failures

https://gerrit.wikimedia.org/r/298554

Mentioned in SAL [2016-07-12T19:30:11Z] <mattflaschen@tin> Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T119736: T140144: Troubleshoot why Echo is still triggering CA failures (duration: 00m 39s)

So it does happen even when CentralAuth consistently gives us a central ID.

The failures are not just during autoCreate, but that is the most serious since it causes corruption (missing user row in wiki DB), and still happening.

Mattflaschen-WMF added a comment.EditedJul 12 2016, 7:53 PM

There is also one about "invalid key type". Not sure if that one happens in auto-create yet. (it is an autocreate, but a different path).

2016-07-12 19:49:17 [V4VJugpAIDEAAGRzeuoAAABN] mw1306 mediawikiwiki 1.28.0-wmf.8 Echo DEBUG: Exception when fetching CentralAuth token: wiki: mediawikiwiki, userName: Jtnayem147, userId: 684
7810, centralId: 47182055, exception: exception 'InvalidArgumentException' with message 'Invalid key type: NULL' in /srv/mediawiki/php-1.28.0-wmf.8/includes/utils/MWCryptHash.php:110
Stack trace:
#0 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/CentralAuthUser.php(607): MWCryptHash::hmac()
#1 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/api/ApiCentralAuthToken.php(62): CentralAuthUser->getAuthToken()
#2 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(1373): ApiCentralAuthToken->execute()
#3 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(438): ApiMain->executeAction()
#4 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(57): ApiMain->execute()
#5 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(112): EchoForeignWikiRequest->getCentralAuthToken()
#6 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(92): EchoForeignWikiRequest->getQueryParams()
#7 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(29): EchoForeignWikiRequest->getRequestParams()
#8 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(709): EchoForeignWikiRequest->execute()
#9 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(770): MWEchoNotifUser->getForeignData()
#10 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(496): MWEchoNotifUser->getForeignTimestamp()
#11 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/Hooks.php(1150): MWEchoNotifUser->resetNotificationCount()
#12 /srv/mediawiki/php-1.28.0-wmf.8/includes/Hooks.php(195): EchoHooks::onUserSaveSettings()
#13 /srv/mediawiki/php-1.28.0-wmf.8/includes/user/User.php(3952): Hooks::run()
#14 /srv/mediawiki/php-1.28.0-wmf.8/includes/auth/AuthManager.php(1678): User->saveSettings()
#15 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/CentralAuthUtils.php(141): MediaWiki\Auth\AuthManager->autoCreateUser()
#16 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/CreateLocalAccountJob.php(56): CentralAuthUtils::autoCreateUser()
#17 /srv/mediawiki/php-1.28.0-wmf.8/includes/jobqueue/JobRunner.php(265): CentralAuthCreateLocalAccountJob->run()
#18 /srv/mediawiki/php-1.28.0-wmf.8/includes/jobqueue/JobRunner.php(179): JobRunner->executeJob()
#19 /srv/mediawiki/rpc/RunJobs.php(47): JobRunner->run()
#20 {main} {"wiki":"mediawikiwiki","userName":"Jtnayem147","userId":6847810,"centralId":47182055} 
[Exception InvalidArgumentException] (/srv/mediawiki/php-1.28.0-wmf.8/includes/utils/MWCryptHash.php:110) Invalid key type: NULL
  #0 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/CentralAuthUser.php(607): MWCryptHash::hmac(string, NULL, boolean)
  #1 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/api/ApiCentralAuthToken.php(62): CentralAuthUser->getAuthToken()
  #2 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(1373): ApiCentralAuthToken->execute()
  #3 /srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php(438): ApiMain->executeAction()
  #4 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(57): ApiMain->execute()
  #5 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(112): EchoForeignWikiRequest->getCentralAuthToken(User)
  #6 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(92): EchoForeignWikiRequest->getQueryParams(string)
  #7 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php(29): EchoForeignWikiRequest->getRequestParams()
  #8 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(709): EchoForeignWikiRequest->execute()
  #9 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(770): MWEchoNotifUser->getForeignData()
  #10 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/includes/NotifUser.php(496): MWEchoNotifUser->getForeignTimestamp(string)
  #11 /srv/mediawiki/php-1.28.0-wmf.8/extensions/Echo/Hooks.php(1150): MWEchoNotifUser->resetNotificationCount()
  #12 /srv/mediawiki/php-1.28.0-wmf.8/includes/Hooks.php(195): EchoHooks::onUserSaveSettings(User)
  #13 /srv/mediawiki/php-1.28.0-wmf.8/includes/user/User.php(3952): Hooks::run(string, array)
  #14 /srv/mediawiki/php-1.28.0-wmf.8/includes/auth/AuthManager.php(1678): User->saveSettings()
  #15 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/CentralAuthUtils.php(141): MediaWiki\Auth\AuthManager->autoCreateUser(User, string, boolean)
  #16 /srv/mediawiki/php-1.28.0-wmf.8/extensions/CentralAuth/includes/CreateLocalAccountJob.php(56): CentralAuthUtils::autoCreateUser(User)
  #17 /srv/mediawiki/php-1.28.0-wmf.8/includes/jobqueue/JobRunner.php(265): CentralAuthCreateLocalAccountJob->run()
  #18 /srv/mediawiki/php-1.28.0-wmf.8/includes/jobqueue/JobRunner.php(179): JobRunner->executeJob(CentralAuthCreateLocalAccountJob, BufferingStatsdDataFactory, integer)
  #19 /srv/mediawiki/rpc/RunJobs.php(47): JobRunner->run(array)
  #20 {main}

Change 298569 had a related patch set uploaded (by Mattflaschen):
CentralAuth: Bail if not fully initialized

https://gerrit.wikimedia.org/r/298569

Change 298569 merged by jenkins-bot:
CentralAuth: Bail if not fully initialized

https://gerrit.wikimedia.org/r/298569

Change 298628 had a related patch set uploaded (by Mattflaschen):
CentralAuth: Bail if not fully initialized

https://gerrit.wikimedia.org/r/298628

Michgrig removed a subscriber: Michgrig.Jul 12 2016, 9:04 PM

Change 298634 had a related patch set uploaded (by Mattflaschen):
CentralAuth: Bail if not fully initialized

https://gerrit.wikimedia.org/r/298634

Change 298628 merged by jenkins-bot:
CentralAuth: Bail if not fully initialized

https://gerrit.wikimedia.org/r/298628

Change 298634 merged by jenkins-bot:
CentralAuth: Bail if not fully initialized

https://gerrit.wikimedia.org/r/298634

Mentioned in SAL [2016-07-12T21:24:21Z] <mattflaschen@tin> Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T140144: Echo/CentralAuth: Bail if not fully initialized (duration: 00m 49s)

There haven't been any more badsession errors since this was deployed, so I'm declaring it fixed.

T140156: CentralAuth 'Invalid key type: NULL' remains an issue, but I think that is on the CentralAuth side.

Also, it now catches the exception. So if it still fails to get a token, it should lead to "Failed to fetch API response", but not an uncaught exception or DB inconsistency.

jayvdb added a subscriber: jayvdb.Jul 13 2016, 12:23 AM

Overview of logs for the Echo part: P3422

Keep in mind the dates of the files are the date of log rotation, not necessarily date of occurrence.

Mattflaschen-WMF added a comment.EditedJul 13 2016, 8:01 PM

@Legoktm is running checkLocalUser.php now, which should fix any users from the last stages of when the Echo bug was present. Current run started 2016-07-13 18:22 UTC.

greg added a comment.Jul 14 2016, 6:24 AM

There haven't been any more badsession errors since this was deployed, so I'm declaring it fixed.

Shall we close this then? Or are you waiting for (second) the checkLocaluser.php run to finish?

T140156: CentralAuth 'Invalid key type: NULL' remains an issue, but I think that is on the CentralAuth side.

Do you/does anyone have an idea of how prevalent it is?

There haven't been any more badsession errors since this was deployed, so I'm declaring it fixed.

Shall we close this then? Or are you waiting for (second) the checkLocaluser.php run to finish?

This is just waiting on QA now.

Tested betalabs with the scenarios that supposedly trigger the error:
(1)

  • a newly created user goes to another wiki (not the wiki where the account was created)
  • an old user goes to a never before visited wikis
  • testing when different (tabs, windows, browsers) are open for the same account and some updated in message count/status happen only in one tab.

(2) Checked betalabs logs with @Mattflaschen-WMF grep
zgrep -c 'Can only obtain a centralauthtoken when using CentralAuth sessions' /a/mw-log/archive/exception.log-20160*
The warning was recorded between 2016-05-01 and 2016-07-11 (not later)

greg lowered the priority of this task from Unbreak Now! to High.Jul 15 2016, 3:26 PM

There haven't been any more badsession errors since this was deployed, so I'm declaring it fixed.

Shall we close this then? Or are you waiting for (second) the checkLocaluser.php run to finish?

This is just waiting on QA now.

Per ^ reducing priority (just because it's in the blocker chain for wmf.10).

jmatazzoni closed this task as Resolved.Jul 20 2016, 9:00 PM