Page MenuHomePhabricator
Create Task
Maniphest T317606

PHP Notice: Undefined index: asOfTime
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   PHP Notice: Undefined index: asOfTime
exception.trace
from /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/position/MySQLPrimaryPos.php(345)
#0 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/position/MySQLPrimaryPos.php(345): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 [internal function]: Wikimedia\Rdbms\MySQLPrimaryPos->__unserialize(array)
#2 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/objectcache/APCUBagOStuff.php(74): apcu_fetch(string)
#3 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/objectcache/MediumSpecificBagOStuff.php(117): APCUBagOStuff->doGet(string, integer)
#4 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(873): MediumSpecificBagOStuff->get(string)
#5 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(573): Wikimedia\Rdbms\LoadBalancer->doWait(integer)
#6 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(517): Wikimedia\Rdbms\LoadBalancer->getReaderIndex(string, string)
#7 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(942): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(integer, array, string)
#8 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/DBConnRef.php(95): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#9 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/DBConnRef.php(101): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#10 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/DBConnRef.php(343): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#11 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(688): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#12 /srv/mediawiki/php-1.39.0-wmf.28/includes/user/UserSelectQueryBuilder.php(191): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#13 /srv/mediawiki/php-1.39.0-wmf.28/includes/user/ActorStore.php(237): MediaWiki\User\UserSelectQueryBuilder->fetchUserIdentity()
#14 /srv/mediawiki/php-1.39.0-wmf.28/includes/user/User.php(937): MediaWiki\User\ActorStore->getUserIdentityByName(string, integer)
#15 /srv/mediawiki/php-1.39.0-wmf.28/extensions/CentralAuth/includes/User/CentralAuthUser.php(2165): User::idFromName(string)
#16 /srv/mediawiki/php-1.39.0-wmf.28/extensions/CentralAuth/includes/User/CentralAuthUser.php(2245): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->canAuthenticate()
#17 /srv/mediawiki/php-1.39.0-wmf.28/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php(212): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->authenticateWithToken(string)
#18 /srv/mediawiki/php-1.39.0-wmf.28/includes/session/SessionManager.php(537): CentralAuthSessionProvider->provideSessionInfo(WebRequest)
#19 /srv/mediawiki/php-1.39.0-wmf.28/includes/session/SessionManager.php(243): MediaWiki\Session\SessionManager->getSessionInfoForRequest(WebRequest)
#20 /srv/mediawiki/php-1.39.0-wmf.28/includes/WebRequest.php(843): MediaWiki\Session\SessionManager->getSessionForRequest(WebRequest)
#21 /srv/mediawiki/php-1.39.0-wmf.28/includes/session/SessionManager.php(164): WebRequest->getSession()
#22 /srv/mediawiki/php-1.39.0-wmf.28/includes/Setup.php(448): MediaWiki\Session\SessionManager::getGlobalSession()
#23 /srv/mediawiki/php-1.39.0-wmf.28/includes/WebStart.php(86): require_once(string)
#24 /srv/mediawiki/php-1.39.0-wmf.28/api.php(43): require(string)
#25 /srv/mediawiki/w/api.php(3): require(string)
#26 {main}
Impact
Notes

Details

Request URL
https://ja.wiktionary.org/w/api.php?action=*
SubjectRepoBranchLines +/-
rdbms: Use plain array to store position datamediawiki/corewmf/1.40.0-wmf.1+13 -3
rdbms: Use plain array to store position datamediawiki/coremaster+13 -3
rdbms: Bump ChronologyProtector cache key versionmediawiki/coremaster+1 -1
rdbms: Bump ChronologyProtector cache key versionmediawiki/corewmf/1.40.0-wmf.1+1 -1
rdbms: Use JSON to store ChronologyProtector position datamediawiki/corewmf/1.39.0-wmf.28+28 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedReleasedancyT314190 1.40.0-wmf.1 deployment blockers
 ResolvedPRODUCTION ERRORZabeT317606 PHP Notice: Undefined index: asOfTime

Event Timeline

taavi triaged this task as Unbreak Now! priority.Sep 13 2022, 7:09 AM
taavi created this task.
Restricted Application added a project: Performance-Team. · View Herald TranscriptSep 13 2022, 7:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
taavi added subscribers: Krinkle, Ladsgroup.Sep 13 2022, 7:10 AM

@Krinkle @Ladsgroup Hey. wmf.1 includes this change https://gerrit.wikimedia.org/r/c/mediawiki/core/+/829316 which looks related - could you have a look?

RhinosF1 subscribed.Sep 13 2022, 7:11 AM
Ladsgroup added a comment.Sep 13 2022, 7:13 AM

I just pinged Krinkle in IRC. Probably that needs a backport, the errors are happening on wmf.28

Ladsgroup added a comment.Sep 13 2022, 7:14 AM

image.png (201×187 px, 7 KB)

Ladsgroup added a comment.Sep 13 2022, 7:17 AM

We probably need to backport this: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/828058

hashar subscribed.Sep 13 2022, 8:16 AM

Picking up a single request, the sequence of errors is:

PHP Notice: Undefined index: asOfTime
PHP Notice: Undefined index: position
InvalidArgumentException: GTID set cannot be empty.

The other messages stacktraces are:

GTID set cannot be empty.
from /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/position/MySQLPrimaryPos.php(94)
#0 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/position/MySQLPrimaryPos.php(345): Wikimedia\Rdbms\MySQLPrimaryPos->init(NULL, NULL)
#1 [internal function]: Wikimedia\Rdbms\MySQLPrimaryPos->__unserialize(array)
#2 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/objectcache/APCUBagOStuff.php(74): apcu_fetch(string)
#3 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/objectcache/MediumSpecificBagOStuff.php(117): APCUBagOStuff->doGet(string, integer)
#4 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(873): MediumSpecificBagOStuff->get(string)
#5 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(573): Wikimedia\Rdbms\LoadBalancer->doWait(integer)
#6 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(517): Wikimedia\Rdbms\LoadBalancer->getReaderIndex(string, string)
#7 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(942): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(integer, array, string)
#8 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/DBConnRef.php(95): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#9 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/DBConnRef.php(101): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#10 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/DBConnRef.php(343): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#11 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(688): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#12 /srv/mediawiki/php-1.39.0-wmf.28/includes/user/UserSelectQueryBuilder.php(191): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#13 /srv/mediawiki/php-1.39.0-wmf.28/includes/user/ActorStore.php(237): MediaWiki\User\UserSelectQueryBuilder->fetchUserIdentity()
#14 /srv/mediawiki/php-1.39.0-wmf.28/includes/user/CentralId/CentralIdLookup.php(270): MediaWiki\User\ActorStore->getUserIdentityByName(string)
#15 /srv/mediawiki/php-1.39.0-wmf.28/extensions/OAuth/src/Backend/Utils.php(314): CentralIdLookup->localUserFromCentralId(integer)
#16 /srv/mediawiki/php-1.39.0-wmf.28/extensions/OAuth/src/SessionProvider.php(159): MediaWiki\Extension\OAuth\Backend\Utils::getLocalUserFromCentralId(integer)
#17 /srv/mediawiki/php-1.39.0-wmf.28/includes/session/SessionManager.php(537): MediaWiki\Extension\OAuth\SessionProvider->provideSessionInfo(WebRequest)
#18 /srv/mediawiki/php-1.39.0-wmf.28/includes/session/SessionManager.php(243): MediaWiki\Session\SessionManager->getSessionInfoForRequest(WebRequest)
#19 /srv/mediawiki/php-1.39.0-wmf.28/includes/WebRequest.php(843): MediaWiki\Session\SessionManager->getSessionForRequest(WebRequest)
#20 /srv/mediawiki/php-1.39.0-wmf.28/includes/session/SessionManager.php(164): WebRequest->getSession()
#21 /srv/mediawiki/php-1.39.0-wmf.28/includes/Setup.php(448): MediaWiki\Session\SessionManager::getGlobalSession()
#22 /srv/mediawiki/php-1.39.0-wmf.28/includes/WebStart.php(86): require_once(string)
#23 /srv/mediawiki/php-1.39.0-wmf.28/api.php(43): require(string)
#24 /srv/mediawiki/w/api.php(3): require(string)
#25 {main}
PHP Notice: Undefined index: position
from /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/position/MySQLPrimaryPos.php(345)
#0 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/position/MySQLPrimaryPos.php(345): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 [internal function]: Wikimedia\Rdbms\MySQLPrimaryPos->__unserialize(array)
#2 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/objectcache/APCUBagOStuff.php(74): apcu_fetch(string)
#3 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/objectcache/MediumSpecificBagOStuff.php(117): APCUBagOStuff->doGet(string, integer)
#4 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(873): MediumSpecificBagOStuff->get(string)
#5 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(573): Wikimedia\Rdbms\LoadBalancer->doWait(integer)
#6 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(517): Wikimedia\Rdbms\LoadBalancer->getReaderIndex(string, string)
#7 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/loadbalancer/LoadBalancer.php(942): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(integer, array, string)
#8 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/DBConnRef.php(95): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#9 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/DBConnRef.php(101): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#10 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/database/DBConnRef.php(343): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#11 /srv/mediawiki/php-1.39.0-wmf.28/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(688): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#12 /srv/mediawiki/php-1.39.0-wmf.28/includes/user/UserSelectQueryBuilder.php(191): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#13 /srv/mediawiki/php-1.39.0-wmf.28/includes/user/ActorStore.php(237): MediaWiki\User\UserSelectQueryBuilder->fetchUserIdentity()
#14 /srv/mediawiki/php-1.39.0-wmf.28/includes/user/CentralId/CentralIdLookup.php(270): MediaWiki\User\ActorStore->getUserIdentityByName(string)
#15 /srv/mediawiki/php-1.39.0-wmf.28/extensions/OAuth/src/Backend/Utils.php(314): CentralIdLookup->localUserFromCentralId(integer)
#16 /srv/mediawiki/php-1.39.0-wmf.28/extensions/OAuth/src/SessionProvider.php(159): MediaWiki\Extension\OAuth\Backend\Utils::getLocalUserFromCentralId(integer)
#17 /srv/mediawiki/php-1.39.0-wmf.28/includes/session/SessionManager.php(537): MediaWiki\Extension\OAuth\SessionProvider->provideSessionInfo(WebRequest)
#18 /srv/mediawiki/php-1.39.0-wmf.28/includes/session/SessionManager.php(243): MediaWiki\Session\SessionManager->getSessionInfoForRequest(WebRequest)
#19 /srv/mediawiki/php-1.39.0-wmf.28/includes/WebRequest.php(843): MediaWiki\Session\SessionManager->getSessionForRequest(WebRequest)
#20 /srv/mediawiki/php-1.39.0-wmf.28/includes/session/SessionManager.php(164): WebRequest->getSession()
#21 /srv/mediawiki/php-1.39.0-wmf.28/includes/Setup.php(448): MediaWiki\Session\SessionManager::getGlobalSession()
#22 /srv/mediawiki/php-1.39.0-wmf.28/includes/WebStart.php(86): require_once(string)
#23 /srv/mediawiki/php-1.39.0-wmf.28/api.php(43): require(string)
#24 /srv/mediawiki/w/api.php(3): require(string)
#25 {main}
gerritbot added a comment.Sep 13 2022, 8:39 AM

Change 831563 had a related patch set uploaded (by Ladsgroup; author: Daniel Kinzler):

[mediawiki/core@wmf/1.39.0-wmf.28] rdbms: Use JSON to store ChronologyProtector position data

https://gerrit.wikimedia.org/r/831563

gerritbot added a project: Patch-For-Review.Sep 13 2022, 8:39 AM
hashar added a comment.EditedSep 13 2022, 8:39 AM

From the first few errors:

03:38:02.321 itwiki PHP Warning: Erroneous data format for unserializing 'Wikimedia\Rdbms\MySQLPrimaryPos'

03:38:04.580 testwiki PHP Warning: Class Wikimedia\Rdbms\MySQLPrimaryPos has no unserializer

03:38:04.971 testwiki (on another server) PHP Warning: Class Wikimedia\Rdbms\MySQLPrimaryPos has no unserializer

03:38:05.142 simplewiki PHP Notice: apcu_fetch(): Error at offset 42 of 856 bytes
03:38:05.142 simplewiki PHP Warning: Erroneous data format for unserializing 'Wikimedia\Rdbms\MySQLPrimaryPos'

03:38:05.528 pawiki PHP Notice: apcu_fetch(): Error at offset 42 of 856 bytes
03:38:05.527 pawiki PHP Warning: Erroneous data format for unserializing 'Wikimedia\Rdbms\MySQLPrimaryPos'

I don't know much more, this blocker would need someone actually understanding the serialization/json data.

gerritbot added a comment.Sep 13 2022, 8:54 AM

Change 831563 abandoned by Ladsgroup:

[mediawiki/core@wmf/1.39.0-wmf.28] rdbms: Use JSON to store ChronologyProtector position data

Reason:

https://gerrit.wikimedia.org/r/831563

Maintenance_bot removed a project: Patch-For-Review.Sep 13 2022, 9:30 AM
gerritbot added a comment.Sep 13 2022, 10:20 AM

Change 831834 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] rdbms: Bump ChronologyProtector cache key version

https://gerrit.wikimedia.org/r/831834

gerritbot added a project: Patch-For-Review.Sep 13 2022, 10:20 AM
gerritbot added a comment.Sep 13 2022, 10:26 AM

Change 831847 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@wmf/1.40.0-wmf.1] rdbms: Bump ChronologyProtector cache key version

https://gerrit.wikimedia.org/r/831847

Krinkle added a comment.Sep 13 2022, 10:27 AM

There was an unforeseen change in data format. The above bumps the cache key version. Backporting it to the new wmf.1 branch only should suffice to stop producing data that wmf.28 can't read. The data is very short lived (less than a minute) so the errors should dry up very quickly after that.

gerritbot added a comment.Sep 13 2022, 11:03 AM

Change 831847 merged by jenkins-bot:

[mediawiki/core@wmf/1.40.0-wmf.1] rdbms: Bump ChronologyProtector cache key version

https://gerrit.wikimedia.org/r/831847

gerritbot added a comment.Sep 13 2022, 11:04 AM

Change 831834 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Bump ChronologyProtector cache key version

https://gerrit.wikimedia.org/r/831834

Ladsgroup closed this task as Resolved.Sep 13 2022, 11:06 AM
Ladsgroup assigned this task to Krinkle.

I'm closing this as it should be fixed but we can't 100% be sure unless we deploy wmf.1. Reopen if that's not the case.

Stashbot added a comment.Sep 13 2022, 11:09 AM

Mentioned in SAL (#wikimedia-operations) [2022-09-13T11:09:32Z] <ladsgroup@deploy1002> Synchronized php-1.40.0-wmf.1/includes/libs/rdbms/ChronologyProtector.php: Backport: [[gerrit:831847|rdbms: Bump ChronologyProtector cache key version (T317606)]] (duration: 03m 49s)

Maintenance_bot removed a project: Patch-For-Review.Sep 13 2022, 11:30 AM
ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.1; 2022-09-12).Sep 13 2022, 12:00 PM
Zabe subscribed.Sep 13 2022, 3:11 PM
dancy reopened this task as Open.Sep 13 2022, 3:20 PM
dancy subscribed.

Reopening because I saw very similar errors when I tried rolling forward to testwikis after the prior fix attempt.

gerritbot added a comment.Sep 13 2022, 3:25 PM

Change 831907 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/core@master] loadbalancer: Bump 'last-known-pos' cache key version

https://gerrit.wikimedia.org/r/831907

gerritbot added a project: Patch-For-Review.Sep 13 2022, 3:25 PM
Zabe merged a task: T317670: Fatal exception when browsing incubator.wikimedia.org.Sep 13 2022, 4:51 PM
Zabe added a subscriber: jhsoby.
aaron moved this task from Inbox, needs triage to Doing: Prio Interrupt on the Performance-Team board.Sep 13 2022, 6:12 PM
Krinkle mentioned this in T317625: DBReplication warning: expected but failed to find position index {cpPosIndex}.Sep 14 2022, 9:05 AM
gerritbot added a comment.Sep 14 2022, 2:59 PM

Change 831907 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Use plain array to store position data

https://gerrit.wikimedia.org/r/831907

ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.2; 2022-09-19); removed MW-1.40-notes (1.40.0-wmf.1; 2022-09-12).Sep 14 2022, 3:00 PM
gerritbot added a comment.Sep 14 2022, 3:38 PM

Change 831985 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/core@wmf/1.40.0-wmf.1] rdbms: Use plain array to store position data

https://gerrit.wikimedia.org/r/831985

gerritbot added a comment.Sep 14 2022, 6:28 PM

Change 831985 merged by jenkins-bot:

[mediawiki/core@wmf/1.40.0-wmf.1] rdbms: Use plain array to store position data

https://gerrit.wikimedia.org/r/831985

ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.1; 2022-09-12); removed MW-1.40-notes (1.40.0-wmf.2; 2022-09-19).Sep 14 2022, 7:00 PM
valerio.bozzolan subscribed.Sep 14 2022, 7:11 PM
dancy added a comment.Sep 14 2022, 9:03 PM

Thanks for the fix @Zabe !

Zabe closed this task as Resolved.Sep 15 2022, 12:28 AM
Zabe claimed this task.
Krinkle merged a task: T317625: DBReplication warning: expected but failed to find position index {cpPosIndex}.Sep 18 2022, 8:23 PM
Krinkle added subscribers: jcrespo, aaron.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL