==== Error ====
* mwversion: `1.39.0-wmf.26`
* reqId: `0621613c-626b-468b-b7d2-d153ddf3ead9`
* [[ https://logstash.wikimedia.org/app/dashboards#/view/AXFV7JE83bOlOASGccsT?_g=(time:(from:'2022-08-28T22:58:52.000Z',to:'2022-08-29T21:01:38.898Z'))&_a=(query:(query_string:(query:'reqId:%220621613c-626b-468b-b7d2-d153ddf3ead9%22'))) | Find reqId in Logstash ]]
```name=normalized_message
[{reqId}] {exception_url} PHP Warning: Erroneous data format for unserializing 'Wikimedia\Rdbms\MySQLPrimaryPos'
```
```name=exception.trace,lines=10
from /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/objectcache/MemcachedPeclBagOStuff.php(196)
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/objectcache/MemcachedPeclBagOStuff.php(196): Memcached->get(string)
#2 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/objectcache/MediumSpecificBagOStuff.php(117): MemcachedPeclBagOStuff->doGet(string, integer)
#3 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/ChronologyProtector.php(485): MediumSpecificBagOStuff->get(string)
#4 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/ChronologyProtector.php(405): Wikimedia\Rdbms\ChronologyProtector->lazyStartup()
#5 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/ChronologyProtector.php(266): Wikimedia\Rdbms\ChronologyProtector->getStartupSessionPositions()
#6 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/lbfactory/LBFactory.php(771): Wikimedia\Rdbms\ChronologyProtector->applySessionReplicationPosition(Wikimedia\Rdbms\LoadBalancer)
#7 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1452): Wikimedia\Rdbms\LBFactory->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer)
#8 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/loadbalancer/LoadBalancer.php(561): Wikimedia\Rdbms\LoadBalancer->lazyLoadReplicationPositions()
#9 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/loadbalancer/LoadBalancer.php(514): Wikimedia\Rdbms\LoadBalancer->getReaderIndex(string, string)
#10 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/loadbalancer/LoadBalancer.php(939): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(integer, array, string)
#11 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/database/DBConnRef.php(95): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#12 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/database/DBConnRef.php(101): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#13 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/database/DBConnRef.php(351): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#14 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(686): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#15 /srv/mediawiki/php-1.39.0-wmf.26/includes/user/UserSelectQueryBuilder.php(191): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#16 /srv/mediawiki/php-1.39.0-wmf.26/includes/user/ActorStore.php(237): MediaWiki\User\UserSelectQueryBuilder->fetchUserIdentity()
#17 /srv/mediawiki/php-1.39.0-wmf.26/includes/user/User.php(937): MediaWiki\User\ActorStore->getUserIdentityByName(string, integer)
#18 /srv/mediawiki/php-1.39.0-wmf.26/extensions/CentralAuth/includes/User/CentralAuthUser.php(2165): User::idFromName(string)
#19 /srv/mediawiki/php-1.39.0-wmf.26/extensions/CentralAuth/includes/User/CentralAuthUser.php(2245): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->canAuthenticate()
#20 /srv/mediawiki/php-1.39.0-wmf.26/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php(212): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->authenticateWithToken(string)
#21 /srv/mediawiki/php-1.39.0-wmf.26/includes/session/SessionManager.php(537): CentralAuthSessionProvider->provideSessionInfo(WebRequest)
#22 /srv/mediawiki/php-1.39.0-wmf.26/includes/session/SessionManager.php(243): MediaWiki\Session\SessionManager->getSessionInfoForRequest(WebRequest)
#23 /srv/mediawiki/php-1.39.0-wmf.26/includes/WebRequest.php(843): MediaWiki\Session\SessionManager->getSessionForRequest(WebRequest)
#24 /srv/mediawiki/php-1.39.0-wmf.26/includes/session/SessionManager.php(164): WebRequest->getSession()
#25 /srv/mediawiki/php-1.39.0-wmf.26/includes/Setup.php(448): MediaWiki\Session\SessionManager::getGlobalSession()
#26 /srv/mediawiki/php-1.39.0-wmf.26/includes/WebStart.php(86): require_once(string)
#27 /srv/mediawiki/php-1.39.0-wmf.26/api.php(43): require(string)
#28 /srv/mediawiki/w/api.php(3): require(string)
#29 {main}
```
Immediately followed by `PHP Warning: Memcached::get(): could not unserialize value`
```
from /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/objectcache/MemcachedPeclBagOStuff.php(196)
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/objectcache/MemcachedPeclBagOStuff.php(196): Memcached->get(string)
#2 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/objectcache/MediumSpecificBagOStuff.php(117): MemcachedPeclBagOStuff->doGet(string, integer)
#3 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/ChronologyProtector.php(485): MediumSpecificBagOStuff->get(string)
#4 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/ChronologyProtector.php(405): Wikimedia\Rdbms\ChronologyProtector->lazyStartup()
#5 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/ChronologyProtector.php(266): Wikimedia\Rdbms\ChronologyProtector->getStartupSessionPositions()
#6 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/lbfactory/LBFactory.php(771): Wikimedia\Rdbms\ChronologyProtector->applySessionReplicationPosition(Wikimedia\Rdbms\LoadBalancer)
#7 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1452): Wikimedia\Rdbms\LBFactory->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer)
#8 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/loadbalancer/LoadBalancer.php(561): Wikimedia\Rdbms\LoadBalancer->lazyLoadReplicationPositions()
#9 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/loadbalancer/LoadBalancer.php(514): Wikimedia\Rdbms\LoadBalancer->getReaderIndex(string, string)
#10 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/loadbalancer/LoadBalancer.php(939): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(integer, array, string)
#11 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/database/DBConnRef.php(95): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#12 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/database/DBConnRef.php(101): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#13 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/database/DBConnRef.php(351): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#14 /srv/mediawiki/php-1.39.0-wmf.26/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(686): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#15 /srv/mediawiki/php-1.39.0-wmf.26/includes/user/UserSelectQueryBuilder.php(191): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#16 /srv/mediawiki/php-1.39.0-wmf.26/includes/user/ActorStore.php(237): MediaWiki\User\UserSelectQueryBuilder->fetchUserIdentity()
#17 /srv/mediawiki/php-1.39.0-wmf.26/includes/user/User.php(937): MediaWiki\User\ActorStore->getUserIdentityByName(string, integer)
#18 /srv/mediawiki/php-1.39.0-wmf.26/extensions/CentralAuth/includes/User/CentralAuthUser.php(2165): User::idFromName(string)
#19 /srv/mediawiki/php-1.39.0-wmf.26/extensions/CentralAuth/includes/User/CentralAuthUser.php(2245): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->canAuthenticate()
#20 /srv/mediawiki/php-1.39.0-wmf.26/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php(212): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->authenticateWithToken(string)
#21 /srv/mediawiki/php-1.39.0-wmf.26/includes/session/SessionManager.php(537): CentralAuthSessionProvider->provideSessionInfo(WebRequest)
#22 /srv/mediawiki/php-1.39.0-wmf.26/includes/session/SessionManager.php(243): MediaWiki\Session\SessionManager->getSessionInfoForRequest(WebRequest)
#23 /srv/mediawiki/php-1.39.0-wmf.26/includes/WebRequest.php(843): MediaWiki\Session\SessionManager->getSessionForRequest(WebRequest)
#24 /srv/mediawiki/php-1.39.0-wmf.26/includes/session/SessionManager.php(164): WebRequest->getSession()
#25 /srv/mediawiki/php-1.39.0-wmf.26/includes/Setup.php(448): MediaWiki\Session\SessionManager::getGlobalSession()
#26 /srv/mediawiki/php-1.39.0-wmf.26/includes/WebStart.php(86): require_once(string)
#27 /srv/mediawiki/php-1.39.0-wmf.26/api.php(43): require(string)
#28 /srv/mediawiki/w/api.php(3): require(string)
#29 {main}
```
==== Impact ====
Central authentication might be broken?
==== Notes ====
30k such messages since August 29th 13:20 UTC. The earliest were at 8:30, some others during the morning but since 13:20 it is a constant stream. Messages matching `PHP Warning: Erroneous data format for unserializing 'Wikimedia\Rdbms\MySQLPrimaryPos'`since 7:00 UTC:
{F35495333 size=full}
From the https://grafana.wikimedia.org/d/000000102/production-logging dashboard:
{F35495401 size=full}
```
name=production sal,lang=irc
13:21 <taavi@deploy1002> Synchronized php-1.39.0-wmf.26/extensions/SecurePoll/: T316150 (duration: 03m 44s)
13:14 <oblivian@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Moving 1% of users to php 7.4 (duration: 04m 18s)
13:12 <vgutierrez> Increase roll-out of query-sorting to 75% - T314868
```