Page MenuHomePhabricator

Dispatching broken on beta - Fatal error: Class 'Memcached' not found in ObjectCache.php on line 186
Closed, ResolvedPublic

Description

Resolving T237551 has immediately revealed another issue with dispatching on beta that seems to have started happening in the last 22-24 hours.

https://wikidata.beta.wmflabs.org/wiki/Special:DispatchStats

Looking at the logs for dispatching (method of which these can be seen below)

ssh deployment-mwmaint01.deployment-prep.eqiad.wmflabs
addshore@deployment-mwmaint01:/var/log/wikidata$ tail -f dispatchChanges-wikidatawiki.log
20736: Starting dispatcher
Fatal error: Class 'Memcached' not found in /srv/mediawiki/php-master/includes/objectcache/ObjectCache.php on line 186
Fatal error: Uncaught TypeError: Argument 1 passed to MWExceptionHandler::prettyPrintTrace() must be of the type array, string given, called in /srv/mediawiki/php-master/includes/debug/logger/monolog/LineFormatter.php on line 153 and defined in /srv/mediawiki/php-master/includes/exception/MWExceptionHandler.php:364
Stack trace:
#0 /srv/mediawiki/php-master/includes/debug/logger/monolog/LineFormatter.php(153): MWExceptionHandler::prettyPrintTrace('#0 [internal fu...', '  ')
#1 /srv/mediawiki/php-master/includes/debug/logger/monolog/LineFormatter.php(83): MediaWiki\Logger\Monolog\LineFormatter->normalizeExceptionArray(Array)
#2 /srv/mediawiki/php-master/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php(37): MediaWiki\Logger\Monolog\LineFormatter->format(Array)
#3 /srv/mediawiki/php-master/vendor/monolog/monolog/src/Monolog/Logger.php(344): Monolog\Handler\AbstractProcessingHandler->handle(Array)
#4 /srv/mediawiki/php-master/vendor/monolog/monolog/src/Monolog/Logger.php(707): Monolog\Logger->addRecord( in /srv/mediawiki/php-master/includes/exception/MWExceptionHandler.php on line 364
20736: Dispatcher exited with 255

Which indicates a problem @ https://github.com/wikimedia/mediawiki/blob/master/includes/objectcache/ObjectCache.php#L186
Which hasn't been tocuhed in 9 years.

Event Timeline

Addshore created this task.Dec 5 2019, 10:32 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2019, 10:32 AM
Addshore triaged this task as Medium priority.Dec 5 2019, 10:33 AM
Addshore updated the task description. (Show Details)Dec 7 2019, 4:16 PM

This is a big misconfiguration. ObjactCache in line 186 dynamically tries to load the cache class defined in "class" configuration (grrr). They all changed for the mcrouter stuff. I'm trying to see if I can fix it.

It's because the mwmaint01 is dirty...

Mentioned in SAL (#wikimedia-releng) [2019-12-09T17:51:01Z] <Amir1> ladsgroup@deployment-mwmaint01:/srv/mediawiki-staging$ sudo apt-get install php-memcached (T239894)

nope, memcached php extension wasn't installed there. Then it was lack of geoip and msgpack php extensions.

Fixing that in puppet should not be super hard even though it's not possible currently, since beta cluster and production puppet has diverged (I can cherry-pick the patch to beta puppet master if needed). But the dispatching should work now.

ladsgroup@deployment-mwmaint01:/var/log/wikidata$ tail dispatchChanges-wikidatawiki.log --lines 50
12887: Starting dispatcher
20:45:02 Starting loop for unlimited passes or 540 seconds
20:45:05 Posted 6 changes to enwiki, up to ID 1681591, timestamp 20191209130228. Lag is 27757 seconds. Next ID is 1681747.
20:45:08 Posted 2 changes to dewiki, up to ID 1680869, timestamp 20191204174336. Lag is 442892 seconds. Next ID is 1681747.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptDec 9 2019, 8:54 PM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Dec 9 2019, 9:15 PM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.

What do we think about the pile of these in the log:

21:31:02 Wikibase\Repo\Store\Sql\SqlChangeDispatchCoordinator::selectClient: Could not lock any of the candidate client wikis for dispatching

Are they a problem?

What do we think about the pile of these in the log:

21:31:02 Wikibase\Repo\Store\Sql\SqlChangeDispatchCoordinator::selectClient: Could not lock any of the candidate client wikis for dispatching

Are they a problem?

no it means, there is nothing to dispatch ^_^ Maybe we should reduce the dispacthing speed for beta.