Page MenuHomePhabricator
Create Task
Maniphest T239894

Dispatching broken on beta - Fatal error: Class 'Memcached' not found in ObjectCache.php on line 186
Closed, ResolvedPublic
Actions

Description

Resolving T237551 has immediately revealed another issue with dispatching on beta that seems to have started happening in the last 22-24 hours.

https://wikidata.beta.wmflabs.org/wiki/Special:DispatchStats

Looking at the logs for dispatching (method of which these can be seen below)

ssh deployment-mwmaint01.deployment-prep.eqiad.wmflabs
addshore@deployment-mwmaint01:/var/log/wikidata$ tail -f dispatchChanges-wikidatawiki.log
20736: Starting dispatcher
Fatal error: Class 'Memcached' not found in /srv/mediawiki/php-master/includes/objectcache/ObjectCache.php on line 186
Fatal error: Uncaught TypeError: Argument 1 passed to MWExceptionHandler::prettyPrintTrace() must be of the type array, string given, called in /srv/mediawiki/php-master/includes/debug/logger/monolog/LineFormatter.php on line 153 and defined in /srv/mediawiki/php-master/includes/exception/MWExceptionHandler.php:364
Stack trace:
#0 /srv/mediawiki/php-master/includes/debug/logger/monolog/LineFormatter.php(153): MWExceptionHandler::prettyPrintTrace('#0 [internal fu...', '  ')
#1 /srv/mediawiki/php-master/includes/debug/logger/monolog/LineFormatter.php(83): MediaWiki\Logger\Monolog\LineFormatter->normalizeExceptionArray(Array)
#2 /srv/mediawiki/php-master/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php(37): MediaWiki\Logger\Monolog\LineFormatter->format(Array)
#3 /srv/mediawiki/php-master/vendor/monolog/monolog/src/Monolog/Logger.php(344): Monolog\Handler\AbstractProcessingHandler->handle(Array)
#4 /srv/mediawiki/php-master/vendor/monolog/monolog/src/Monolog/Logger.php(707): Monolog\Logger->addRecord( in /srv/mediawiki/php-master/includes/exception/MWExceptionHandler.php on line 364
20736: Dispatcher exited with 255

Which indicates a problem @ https://github.com/wikimedia/mediawiki/blob/master/includes/objectcache/ObjectCache.php#L186
Which hasn't been tocuhed in 9 years.

Related Objects

Event Timeline

Addshore created this task.Dec 5 2019, 10:32 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2019, 10:32 AM
Addshore triaged this task as Medium priority.Dec 5 2019, 10:33 AM
ArielGlenn subscribed.Dec 5 2019, 1:11 PM
Addshore updated the task description. (Show Details)Dec 7 2019, 4:16 PM
Ladsgroup added a comment.Dec 9 2019, 5:34 PM

This is a big misconfiguration. ObjactCache in line 186 dynamically tries to load the cache class defined in "class" configuration (grrr). They all changed for the mcrouter stuff. I'm trying to see if I can fix it.

Ladsgroup added a comment.Dec 9 2019, 5:43 PM

It's because the mwmaint01 is dirty...

Stashbot added a comment.Dec 9 2019, 5:51 PM

Mentioned in SAL (#wikimedia-releng) [2019-12-09T17:51:01Z] <Amir1> ladsgroup@deployment-mwmaint01:/srv/mediawiki-staging$ sudo apt-get install php-memcached (T239894)

Ladsgroup added a comment.Dec 9 2019, 8:46 PM

nope, memcached php extension wasn't installed there. Then it was lack of geoip and msgpack php extensions.

Fixing that in puppet should not be super hard even though it's not possible currently, since beta cluster and production puppet has diverged (I can cherry-pick the patch to beta puppet master if needed). But the dispatching should work now.

Ladsgroup added a comment.Dec 9 2019, 8:50 PM
ladsgroup@deployment-mwmaint01:/var/log/wikidata$ tail dispatchChanges-wikidatawiki.log --lines 50
12887: Starting dispatcher
20:45:02 Starting loop for unlimited passes or 540 seconds
20:45:05 Posted 6 changes to enwiki, up to ID 1681591, timestamp 20191209130228. Lag is 27757 seconds. Next ID is 1681747.
20:45:08 Posted 2 changes to dewiki, up to ID 1680869, timestamp 20191204174336. Lag is 442892 seconds. Next ID is 1681747.
Ladsgroup mentioned this in T240263: Add memcached to mwmaint01 using puppet.Dec 9 2019, 8:54 PM
Ladsgroup edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.

Filled the follow up: T240263

Ladsgroup claimed this task.Dec 9 2019, 8:54 PM
Ladsgroup moved this task from To Do (prioritised from top to bottom) to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptDec 9 2019, 8:54 PM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Dec 9 2019, 9:15 PM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.
ArielGlenn added a comment.Dec 9 2019, 9:32 PM

What do we think about the pile of these in the log:

21:31:02 Wikibase\Repo\Store\Sql\SqlChangeDispatchCoordinator::selectClient: Could not lock any of the candidate client wikis for dispatching

Are they a problem?

Ladsgroup added a comment.Dec 9 2019, 9:37 PM
In T239894#5725927, @ArielGlenn wrote:

What do we think about the pile of these in the log:

21:31:02 Wikibase\Repo\Store\Sql\SqlChangeDispatchCoordinator::selectClient: Could not lock any of the candidate client wikis for dispatching

Are they a problem?

no it means, there is nothing to dispatch ^_^ Maybe we should reduce the dispacthing speed for beta.

ArielGlenn added a comment.Dec 9 2019, 9:41 PM

\o/ awesome!

Addshore closed this task as Resolved.Dec 10 2019, 10:46 AM
Addshore moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.Dec 10 2019, 11:15 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL