Page MenuHomePhabricator

PHP Fatal Errors on mw1275 after deployment
Closed, DuplicatePublicPRODUCTION ERROR

Description

After a deploy today:

10:02 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.34.0-wmf.3/extensions/WikibaseLexemeCirrusSearch/: Fix reference to classes that moved (T222347) (duration: 00m 55s)

there were multiple errors on mw1275, which was depooled a little bit after.

Logs:

May  3 10:02:16 mw1275 php7.2-fpm: PHP Fatal error:  Uncaught Error: Call to undefined method ResourceLoaderOOUIFileModule::getSkinThemeMap() in /srv/mediawiki/php-1.34.0-wmf.3/includes/resourceloader/ResourceLoaderOOUIFileModule.php:58#012Stack trace:#012#0 /srv/mediawiki/php-1.34.0-wmf.3/includes/resourceloader/ResourceLoaderOOUIFileModule.php(39): ResourceLoaderOOUIFileModule->getSkinSpecific('core', 'styles')#012#1 /srv/mediawiki/php-1.34.0-wmf.3/includes/resourceloader/ResourceLoader.php(547): ResourceLoaderOOUIFileModule->__construct(Array)#012#2 /srv/mediawiki/php-1.34.0-wmf.3/includes/OutputPage.php(495): ResourceLoader->getModule('oojs-ui-core.st...')#012#3 /srv/mediawiki/php-1.34.0-wmf.3/includes/OutputPage.php(538): OutputPage->filterModules(Array, NULL, 'styles')#012#4 /srv/mediawiki/php-1.34.0-wmf.3/includes/OutputPage.php(569): OutputPage->getModules(true, NULL, 'mModuleStyles', 'styles')#012#5 /srv/mediawiki/php-1.34.0-wmf.3/includes/OutputPage.php(3167): OutputPage->getModuleStyles(true)#012#6 /srv/mediawiki/php-1.34.0-wmf.3/includes/OutputPage.php(3372): OutputPage->g in /srv/mediawiki/php-1.34.0-wmf.3/includes/resourceloader/ResourceLoaderOOUIFileModule.php on line 58

May  3 10:02:16 mw1275 php7.2-fpm: PHP Fatal error:  require(): Failed opening required '/srv/mediawiki/php-1.34.0-wmf.3/vendor/composer/../mustangostang/spyc/Spyc.php' (include_path='/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/console_getopt:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/mail:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/mail_mime:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/net_smtp:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/net_socket:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/pear-core-minimal/src:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/pear_exception:.:/usr/share/php') in /srv/mediawiki/php-1.34.0-wmf.3/vendor/composer/autoload_real.php on line 61

May  3 10:02:16 mw1275 php7.2-fpm: PHP Fatal error:  require(): Failed opening required '/srv/mediawiki/php-1.34.0-wmf.3/vendor/composer/../mustangostang/spyc/Spyc.php' (include_path='/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/console_getopt:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/mail:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/mail_mime:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/net_smtp:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/net_socket:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/pear-core-minimal/src:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/pear_exception:.:/usr/share/php') in /srv/mediawiki/php-1.34.0-wmf.3/vendor/composer/autoload_real.php on line 61

Event Timeline

jijiki triaged this task as Medium priority.May 3 2019, 1:39 PM

That deployment (fixing T222347) was a backport (I3e4bf4b12d), which I force-submitted because gate-and-submit-swat consistently failed with a test failure that we judged to be unrelated. Perhaps gate-and-submit-swat does some extra work that was/is now missing?

@Lucas_Werkmeister_WMDE We will look into it, it only happened on a single server so we believe, for now, that it could not be related to the change per se.

some things from my very initial analysis:

  • I tried to purge first the directory that the deployment had invalidated, the error didn't go away
  • I tried purging the autoload file in composer that was suppsedly loading the library from the wrong place, still no dice
  • I finally tried purging all of the opcache, which solved the problem.

Now, mitigations for this will be:

I saved a dump of the opcache metadata (via php7adm /opcache-free) for further analysis. All the opcache infos gave me no hint at what could be wrong.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM