Page MenuHomePhabricator

PHP Fatal Errors on mw1275 after deployment
Closed, DuplicatePublic

Description

After a deploy today:

10:02 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.34.0-wmf.3/extensions/WikibaseLexemeCirrusSearch/: Fix reference to classes that moved (T222347) (duration: 00m 55s)

there were multiple errors on mw1275, which was depooled a little bit after.

Logs:

May  3 10:02:16 mw1275 php7.2-fpm: PHP Fatal error:  Uncaught Error: Call to undefined method ResourceLoaderOOUIFileModule::getSkinThemeMap() in /srv/mediawiki/php-1.34.0-wmf.3/includes/resourceloader/ResourceLoaderOOUIFileModule.php:58#012Stack trace:#012#0 /srv/mediawiki/php-1.34.0-wmf.3/includes/resourceloader/ResourceLoaderOOUIFileModule.php(39): ResourceLoaderOOUIFileModule->getSkinSpecific('core', 'styles')#012#1 /srv/mediawiki/php-1.34.0-wmf.3/includes/resourceloader/ResourceLoader.php(547): ResourceLoaderOOUIFileModule->__construct(Array)#012#2 /srv/mediawiki/php-1.34.0-wmf.3/includes/OutputPage.php(495): ResourceLoader->getModule('oojs-ui-core.st...')#012#3 /srv/mediawiki/php-1.34.0-wmf.3/includes/OutputPage.php(538): OutputPage->filterModules(Array, NULL, 'styles')#012#4 /srv/mediawiki/php-1.34.0-wmf.3/includes/OutputPage.php(569): OutputPage->getModules(true, NULL, 'mModuleStyles', 'styles')#012#5 /srv/mediawiki/php-1.34.0-wmf.3/includes/OutputPage.php(3167): OutputPage->getModuleStyles(true)#012#6 /srv/mediawiki/php-1.34.0-wmf.3/includes/OutputPage.php(3372): OutputPage->g in /srv/mediawiki/php-1.34.0-wmf.3/includes/resourceloader/ResourceLoaderOOUIFileModule.php on line 58

May  3 10:02:16 mw1275 php7.2-fpm: PHP Fatal error:  require(): Failed opening required '/srv/mediawiki/php-1.34.0-wmf.3/vendor/composer/../mustangostang/spyc/Spyc.php' (include_path='/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/console_getopt:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/mail:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/mail_mime:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/net_smtp:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/net_socket:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/pear-core-minimal/src:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/pear_exception:.:/usr/share/php') in /srv/mediawiki/php-1.34.0-wmf.3/vendor/composer/autoload_real.php on line 61

May  3 10:02:16 mw1275 php7.2-fpm: PHP Fatal error:  require(): Failed opening required '/srv/mediawiki/php-1.34.0-wmf.3/vendor/composer/../mustangostang/spyc/Spyc.php' (include_path='/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/console_getopt:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/mail:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/mail_mime:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/net_smtp:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/net_socket:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/pear-core-minimal/src:/srv/mediawiki/php-1.34.0-wmf.3/vendor/pear/pear_exception:.:/usr/share/php') in /srv/mediawiki/php-1.34.0-wmf.3/vendor/composer/autoload_real.php on line 61

Event Timeline

jijiki created this task.May 3 2019, 1:39 PM
Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptMay 3 2019, 1:39 PM
jijiki triaged this task as Normal priority.May 3 2019, 1:39 PM
Reedy updated the task description. (Show Details)May 3 2019, 2:12 PM

That deployment (fixing T222347) was a backport (I3e4bf4b12d), which I force-submitted because gate-and-submit-swat consistently failed with a test failure that we judged to be unrelated. Perhaps gate-and-submit-swat does some extra work that was/is now missing?

jijiki added a comment.May 3 2019, 2:19 PM

@Lucas_Werkmeister_WMDE We will look into it, it only happened on a single server so we believe, for now, that it could not be related to the change per se.

Joe added a comment.May 3 2019, 2:55 PM

some things from my very initial analysis:

  • I tried to purge first the directory that the deployment had invalidated, the error didn't go away
  • I tried purging the autoload file in composer that was suppsedly loading the library from the wrong place, still no dice
  • I finally tried purging all of the opcache, which solved the problem.

Now, mitigations for this will be:

I saved a dump of the opcache metadata (via php7adm /opcache-free) for further analysis. All the opcache infos gave me no hint at what could be wrong.

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptMay 29 2019, 3:57 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM