PHP 5.6 and 5.5 does not work with Mediawiki 1.26 on Centos 6.6 with default APCu settings
Closed, ResolvedPublic

Description

"PHP Fatal error: Unsupported operand types in /var/www/html/wiki/includes/registration/ExtensionRegistry.php on line 272"
It appears that this is the problem line in ExtensionsRegistry.php

$this->loaded += $info['credits'];
This error occurs on PHP 5.5 and PHP 5.6

It does not occur on PHP 5.4

Maniphest changed the visibility from "Public (No Login Required)" to "Custom Policy".Dec 3 2015, 7:59 PM
Maniphest changed the edit policy from "All Users" to "Custom Policy".
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 3 2015, 7:59 PM
Minddragon updated the task description. (Show Details)
Minddragon changed Security from None to Software security bug.
Minddragon edited subscribers, added: Minddragon; removed: Aklapper.
Krenair added a subscriber: Krenair.Dec 3 2015, 8:02 PM

How is this a software security bug?

MaxSem changed the visibility from "Custom Policy" to "Public (No Login Required)".
MaxSem changed Security from Software security bug to None.
MaxSem removed a subscriber: Krenair.
Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 3 2015, 8:02 PM
Legoktm added a subscriber: Legoktm.

Can you please indicate what extensions and skins you have installed and their versions? One of them likely has an invalid extension.json or skin.json file.

Steps to replicate:

  1. Have a CentOS machine, apache, Mysql, PHP 5.6w (I did yum install php56w php56w-common php56w-pecl* and a few others
  2. Download Mediawiki 1.26 tarball
  3. Ran the initial Mediawiki setup with no extensions enabled
  4. Observed the crash. I couldn't get skins to work as well.
  5. Tried Yum remove php56* and installed PHP55* and got the same thing
  6. Did Yum remove php55* and installed 54* and everything worked including default Mediawiki extensions.

Please note several other people on support forums are reporting similar conditions.

Anomie added a subscriber: Anomie.EditedDec 3 2015, 9:41 PM

I managed to reproduce this with a virtual machine:

Then I found that a package "php56w" does not exist. I'm guessing you're using https://mirror.webtatic.com/yum/el6/latest.rpm as an additional repo? Let's try that.

After poking at it a bit, it looks like ExtensionRegistry decided to use APC for caching and the APC cache is somehow or other corrupting the keys in the stored arrays. Uninstalling php56w-pecl-apcu makes it work by not using APC. Probably this didn't happen in 1.25 because the code removed in https://gerrit.wikimedia.org/r/#/c/226441/ avoided the corruption by only storing strings in APC.

ori added a subscriber: ori.Dec 13 2015, 12:01 AM
ori added a comment.Dec 13 2015, 12:46 AM

Following @Anomie's instructions, I end up with these warnings & notices:

Can you stick a var_dump($data) right after ExtensionRegistry reads out of the APC cache at line 133?

ori added a comment.Dec 13 2015, 1:17 AM

Can you stick a var_dump($data) right after ExtensionRegistry reads out of the APC cache at line 133?

ori added a comment.Dec 13 2015, 1:19 AM

Can you stick a var_dump($data) right after ExtensionRegistry reads out of the APC cache at line 133?

It does appear to be corrupt somehow. Replacing $data = $this->cache->get( $key ); with $data = false; (and thus forcing the data to be computed rather than loaded from the cache) works.

ori added a comment.Dec 13 2015, 1:25 AM
This comment was removed by ori.
ori added a comment.Dec 13 2015, 1:30 AM

Changing apc.serializer = 'default' to apc.serializer = 'php' in /etc/php.d/apcu.ini and then restarting httpd makes the problem go away.

ori added a comment.Dec 13 2015, 1:38 AM

It'd be great to have a minimal reproduction case (e.g. some cooked-up array that reliably triggers this corruption). Then, when APCBagOStuff is first initialized, try to set/get it, and if it is corrupt, default to the old behavior (serialize and deserialize in user code).

ori added a comment.Dec 13 2015, 5:53 AM

I'm pretty certain the bug is the one described here:

@Minddragon, you should edit /etc/php.d/apcu.ini and ensure apc.serializer is set to php rather than default.

Since this is a possible use-after-free in apcu's serialization code, I don't think that we should try to trigger it, as I suggested above. We should just fall back to the old behavior of serializing everything in PHP "user space" when APCu is used with the default serializer.

Change 258734 had a related patch set uploaded (by Ori.livneh):
Work around APCu memory corruption bug

https://gerrit.wikimedia.org/r/258734

@ori thank you guys for figuring it out :)

Change 258734 merged by jenkins-bot:
Work around APCu memory corruption bug

https://gerrit.wikimedia.org/r/258734

Change 258930 had a related patch set uploaded (by Legoktm):
Work around APCu memory corruption bug

https://gerrit.wikimedia.org/r/258930

Change 258930 merged by jenkins-bot:
Work around APCu memory corruption bug

https://gerrit.wikimedia.org/r/258930

Legoktm assigned this task to ori.Dec 14 2015, 7:04 AM
Legoktm closed this task as Resolved.

Fixed in master and backported to REL1_26. Will be included as part of the 1.26.1 release. Thanks everyone for investigating and fixing this :)

Legoktm renamed this task from PHP 5.6 and 5.5 does not work with Mediawiki 1.26 on Centos 6.6 to PHP 5.6 and 5.5 does not work with Mediawiki 1.26 on Centos 6.6 with default APCu settings.Dec 14 2015, 7:05 AM
Legoktm added a project: MediaWiki-Cache.