Page MenuHomePhabricator

"mediawiki-phpunit-hhvm" failures on all changes in mediawiki/core due to hhvm upgrade from 3.3.1+dfsg1-1+wm3.1 to 3.6.1+dfsg1-1+wm2
Closed, ResolvedPublic

Description

"mediawiki-phpunit-hhvm" jobs fail on all changes in mediawiki/core.

Example bogus failure: https://gerrit.wikimedia.org/r/#/c/205719/ https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/7969/console

The errors seems to correlate with hhvm being upgraded from 3.3.1+dfsg1-1+wm3.1 to 3.6.1+dfsg1-1+wm2.

Event Timeline

matmarex created this task.May 12 2015, 2:55 PM
matmarex raised the priority of this task from to Unbreak Now!.
matmarex updated the task description. (Show Details)
matmarex added a subscriber: matmarex.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2015, 2:55 PM
hashar added a subscriber: hashar.

From the console output:

HHVM 3.6.1 is installed.

There were 7 errors:

1) WfBaseConvertTest::testDigitToBase2 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer

includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:53
tests/phpunit/MediaWikiTestCase.php:131

2) WfBaseConvertTest::testDigitToBase3 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer

includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:65
tests/phpunit/MediaWikiTestCase.php:131

3) WfBaseConvertTest::testDigitToBase5 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer

includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:77
tests/phpunit/MediaWikiTestCase.php:131

4) WfBaseConvertTest::testDigitToBase8 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer

includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:89
tests/phpunit/MediaWikiTestCase.php:131

5) WfBaseConvertTest::testDigitToBase10 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer

includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:101
tests/phpunit/MediaWikiTestCase.php:131

6) WfBaseConvertTest::testDigitToBase16 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer

includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:113
tests/phpunit/MediaWikiTestCase.php:131

7) WfBaseConvertTest::testDigitToBase36 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer

includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:125
tests/phpunit/MediaWikiTestCase.php:131

--


There was 1 failure:

1) IPTCTest::testIPTCParseForcedUTFButInvalid
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => 'ø'
+    0 => ''
 )

According to the build time trend, that started failing roughly around 12:30 UTC. Looking on integration-slave-trusty-1015 in /var/log/apt/history.log:

Start-Date: 2015-05-12  12:15:14
Commandline: apt-get upgrade -y

Upgrade:
apt:amd64 (1.0.1ubuntu2.6, 1.0.1ubuntu2.7),
debootstrap:amd64 (1.0.59ubuntu0.2, 1.0.59ubuntu0.3),
hhvm-tidy:amd64 (0.1-3, 0.1-4),
apt-transport-https:amd64 (1.0.1ubuntu2.6, 1.0.1ubuntu2.7),
php5-fss:amd64 (1.1-3.1, 1.1.4),
apt-utils:amd64 (1.0.1ubuntu2.6, 1.0.1ubuntu2.7),
librbd1:amd64 (0.80.9-0ubuntu0.14.04.1, 0.80.9-0ubuntu0.14.04.2),
python3-update-manager:amd64 (0.196.12, 0.196.13),
hhvm-luasandbox:amd64 (2.0-8, 2.0.8),
librados2:amd64 (0.80.9-0ubuntu0.14.04.1, 0.80.9-0ubuntu0.14.04.2),
libapt-inst1.5:amd64 (1.0.1ubuntu2.6, 1.0.1ubuntu2.7),
php-luasandbox:amd64 (2.0-8, 2.0.8),
hhvm:amd64 (3.3.1+dfsg1-1+wm3.1, 3.6.1+dfsg1-1+wm2),
xtrans-dev:amd64 (1.3.4-1~ubuntu2, 1.3.5-1~ubuntu14.04.1),
libapt-pkg4.12:amd64 (1.0.1ubuntu2.6, 1.0.1ubuntu2.7),
hhvm-wikidiff2:amd64 (1.3-3.1, 1.3.4),
salt-minion:amd64 (2014.1.11+ds-2trusty1, 2014.7.5+ds-1ubuntu1),
hhvm-fss:amd64 (1.1-3.1, 1.1.4),
php-wikidiff2:amd64 (1.3-3.1, 1.3.4),
libdrm2:amd64 (2.4.56-1~ubuntu2, 2.4.60-2~ubuntu14.04.1),
distro-info-data:amd64 (0.18ubuntu0.2, 0.18ubuntu0.3),
hhvm-dev:amd64 (3.3.1+dfsg1-1+wm3.1, 3.6.1+dfsg1-1+wm2),
salt-common:amd64 (2014.1.11+ds-2trusty1, 2014.7.5+ds-1ubuntu1)

Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2015-05-12  12:15:51

Start-Date: 2015-05-12  12:34:11
Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install salt-minion
End-Date: 2015-05-12  12:34:12

Or in short hhvm got upgraded from 3.3.1+dfsg1-1+wm3.1 to 3.6.1+dfsg1-1+wm2).

hashar renamed this task from "mediawiki-phpunit-hhvm" failures on all changes in mediawiki/core to "mediawiki-phpunit-hhvm" failures on all changes in mediawiki/core due to hhvm upgrade from 3.3.1+dfsg1-1+wm3.1 to 3.6.1+dfsg1-1+wm2.May 12 2015, 3:09 PM
hashar updated the task description. (Show Details)
hashar set Security to None.
hashar added a subscriber: Joe.
hashar claimed this task.May 12 2015, 3:13 PM

So I probably screwed it up when running a mass apt-get upgrade on all Trusty machines. Need to find the previous .deb :/

hashar added a comment.EditedMay 12 2015, 3:18 PM

I found the old .deb in /var/cache/apt/archives:

archives/hhvm-luasandbox_2.0-8_amd64.deb
archives/hhvm-fss_1.1-3.1_amd64.deb
archives/hhvm-wikidiff2_1.3-3.1_amd64.deb
archives/hhvm_3.3.1+dfsg1-1+wm3.1_amd64.deb
archives/hhvm-dev_3.3.1+dfsg1-1+wm3.1_amd64.deb
archives/hhvm-tidy_0.1-3_amd64.deb

Copied them under /data/project/T98876 and running salt to dpkg -i them:

dpkg -i /data/project/T98876/*
(Reading database ... 178557 files and directories currently installed.)
Preparing to unpack .../hhvm_3.3.1+dfsg1-1+wm3.1_amd64.deb ...
hhvm stop/waiting
Unpacking hhvm (3.3.1+dfsg1-1+wm3.1) over (3.3.1+dfsg1-1+wm3.1) ...
Preparing to unpack .../hhvm-dev_3.3.1+dfsg1-1+wm3.1_amd64.deb ...
Unpacking hhvm-dev (3.3.1+dfsg1-1+wm3.1) over (3.3.1+dfsg1-1+wm3.1) ...
Preparing to unpack .../hhvm-fss_1.1-3.1_amd64.deb ...
Unpacking hhvm-fss (1.1-3.1) over (1.1-3.1) ...
Preparing to unpack .../hhvm-luasandbox_2.0-8_amd64.deb ...
Unpacking hhvm-luasandbox (2.0-8) over (2.0-8) ...
Preparing to unpack .../hhvm-tidy_0.1-3_amd64.deb ...
Unpacking hhvm-tidy (0.1-3) over (0.1-3) ...
Preparing to unpack .../hhvm-wikidiff2_1.3-3.1_amd64.deb ...
Unpacking hhvm-wikidiff2 (1.3-3.1) over (1.3-3.1) ...
Setting up hhvm (3.3.1+dfsg1-1+wm3.1) ...
hhvm start/running, process 700
Setting up hhvm-dev (3.3.1+dfsg1-1+wm3.1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up hhvm-tidy (0.1-3) ...
Setting up hhvm-wikidiff2 (1.3-3.1) ...
Setting up hhvm-fss (1.1-3.1) ...
Setting up hhvm-luasandbox (2.0-8) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
hashar lowered the priority of this task from Unbreak Now! to Normal.May 12 2015, 3:33 PM

The job is passing again so lowering the priority

I have filled a sub task for mediawiki/core test failures T98882: MediaWiki core tests fail with HHVM 3.6.1+dfsg1-1+wm2 : gmp_init(): Unable to convert variable to GMP - string is not an integer

Change 210391 had a related patch set uploaded (by Hashar):
contint: disable unattended upgrade

https://gerrit.wikimedia.org/r/210391

I have cherry picked the gerrit change to disable unattended upgrade mechanism:

Cherry picked on integration puppetmaster

Notice: /Stage[main]/Contint::Packages::Labs/File[/etc/cron.hourly/apt]/ensure: removed
Notice: /Stage[main]/Contint::Packages::Labs/Apt::Conf[lower-periodic-randomsleep]/File[/etc/apt/apt.conf.d/51lower-periodic-randomsleep]/ensure: removed
Notice: /Stage[main]/Apt::Unattendedupgrades/Package[unattended-upgrades]/ensure: removed
Notice: /Stage[main]/Apt::Unattendedupgrades/Apt::Conf[auto-upgrades]/File[/etc/apt/apt.conf.d/20auto-upgrades]/ensure: removed
Notice: /Stage[main]/Contint::Packages::Labs/Apt::Conf[unattended-upgrades-wikimedia]/File[/etc/apt/apt.conf.d/51unattended-upgrades-wikimedia]/ensure: removed

So puppet deployed it and that would prevent HHVM magic upgrade. To be followed up later on.

Change 210391 merged by Dzahn:
contint: disable unattended upgrade

https://gerrit.wikimedia.org/r/210391

From T98882 /etc/hhvm/php.ini now has hhvm.hack.lang.iconv_ignore_correct = true which should fix the gmp_init() error.

hashar closed this task as Resolved.Jun 16 2015, 2:03 PM

I have upgraded HHVM and the jobs are all happy as far as I can tell. That was made possible thanks to a workaround for gmp_init() T98882