"mediawiki-phpunit-hhvm" jobs fail on all changes in mediawiki/core.
Example bogus failure: https://gerrit.wikimedia.org/r/#/c/205719/ https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/7969/console
The errors seems to correlate with hhvm being upgraded from 3.3.1+dfsg1-1+wm3.1 to 3.6.1+dfsg1-1+wm2.
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| operations/puppet | production | +7 -5 | contint: disable unattended upgrade |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | PRODUCTION ERROR | Anomie | T95813 Fix "Warning: Unable to allocate timer" and "Warning: Unable to start limit timer" | ||
| Resolved | hashar | T101275 Build a new version of php-luasandbox and hhvm-luasandbox, and deploy to integration hosts | |||
| Resolved | hashar | T98876 "mediawiki-phpunit-hhvm" failures on all changes in mediawiki/core due to hhvm upgrade from 3.3.1+dfsg1-1+wm3.1 to 3.6.1+dfsg1-1+wm2 | |||
| Resolved | PRODUCTION ERROR | Krinkle | T98882 MediaWiki core tests fail with HHVM 3.6.1+dfsg1-1+wm2 : gmp_init(): Unable to convert variable to GMP - string is not an integer | ||
| Resolved | hashar | T98885 Unattended upgrade seems to only run daily instead of hourly | |||
| Resolved | hashar | T102616 Upgrade HHVM on CI Trusty slaves from 3.3.1 to 3.6.1 |
From the console output:
HHVM 3.6.1 is installed.
There were 7 errors:
1) WfBaseConvertTest::testDigitToBase2 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer
includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:53
tests/phpunit/MediaWikiTestCase.php:131
2) WfBaseConvertTest::testDigitToBase3 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer
includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:65
tests/phpunit/MediaWikiTestCase.php:131
3) WfBaseConvertTest::testDigitToBase5 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer
includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:77
tests/phpunit/MediaWikiTestCase.php:131
4) WfBaseConvertTest::testDigitToBase8 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer
includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:89
tests/phpunit/MediaWikiTestCase.php:131
5) WfBaseConvertTest::testDigitToBase10 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer
includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:101
tests/phpunit/MediaWikiTestCase.php:131
6) WfBaseConvertTest::testDigitToBase16 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer
includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:113
tests/phpunit/MediaWikiTestCase.php:131
7) WfBaseConvertTest::testDigitToBase36 with data set #0 ('0', '0', '0', '0', '0', '0', '0')
gmp_init(): Unable to convert variable to GMP - string is not an integer
includes/GlobalFunctions.php:3356
tests/phpunit/includes/GlobalFunctions/wfBaseConvertTest.php:125
tests/phpunit/MediaWikiTestCase.php:131
--
There was 1 failure:
1) IPTCTest::testIPTCParseForcedUTFButInvalid
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
Array (
- 0 => 'ø'
+ 0 => ''
)According to the build time trend, that started failing roughly around 12:30 UTC. Looking on integration-slave-trusty-1015 in /var/log/apt/history.log:
Start-Date: 2015-05-12 12:15:14 Commandline: apt-get upgrade -y Upgrade: apt:amd64 (1.0.1ubuntu2.6, 1.0.1ubuntu2.7), debootstrap:amd64 (1.0.59ubuntu0.2, 1.0.59ubuntu0.3), hhvm-tidy:amd64 (0.1-3, 0.1-4), apt-transport-https:amd64 (1.0.1ubuntu2.6, 1.0.1ubuntu2.7), php5-fss:amd64 (1.1-3.1, 1.1.4), apt-utils:amd64 (1.0.1ubuntu2.6, 1.0.1ubuntu2.7), librbd1:amd64 (0.80.9-0ubuntu0.14.04.1, 0.80.9-0ubuntu0.14.04.2), python3-update-manager:amd64 (0.196.12, 0.196.13), hhvm-luasandbox:amd64 (2.0-8, 2.0.8), librados2:amd64 (0.80.9-0ubuntu0.14.04.1, 0.80.9-0ubuntu0.14.04.2), libapt-inst1.5:amd64 (1.0.1ubuntu2.6, 1.0.1ubuntu2.7), php-luasandbox:amd64 (2.0-8, 2.0.8), hhvm:amd64 (3.3.1+dfsg1-1+wm3.1, 3.6.1+dfsg1-1+wm2), xtrans-dev:amd64 (1.3.4-1~ubuntu2, 1.3.5-1~ubuntu14.04.1), libapt-pkg4.12:amd64 (1.0.1ubuntu2.6, 1.0.1ubuntu2.7), hhvm-wikidiff2:amd64 (1.3-3.1, 1.3.4), salt-minion:amd64 (2014.1.11+ds-2trusty1, 2014.7.5+ds-1ubuntu1), hhvm-fss:amd64 (1.1-3.1, 1.1.4), php-wikidiff2:amd64 (1.3-3.1, 1.3.4), libdrm2:amd64 (2.4.56-1~ubuntu2, 2.4.60-2~ubuntu14.04.1), distro-info-data:amd64 (0.18ubuntu0.2, 0.18ubuntu0.3), hhvm-dev:amd64 (3.3.1+dfsg1-1+wm3.1, 3.6.1+dfsg1-1+wm2), salt-common:amd64 (2014.1.11+ds-2trusty1, 2014.7.5+ds-1ubuntu1) Error: Sub-process /usr/bin/dpkg returned an error code (1) End-Date: 2015-05-12 12:15:51 Start-Date: 2015-05-12 12:34:11 Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install salt-minion End-Date: 2015-05-12 12:34:12
Or in short hhvm got upgraded from 3.3.1+dfsg1-1+wm3.1 to 3.6.1+dfsg1-1+wm2).
So I probably screwed it up when running a mass apt-get upgrade on all Trusty machines. Need to find the previous .deb :/
I found the old .deb in /var/cache/apt/archives:
archives/hhvm-luasandbox_2.0-8_amd64.deb archives/hhvm-fss_1.1-3.1_amd64.deb archives/hhvm-wikidiff2_1.3-3.1_amd64.deb archives/hhvm_3.3.1+dfsg1-1+wm3.1_amd64.deb archives/hhvm-dev_3.3.1+dfsg1-1+wm3.1_amd64.deb archives/hhvm-tidy_0.1-3_amd64.deb
Copied them under /data/project/T98876 and running salt to dpkg -i them:
dpkg -i /data/project/T98876/* (Reading database ... 178557 files and directories currently installed.) Preparing to unpack .../hhvm_3.3.1+dfsg1-1+wm3.1_amd64.deb ... hhvm stop/waiting Unpacking hhvm (3.3.1+dfsg1-1+wm3.1) over (3.3.1+dfsg1-1+wm3.1) ... Preparing to unpack .../hhvm-dev_3.3.1+dfsg1-1+wm3.1_amd64.deb ... Unpacking hhvm-dev (3.3.1+dfsg1-1+wm3.1) over (3.3.1+dfsg1-1+wm3.1) ... Preparing to unpack .../hhvm-fss_1.1-3.1_amd64.deb ... Unpacking hhvm-fss (1.1-3.1) over (1.1-3.1) ... Preparing to unpack .../hhvm-luasandbox_2.0-8_amd64.deb ... Unpacking hhvm-luasandbox (2.0-8) over (2.0-8) ... Preparing to unpack .../hhvm-tidy_0.1-3_amd64.deb ... Unpacking hhvm-tidy (0.1-3) over (0.1-3) ... Preparing to unpack .../hhvm-wikidiff2_1.3-3.1_amd64.deb ... Unpacking hhvm-wikidiff2 (1.3-3.1) over (1.3-3.1) ... Setting up hhvm (3.3.1+dfsg1-1+wm3.1) ... hhvm start/running, process 700 Setting up hhvm-dev (3.3.1+dfsg1-1+wm3.1) ... Processing triggers for ureadahead (0.100.0-16) ... Setting up hhvm-tidy (0.1-3) ... Setting up hhvm-wikidiff2 (1.3-3.1) ... Setting up hhvm-fss (1.1-3.1) ... Setting up hhvm-luasandbox (2.0-8) ... Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
The job is passing again so lowering the priority
I have filled a sub task for mediawiki/core test failures T98882: MediaWiki core tests fail with HHVM 3.6.1+dfsg1-1+wm2 : gmp_init(): Unable to convert variable to GMP - string is not an integer
Change 210391 had a related patch set uploaded (by Hashar):
contint: disable unattended upgrade
I have cherry picked the gerrit change to disable unattended upgrade mechanism:
Cherry picked on integration puppetmaster
Notice: /Stage[main]/Contint::Packages::Labs/File[/etc/cron.hourly/apt]/ensure: removed Notice: /Stage[main]/Contint::Packages::Labs/Apt::Conf[lower-periodic-randomsleep]/File[/etc/apt/apt.conf.d/51lower-periodic-randomsleep]/ensure: removed Notice: /Stage[main]/Apt::Unattendedupgrades/Package[unattended-upgrades]/ensure: removed Notice: /Stage[main]/Apt::Unattendedupgrades/Apt::Conf[auto-upgrades]/File[/etc/apt/apt.conf.d/20auto-upgrades]/ensure: removed Notice: /Stage[main]/Contint::Packages::Labs/Apt::Conf[unattended-upgrades-wikimedia]/File[/etc/apt/apt.conf.d/51unattended-upgrades-wikimedia]/ensure: removed
So puppet deployed it and that would prevent HHVM magic upgrade. To be followed up later on.
From T98882 /etc/hhvm/php.ini now has hhvm.hack.lang.iconv_ignore_correct = true which should fix the gmp_init() error.
I have upgraded HHVM and the jobs are all happy as far as I can tell. That was made possible thanks to a workaround for gmp_init() T98882