Not just Wikibase. Errors in VE-MW and MobileFrontend. :-( Can we revert the test for now to see if that fixes it?

@MoritzMuehlenhoff HHVM upgrade is causing segfaults in CI

May be related T165043

Can someone create a simple repo case? Or at least a backtrace?

In T165074#3256198, @greg wrote:

Can someone create a simple repo case? Or at least a backtrace?

Do we have a jessie hhvm host we can trivially run tests on? I can't ssh onto ci-trusty-wikimedia-* as I get Permission denied. integration-slave-jessie-1001 seems to only have hhvm-dev (which feels like a different bug)

The snapshots we have:

Mentioned in SAL (#wikimedia-releng) [2017-05-11T20:43:40Z] <hashar> nodepool: delete today jessie image snapshot. It comes with HHVM 3.18 which segfault with MediaWiki/PHPUnit. Rolled back to snapshot-ci-jessie-1494425642 from 30 hours ago. T165074

@MoritzMuehlenhoff we should probably downgrade the HHVM version from Beta and CI and work on repro'ing elsewhere. This is preventing merges and is UBN.

Jessie instances are now being booted from snapshot-ci-jessie-1494425642 which should have the previous HHVM version.

What is left to do is to bring back HHVM 3.12 to jessie-wikimedia/main. Else tomorrow 3.18 will end up being installed again and breaks CI.

Ok, so a clean vagrant vm (with 4GB ram!), will segfault by running phpunit with no extensions

From gdb attached...

Continuing.
[New Thread 0x7f1b053ff700 (LWP 2337)]
[Thread 0x7f1b053ff700 (LWP 2337) exited]
[New Thread 0x7f1b053ff700 (LWP 2338)]
[Thread 0x7f1b053ff700 (LWP 2338) exited]

Program received signal SIGSEGV, Segmentation fault.
HPHP::UserFile::close (this=0x7f1afd461090) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/user-file.cpp:212
212	/tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/user-file.cpp: No such file or directory.
(gdb) bt
#0  HPHP::UserFile::close (this=0x7f1afd461090) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/user-file.cpp:212
#1  0x0000000001a74ec2 in HPHP::XMLReader::close (this=0x7f1afd411ec0) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/ext/xmlreader/ext_xmlreader.cpp:95
#2  0x0000000002133130 in HPHP::MemoryManager::sweep (this=0x7f1b25a8c840, this@entry=<error reading variable: Asked for position 0 of stack, stack only has 0 elements on it.>)
    at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/memory-manager.cpp:471
#3  0x00000000021324dd in HPHP::hphp_memory_cleanup () at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/program-functions.cpp:2322
#4  0x0000000000fd6605 in HPHP::hphp_session_exit () at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/program-functions.cpp:2370
#5  0x0000000000fda5df in HPHP::execute_command_line_end (xhprof=0, coverage=coverage@entry=true, program=0x7f1b0ee48d58 "tests/phpunit/phpunit.php")
    at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/program-functions.cpp:721
#6  0x0000000000fe2d95 in HPHP::execute_program_impl (argc=argc@entry=4, argv=argv@entry=0x7fff36a84068)
    at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/program-functions.cpp:1830
#7  0x0000000000fe50be in HPHP::execute_program (argc=argc@entry=4, argv=argv@entry=0x7fff36a84068) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/program-functions.cpp:1148
#8  0x0000000000b56d6a in main (argc=<optimized out>, argv=0x7fff36a84068) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/hhvm/main.cpp:83

We can't easily downgrade the HHVM package in the main repo, it's otherwise working fine in production and running on about 20 servers for more exposure. If that prevents merges I suggest to disable the failing test temporarily. Or alternatively we can add a new archive component with 3.12 tomorrow and configure the CI setup to use that for a while.

We should really use the new HHVM in testing first before going to production. If the tests are broken it means fix the tests/the underlying issue, not work around it by disabling the test and continuing to use the (now uncertain if that test actually found a bug) in production.

CI is fine now, we've downgraded there. But we should be in sync on this going forward.

Mentioned in SAL (#wikimedia-operations) [2017-05-11T20:57:55Z] <hashar> CI Phpunit jobs were segfaulting due to an upgrade of HHVM to 3.18. Got rolled back to 3.12 - T165074

The new HHVM version has been extensively tested on five canary servers in production for 5-6 week now. As per Reedy's backtrace from above it has in fact identified the crash we already knew https://phabricator.wikimedia.org/T162586 (reported upstream at https://github.com/facebook/hhvm/issues/7779 but so far there wasn't a better reproducer than "expose 3 days of Wikimedia production traffic" for it :-) So the test suite has proven to be really useful.

Going forward from here, we should extend CI to also cover the "experimental" component from our repo (at least for partial CI runs). That would have identified that much quicker.

Might be T156923: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests surfacing again which mentionned XMLReader.

Change 353533 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: pin HHVM packages to use experimental component

https://gerrit.wikimedia.org/r/353533

Change 353533 merged by Muehlenhoff:
[operations/puppet@production] contint: pin HHVM packages to use experimental component

https://gerrit.wikimedia.org/r/353533

I've built new HHVM packages with a patch as proposed by upstream in https://github.com/facebook/hhvm/issues/7779, that avoids the segfault and makes the phpunit tests pass with 3.18 again. I'll upload the new packages later on, after that we can revert the HHVM 3.12 hack again.

(Tested in mediawiki-vagrant)

Mentioned in SAL (#wikimedia-operations) [2017-05-15T13:27:45Z] <moritzm> uploaded HHVM 3.18.2+dfsg-1+wmf3 to apt.wikimedia.org (addresses segfault in XML reader (T162586, T165074)

Just saying this also happens in travis instances causing Wikibase travis tests to fail. https://travis-ci.org/wikimedia/mediawiki-extensions-Wikibase/jobs/232431659 if there is anything specific needs to be done (like using 3.17 instead of 3.18) please tell me to apply to travis config.

@Ladsgroup I'm not sure how that Travis setup is configured, but if you make it install HHVM 3.18.2+dfsg-1+wmf3, that should be fixed.

Change 358341 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: install HHVM from main

https://gerrit.wikimedia.org/r/358341

Change 358341 merged by Muehlenhoff:
[operations/puppet@production] contint: install HHVM from main

https://gerrit.wikimedia.org/r/358341

Mentioned in SAL (#wikimedia-releng) [2017-06-12T12:30:28Z] <hashar> nodepool: refreshing Jessie snapshot to upgrade HHVM from 3.12 to 3.18 - T167493 T165074