Do we have a core dump?

In T142158#2525400, @Legoktm wrote:

Do we have a core dump?

Getting the core file is going to be the only real way to debug this. The most common cause of a PHP segfault is a double free bug in a PECL extension. These can be really sneaky to track down.

This is really annoying as it happens far more often than not.

I've seen similar segfaults on travis when running the Wikibase tests there in the last couple of days (on php55 only), but they went away again a few days ago after consistently occurring for a few days (for example https://travis-ci.org/wikimedia/mediawiki-extensions-Wikibase/jobs/150951279). (Note: We use MediaWiki master there to run our tests)

Maybe it is mediawiki coursing the segfault?

https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55/6750/console

dmesg on integration-slave-trusty-1001 reports we are consistently segfaulting at the same instruction pointer:

[12600829.516010] traps: php5[3756] general protection ip:70c450 sp:7ffd02145a28 error:0 in php5[400000+7f3000]
[13109969.662961] php5[31621]: segfault at 3 ip 000000000070c450 sp 00007fff7486b2b8 error 6 in php5[400000+7f3000]
[13121365.417743] php5[29445]: segfault at 3 ip 000000000070c450 sp 00007ffd296d1a68 error 6 in php5[400000+7f3000]
[13126836.791557] traps: php5[2279] general protection ip:70c450 sp:7fff7281d078 error:0 in php5[400000+7f3000]
[13138327.663403] php5[28120]: segfault at 3 ip 000000000070c450 sp 00007ffe2d34be48 error 6 in php5[400000+7f3000]
[13139900.720581] php5[1421]: segfault at 3 ip 000000000070c450 sp 00007ffca4008a98 error 6 in php5[400000+7f3000]

Using addr2line we can see this is in the garbage collector

ebernhardson@integration-slave-trusty-1001:~$ ~/addr2line -e $(which php5) 70c450
/build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h:189

Specifically that bit of code is the first line of this function (from php 5.5.9 sources):

static zend_always_inline void gc_remove_from_buffer(gc_root_buffer *root TSRMLS_DC)
{
    root->next->prev = root->prev;
    root->prev->next = root->next;
    root->prev = GC_G(unused);
    GC_G(unused) = root;
    GC_BENCH_DEC(root_buf_length);
}

Checking the php bugtracker there are a few different fatals/errors related to garbage collection and gc_remove_from_buffer. We could try a few of them out, but it would take a bit of effort to test the various patches until we find the relevant patch. As a workaround, i believe we can disable the garbage collector when running tests and things might "just work". Incoming patch that will set this for the mediawiki-extensions-php55 job, and if it does the trick we can also set it elsewhere. refcounted GC should still work with this disabled, only the cycle collector will be disabled.

Change 306072 had a related patch set uploaded (by EBernhardson):
Try disable garbage collection to prevent segfaults

https://gerrit.wikimedia.org/r/306072

should note that disabling gc is just a workaround,

16:19 <+SMalyshev> the thing is it may not even be in GC... GC is just a thing that everyone dumps it's vars on it, so if there's a corrupt var somewhere, GC will get it eventually

Change 306072 merged by jenkins-bot:
Try disable garbage collection to prevent segfaults

https://gerrit.wikimedia.org/r/306072

Looking at the overnight runs of mediawiki-extensions-php55, the problem seems to have subsided with the deployment of the above patch.

Could be unrelated but just noticed another one of these in a similar (?) job:

https://gerrit.wikimedia.org/r/#/c/307729/
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/900/console

............................................................. 2135 / 5399 ( 39%)
................................................./tmp/hudson4524155360135912232.sh: line 6: 
  9997 Segmentation fault      php phpunit.php --log-junit "$WORKSPACE/log/junit-wikidata.xml" --group Wikibase,WikibaseAPI,Purtle
Build step 'Execute shell' marked build as failure

I also faced segfaults today in https://gerrit.wikimedia.org/r/#/c/309972/4 in the mwext-Wikibase-repo-tests-sqlite-php55 job that are potentially related:

https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/973/console
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/972/console
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/970/console

I tried to narrow down the cause, by removing all Wikibase repo tests but the test changed in that patch, but that also made the jobs pass, so the only test touched in that change is not responsible (alone).

The workaround in https://gerrit.wikimedia.org/r/#/c/309972/6/repo/tests/phpunit/includes/Notifications/JobQueueChangeNotificationSenderTest.php (in setUp) makes the changes pass as well, which is in contrast to the findings above.

+ wikibase / wikidata related projects.

T64623 was the previous PHP segfault debugging session. With the very useful patch by Faidon: https://gerrit.wikimedia.org/r/#/c/119222/

Change 310673 had a related patch set uploaded (by Hashar):
Reapply "Dump corefiles on segfaults"

https://gerrit.wikimedia.org/r/310673

Mentioned in SAL (#wikimedia-releng) [2016-09-14T21:37:26Z] <hashar> integration: setting "ulimit -c 2097152" on all slaves due to Zend PHP segfaulting T142158

Change 310673 merged by jenkins-bot:
Reapply "Dump corefiles on segfaults"

https://gerrit.wikimedia.org/r/310673

2GBytes core dump files are now enabled. They should land on the slaves as /var/tmp/core/core.*.

There is a bunch caused by lua. I have removed them all to start again with a clean state.

Command to check for core from integration-saltmaster :

salt -v '*trusty*' cmd.run 'ls -lrta /var/tmp/core/core.*'

Triggered a few builds manually:

https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/983/
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/984/
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/985/
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/986/
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/987/
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/988/

Quickly looked at one on integration-slave-trusty-1013

gdb /usr/bin/php5 /var/tmp/core/core.integration-slave-trusty-1013.php5.27576.1473889797
Reading symbols from /usr/bin/php5...Reading symbols from /usr/lib/debug//usr/bin/php5...done.
[New LWP 27576]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).

Mysql is not happy :/

Core was generated by `php5 phpunit.php --log-junit /mnt/jenkins-workspace/workspace/mwext-Wikibase-re'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000070c450 in gc_remove_from_buffer (root=0x23993500) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h:189
189     /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h: No such file or directory.
(gdb)

So gc_remove_from_buffer which sounds familiar with T64623. We had gc disabled for tests.

The 154 lines stacktrace is in P4049

Might be worth trying disabling the gc again such as with https://gerrit.wikimedia.org/r/#/c/127485/1/tests/phpunit/phpunit.php

Prepare a patch for mediawiki/core phpunit.php with:

	register_shutdown_function( function() {
		gc_collect_cycles();
		gc_disable();
	} );

Create a change for Wikibase that Depends-On. Rebuild a dozen a time via the Jenkins UI. Check whether there are still segfaults.

And here is a night worth of core files:

`
root@integration-saltmaster:~# salt -v '*trusty*' cmd.run 'ls -hlrta /var/tmp/core/core.*'

integration-slave-trusty-1011.integration.eqiad.wmflabs:
 595M Sep 14 21:49 /var/tmp/core/core.integration-slave-trusty-1011.php5.21391.1473889766
 595M Sep 15 04:18 /var/tmp/core/core.integration-slave-trusty-1011.php5.15988.1473913130
integration-slave-trusty-1018.integration.eqiad.wmflabs:
 596M Sep 15 04:08 /var/tmp/core/core.integration-slave-trusty-1018.php5.15464.1473912526

integration-slave-trusty-1004.integration.eqiad.wmflabs:
 596M Sep 15 04:40 /var/tmp/core/core.integration-slave-trusty-1004.php5.28493.1473914446

integration-slave-trusty-1013.integration.eqiad.wmflabs:
595M Sep 14 21:49 /var/tmp/core/core.integration-slave-trusty-1013.php5.27576.1473889797
`

I have added a gdb script from puppet modules/admin/files/home/hashar/gdbinit . Put it to /mnt/home/jenkins-deploy

Then start gdb and source gdbinit. That gives a bunch of commands that are listed with help user-defined.

So:

integration-slave-trusty-1018:~$ gdb /usr/bin/php5 /var/tmp/core/core.integration-slave-trusty-1018.php5.15464.1473912526

Core was generated by `php5 phpunit.php --log-junit /mnt/jenkins-workspace/workspace/mwext-Wikibase-re'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000070c450 in gc_remove_from_buffer (root=0x224926c0) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h:189
189	/build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h: No such file or directory.

(gdb) source gbinit
(gdb) set logging on
Copying output to gdb.txt.
(gdb) zbacktrace

End result is pages of stack traces with the PHP file and line number. Will paste soonish

And the PHP aware stacktrace is P4052

Looks like it is whatever test defined at extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php:431

It eventually reach a code path in includes/site/SiteList.php which is class SiteList extends GenericArrayObject. And that explodes on line 89:

79     protected function preSetElement( $index, $site ) {
80         if ( $this->hasSite( $site->getGlobalId() ) ) {
81             $this->removeSite( $site->getGlobalId() );
82         }
83 
84         $this->byGlobalId[$site->getGlobalId()] = $index;
85         $this->byInternalId[$site->getInternalId()] = $index;
86         
87         $ids = $site->getNavigationIds();
88         foreach ( $ids as $navId ) {
89             $this->byNavigationId[$navId] = $index;   # SEGFAULT!!!!!!
90         }
91         
92         return true;
93     }

No clue how to reliably reproduce it though :-(

at the moment, i am running php 5.5.9-1ubuntu4.19 and am getting segmentation fault when I run wikibase repo tests locally (but not always)

seems I am able to reproduce only when running the tests on a fresh checkout of a branch (of core + wikibase)

on jenkins, it could be sometimes that a jenkins slave is using an existing checkout of mediawiki + wikibase and sometimes has to do fresh checkout.

maybe the segmentation fault is happening when there is garbage collection

with set env MALLOC_CHECK_=3 in gdb, I get:

Wikibase\Test\StoreTest::testRebuild with data set #0 (Wikibase\SqlStore Object (...))
Cannot use a scalar value as an array

/var/www/wiki/w/includes/site/SiteList.php:89
/var/www/wiki/w/includes/libs/GenericArrayObject.php:155
/var/www/wiki/w/includes/libs/GenericArrayObject.php:109
/var/www/wiki/w/includes/site/SiteList.php:264
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/SiteLinksView.php:216
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/SiteLinksView.php:164
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/SiteLinksView.php:123
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/ItemView.php:124
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/ItemView.php:107
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/EntityView.php:84
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/includes/ParserOutput/EntityParserOutputGenerator.php:351
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/includes/ParserOutput/EntityParserOutputGenerator.php:168
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php:294
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php:232
/var/www/wiki/w/includes/page/WikiPage.php:2182
/var/www/wiki/w/includes/page/WikiPage.php:2265
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php:258
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Store/StoreTest.php:60
/var/www/wiki/w/tests/phpunit/MediaWikiTestCase.php:388
/var/www/wiki/w/maintenance/doMaintenance.php:110

@hashar you set this to UBN! back on Sept 14th. Is that still the case? (I know I'm not suppose to distract you from your other work ;) ).

We could do https://bugs.php.net/bugs-generating-backtrace.php and run it on the specific file that causes the fault to get a trace, and then possibly try and find similar errors to that and if we doint find any similar, report it.

I'm wondering could it be this bug https://bugs.php.net/bug.php?id=72286

Since that also affects 5.5 but not 7.0?

Apparently it is fixed in one of the php 5.5 releases but still doesn't work with php 5.6.

Also they are saying the problem is reproducible if we have opcache disabled.

Or we can add zend.enable_gc=0 to php.ini

or in phpunit do

phpunit -d zend.enable_gc=0

In T142158#2573982, @EBernhardson wrote:

dmesg on integration-slave-trusty-1001 reports we are consistently segfaulting at the same instruction pointer:

[12600829.516010] traps: php5[3756] general protection ip:70c450 sp:7ffd02145a28 error:0 in php5[400000+7f3000]
[13109969.662961] php5[31621]: segfault at 3 ip 000000000070c450 sp 00007fff7486b2b8 error 6 in php5[400000+7f3000]
[13121365.417743] php5[29445]: segfault at 3 ip 000000000070c450 sp 00007ffd296d1a68 error 6 in php5[400000+7f3000]
[13126836.791557] traps: php5[2279] general protection ip:70c450 sp:7fff7281d078 error:0 in php5[400000+7f3000]
[13138327.663403] php5[28120]: segfault at 3 ip 000000000070c450 sp 00007ffe2d34be48 error 6 in php5[400000+7f3000]
[13139900.720581] php5[1421]: segfault at 3 ip 000000000070c450 sp 00007ffca4008a98 error 6 in php5[400000+7f3000]

Using addr2line we can see this is in the garbage collector

ebernhardson@integration-slave-trusty-1001:~$ ~/addr2line -e $(which php5) 70c450
/build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h:189

Specifically that bit of code is the first line of this function (from php 5.5.9 sources):

static zend_always_inline void gc_remove_from_buffer(gc_root_buffer *root TSRMLS_DC)
{
    root->next->prev = root->prev;
    root->prev->next = root->next;
    root->prev = GC_G(unused);
    GC_G(unused) = root;
    GC_BENCH_DEC(root_buf_length);
}

Checking the php bugtracker there are a few different fatals/errors related to garbage collection and gc_remove_from_buffer. We could try a few of them out, but it would take a bit of effort to test the various patches until we find the relevant patch. As a workaround, i believe we can disable the garbage collector when running tests and things might "just work". Incoming patch that will set this for the mediawiki-extensions-php55 job, and if it does the trick we can also set it elsewhere. refcounted GC should still work with this disabled, only the cycle collector will be disabled.

Other users seem to be having this problem

http://www.serverphorums.com/read.php?7,834007 looks very similar to the line you pointed at.

Maybe

• "(Segmentation fault in gc_remove_zval_from_buffer)."

https://bugs.php.net/bug.php?id=67314

?

I belive I have found the fixes for this.

Fixed in php 5.6.15, per changelog

We should setup a php 5.6 test for the test that fails and try it in there using php 5.6.15.

https://bugs.php.net/bug.php?id=70601
https://bugs.php.net/bug.php?id=70631
https://bugs.php.net/bug.php?id=70632

All these bugs linked are very similar to the error's here.

Fixes we should backport to 5.5.9 are

http://git.php.net/?p=php-src.git;a=commitdiff;h=c147d90dbfcc4f9fd494215b7e5740e497d818c1;hp=83bef0ef83f645a6b84ba77b7c69ecdbad68f0a1

http://git.php.net/?p=php-src.git;a=commitdiff;h=4744eec753786e18b28887463bdcb3f1e91efb60;hp=925412ee1c123c55e2b5badf6daf193e6c1bdd5b

http://git.php.net/?p=php-src.git;a=commitdiff;h=2fb8bb1157958465a8b12c1a41015b907787c17a;hp=6876112c89e6dc4434e64a78084ca3bb118c98bd

could make a php 5.5.9 wmf release?

Change 313048 had a related patch set (by Paladox) published:
Correctly disable garbage collection

https://gerrit.wikimedia.org/r/313048

Change 313051 had a related patch set uploaded (by Paladox):
Disable garbage collection for mw-phpunit.sh too

https://gerrit.wikimedia.org/r/313051

Change 313048 abandoned by Hashar:
Correctly disable garbage collection

Reason:
The space does not matter:

Default:

$ php -r 'print ini_get("zend.enable_gc") . "\n";'
1

Without a space:

$ php -dzend.enable_gc=0 -r 'print ini_get("zend.enable_gc") . "\n";'
0

With a space:

$ php -d zend.enable_gc=0 -r 'print ini_get("zend.enable_gc") . "\n";'
0

https://gerrit.wikimedia.org/r/313048

Change 313056 had a related patch set uploaded (by Paladox):
Disable garbage collection for wikibase tests

https://gerrit.wikimedia.org/r/313056

In T142158#2603294, @Krinkle wrote:

Could be unrelated but just noticed another one of these in a similar (?) job:

https://gerrit.wikimedia.org/r/#/c/307729/
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/900/console

............................................................. 2135 / 5399 ( 39%)
................................................./tmp/hudson4524155360135912232.sh: line 6: 
  9997 Segmentation fault      php phpunit.php --log-junit "$WORKSPACE/log/junit-wikidata.xml" --group Wikibase,WikibaseAPI,Purtle
Build step 'Execute shell' marked build as failure

That's because https://gerrit.wikimedia.org/r/#/c/306072/ never applied to wikibase, but this is now fixed in https://gerrit.wikimedia.org/r/#/c/313056/

Thank you for looking into this @Paladox!

Your welcome :)

Change 313051 merged by jenkins-bot:
Disable garbage collection for mw-phpunit.sh too

https://gerrit.wikimedia.org/r/313051

Mentioned in SAL (#wikimedia-releng) [2016-09-28T10:43:23Z] <hashar> Updating slave scripts for "Disable garbage collection for mw-phpunit.sh" https://gerrit.wikimedia.org/r/313051 T142158

Mentioned in SAL (#wikimedia-releng) [2016-09-28T10:44:54Z] <hashar> CI updating all mwext-Wikibase* jenkins jobs for https://gerrit.wikimedia.org/r/#/c/313056/ T142158

Change 313056 merged by jenkins-bot:
Disable garbage collection for wikibase tests

https://gerrit.wikimedia.org/r/313056

@daniel and @hoo the wikibase tests should work now. With the patches that @hashar has merged now :)

@hoo could you revert your hack to see if the segfault has gone with those patches please?

In T142158#2673757, @Paladox wrote:

@daniel and @hoo the wikibase tests should work now. With the patches that @hashar has merged now :)

@hoo could you revert your hack to see if the segfault has gone with those patches please?

Status?

In T142158#3278995, @greg wrote:

In T142158#2673757, @Paladox wrote:

@daniel and @hoo the wikibase tests should work now. With the patches that @hashar has merged now :)

@hoo could you revert your hack to see if the segfault has gone with those patches please?

Status?

.... :)

Change 427330 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: disable garbage collector on Zend php5.5

https://gerrit.wikimedia.org/r/427330

Change 427330 merged by jenkins-bot:
[integration/config@master] docker: disable garbage collector on Zend php5.5

https://gerrit.wikimedia.org/r/427330