Page MenuHomePhabricator

Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault"
Closed, ResolvedPublic

Description

Over the past 3 hours, I've seen about half a dozen instances where this job fails in the gate-and-submit queue due to a segmentation fault about half-way through the PHPUnit tests.

https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55/6230/console

00:23:08 ...........................................................  7316 / 11984 ( 61%)
00:23:19 ...........................................................  7375 / 11984 ( 61%)
00:23:19 ...........................................................  7434 / 11984 ( 62%)
00:23:36 .........................................../srv/deployment/integration/slave-scripts/bin/mw-run-phpunit-allexts.sh: line 22: 30199 Segmentation fault

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added subscribers: Luke081515, TerraCodes. · View Herald TranscriptAug 5 2016, 12:50 AM

Do we have a core dump?

Paladox added a subscriber: Paladox.Aug 5 2016, 4:23 PM
greg lowered the priority of this task from Unbreak Now! to High.Aug 8 2016, 4:20 PM
greg added a subscriber: greg.

It's not happening across instances consistently. It's not happening on specific patches consistently. Reducing priority for now.

If you run into this, please try rechecking (as a comment in Gerrit) to get around it.

If anyone does run into this, please try to reproduce this locally and report back what you find.

bd808 added a subscriber: bd808.Aug 8 2016, 4:23 PM

Do we have a core dump?

Getting the core file is going to be the only real way to debug this. The most common cause of a PHP segfault is a double free bug in a PECL extension. These can be really sneaky to track down.

hoo added a subscriber: hoo.Aug 11 2016, 12:06 PM

This is really annoying as it happens far more often than not.

I've seen similar segfaults on travis when running the Wikibase tests there in the last couple of days (on php55 only), but they went away again a few days ago after consistently occurring for a few days (for example https://travis-ci.org/wikimedia/mediawiki-extensions-Wikibase/jobs/150951279). (Note: We use MediaWiki master there to run our tests)

Maybe it is mediawiki coursing the segfault?

daniel added a subscriber: daniel.

dmesg on integration-slave-trusty-1001 reports we are consistently segfaulting at the same instruction pointer:

[12600829.516010] traps: php5[3756] general protection ip:70c450 sp:7ffd02145a28 error:0 in php5[400000+7f3000]
[13109969.662961] php5[31621]: segfault at 3 ip 000000000070c450 sp 00007fff7486b2b8 error 6 in php5[400000+7f3000]
[13121365.417743] php5[29445]: segfault at 3 ip 000000000070c450 sp 00007ffd296d1a68 error 6 in php5[400000+7f3000]
[13126836.791557] traps: php5[2279] general protection ip:70c450 sp:7fff7281d078 error:0 in php5[400000+7f3000]
[13138327.663403] php5[28120]: segfault at 3 ip 000000000070c450 sp 00007ffe2d34be48 error 6 in php5[400000+7f3000]
[13139900.720581] php5[1421]: segfault at 3 ip 000000000070c450 sp 00007ffca4008a98 error 6 in php5[400000+7f3000]

Using addr2line we can see this is in the garbage collector

ebernhardson@integration-slave-trusty-1001:~$ ~/addr2line -e $(which php5) 70c450
/build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h:189

Specifically that bit of code is the first line of this function (from php 5.5.9 sources):

static zend_always_inline void gc_remove_from_buffer(gc_root_buffer *root TSRMLS_DC)
{
    root->next->prev = root->prev;
    root->prev->next = root->next;
    root->prev = GC_G(unused);
    GC_G(unused) = root;
    GC_BENCH_DEC(root_buf_length);
}

Checking the php bugtracker there are a few different fatals/errors related to garbage collection and gc_remove_from_buffer. We could try a few of them out, but it would take a bit of effort to test the various patches until we find the relevant patch. As a workaround, i believe we can disable the garbage collector when running tests and things might "just work". Incoming patch that will set this for the mediawiki-extensions-php55 job, and if it does the trick we can also set it elsewhere. refcounted GC should still work with this disabled, only the cycle collector will be disabled.

Change 306072 had a related patch set uploaded (by EBernhardson):
Try disable garbage collection to prevent segfaults

https://gerrit.wikimedia.org/r/306072

should note that disabling gc is just a workaround,

16:19 <+SMalyshev> the thing is it may not even be in GC... GC is just a thing that everyone dumps it's vars on it, so if there's a corrupt var somewhere, GC will get it eventually

Change 306072 merged by jenkins-bot:
Try disable garbage collection to prevent segfaults

https://gerrit.wikimedia.org/r/306072

Looking at the overnight runs of mediawiki-extensions-php55, the problem seems to have subsided with the deployment of the above patch.

Could be unrelated but just noticed another one of these in a similar (?) job:

https://gerrit.wikimedia.org/r/#/c/307729/
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/900/console

............................................................. 2135 / 5399 ( 39%)
................................................./tmp/hudson4524155360135912232.sh: line 6: 
  9997 Segmentation fault      php phpunit.php --log-junit "$WORKSPACE/log/junit-wikidata.xml" --group Wikibase,WikibaseAPI,Purtle
Build step 'Execute shell' marked build as failure
hoo added a comment.Sep 14 2016, 5:36 PM

I also faced segfaults today in https://gerrit.wikimedia.org/r/#/c/309972/4 in the mwext-Wikibase-repo-tests-sqlite-php55 job that are potentially related:

https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/973/console
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/972/console
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/970/console

I tried to narrow down the cause, by removing all Wikibase repo tests but the test changed in that patch, but that also made the jobs pass, so the only test touched in that change is not responsible (alone).

The workaround in https://gerrit.wikimedia.org/r/#/c/309972/6/repo/tests/phpunit/includes/Notifications/JobQueueChangeNotificationSenderTest.php (in setUp) makes the changes pass as well, which is in contrast to the findings above.

hashar raised the priority of this task from High to Unbreak Now!.Sep 14 2016, 9:13 PM
hashar edited projects, added Release-Engineering-Team; removed Patch-For-Review.
hashar added a subscriber: hashar.

So that happens on a bunch of other patches as well. This evening Jonas_WMDE had troubles to get Wikibase patches to land, so I guess hoo patch is unrelated :-(

Stuff to figure out:

  • when did it start (Jenkins build trend helps)
  • which slaves?
  • check updates on slaves (we auto upgrade .deb packages for example)
  • reproduce (hook on a slave, copy workspace to home dir, load envs variables, run phpunit.php
  • figure out how to get a core file for investigation with gdb

Then it is going a nasty bisect session to narrow down the root cause :-(

Restricted Application added a subscriber: Jay8g. · View Herald TranscriptSep 14 2016, 9:13 PM

T64623 was the previous PHP segfault debugging session. With the very useful patch by Faidon: https://gerrit.wikimedia.org/r/#/c/119222/

Change 310673 had a related patch set uploaded (by Hashar):
Reapply "Dump corefiles on segfaults"

https://gerrit.wikimedia.org/r/310673

Mentioned in SAL (#wikimedia-releng) [2016-09-14T21:37:26Z] <hashar> integration: setting "ulimit -c 2097152" on all slaves due to Zend PHP segfaulting T142158

Change 310673 merged by jenkins-bot:
Reapply "Dump corefiles on segfaults"

https://gerrit.wikimedia.org/r/310673

2GBytes core dump files are now enabled. They should land on the slaves as /var/tmp/core/core.*.

There is a bunch caused by lua. I have removed them all to start again with a clean state.

Command to check for core from integration-saltmaster :

salt -v '*trusty*' cmd.run 'ls -lrta /var/tmp/core/core.*'

Quickly looked at one on integration-slave-trusty-1013

gdb /usr/bin/php5 /var/tmp/core/core.integration-slave-trusty-1013.php5.27576.1473889797
Reading symbols from /usr/bin/php5...Reading symbols from /usr/lib/debug//usr/bin/php5...done.
[New LWP 27576]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).

Mysql is not happy :/

Core was generated by `php5 phpunit.php --log-junit /mnt/jenkins-workspace/workspace/mwext-Wikibase-re'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000070c450 in gc_remove_from_buffer (root=0x23993500) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h:189
189     /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h: No such file or directory.
(gdb)

So gc_remove_from_buffer which sounds familiar with T64623. We had gc disabled for tests.

The 154 lines stacktrace is in P4049

1#0 0x000000000070c450 in gc_remove_from_buffer (root=0x23993500) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h:189
2#1 gc_remove_zval_from_buffer (zv=zv@entry=0x245ccf58) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.c:265
3#2 0x00000000007906a8 in i_zval_ptr_dtor (zval_ptr=0x245ccf58) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_execute.h:80
4#3 ZEND_ASSIGN_DIM_SPEC_VAR_CV_HANDLER (execute_data=0x7fe61d9abff0) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:23297
5#4 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9abff0) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
6#5 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
7#6 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9abff0) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
8#7 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9abb98) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
9#8 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9abb98) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
10#9 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
11#10 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9abb98) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
12#11 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9ab800) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
13#12 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9ab800) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
14#13 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
15#14 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9ab800) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
16#15 0x00000000006dfdc1 in zend_call_function (fci=fci@entry=0x7ffe3f70b7e0, fci_cache=<optimized out>, fci_cache@entry=0x7ffe3f70b7b0) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_execute_API.c:939
17#16 0x0000000000704905 in zend_call_method (object_pp=object_pp@entry=0x7ffe3f70b898, obj_ce=<optimized out>, fn_proxy=fn_proxy@entry=0x243e3530, function_name=function_name@entry=0xb33fa3 "offsetSet", function_name_len=function_name_len@entry=9, retval_ptr_ptr=retval_ptr_ptr@entry=0x0,
18 param_count=param_count@entry=2, arg1=arg1@entry=0x244c58b8, arg2=arg2@entry=0x21c26398) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_interfaces.c:97
19#17 0x00000000005ee0ac in spl_array_write_dimension_ex (check_inherited=<optimized out>, object=0x2461b6a8, offset=0x244c58b8, value=0x21c26398) at /build/php5-i8TFRg/php5-5.5.9+dfsg/ext/spl/spl_array.c:443
20#18 0x000000000078a6a0 in zend_assign_to_object (key=0x0, opcode=147, execute_data=0x7fe61d9ab708, value_op=<optimized out>, value_type=<optimized out>, property_name=0x0, object_ptr=0x7fe61d9ab7a0, retval=0x0) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_execute.c:745
21#19 ZEND_ASSIGN_DIM_SPEC_CV_UNUSED_HANDLER (execute_data=0x7fe61d9ab708) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:38364
22#20 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9ab708) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
23#21 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
24#22 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9ab708) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
25#23 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9ab5a8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
26#24 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9ab5a8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
27#25 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
28#26 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9ab5a8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
29#27 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9ab188) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
30#28 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9ab188) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
31#29 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
32#30 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9ab188) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
33#31 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9aaaa8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
34#32 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9aaaa8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
35#33 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
36#34 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9aaaa8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
37#35 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9aa758) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
38#36 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9aa758) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
39#37 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
40#38 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9aa758) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
41#39 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9aa558) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
42#40 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9aa558) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
43#41 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
44#42 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9aa558) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
45#43 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9aa2c8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
46#44 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9aa2c8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
47#45 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
48#46 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9aa2c8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
49#47 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a9e88) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
50#48 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a9e88) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
51#49 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
52#50 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a9e88) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
53#51 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a9088) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
54#52 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a9088) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
55#53 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
56#54 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a9088) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
57#55 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a8a70) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
58#56 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a8a70) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
59#57 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
60#58 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a8a70) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
61#59 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a8648) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
62#60 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a8648) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
63#61 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
64#62 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a8648) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
65#63 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a82e8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
66#64 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a82e8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
67#65 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
68#66 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a82e8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
69#67 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a7218) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
70#68 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a7218) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
71#69 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
72#70 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a7218) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
73#71 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a6278) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
74#72 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a6278) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
75#73 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
76#74 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a6278) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
77#75 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a5be8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
78#76 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a5be8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
79#77 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
80#78 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a5be8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
81#79 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a56c8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
82#80 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a56c8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
83#81 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
84#82 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a56c8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
85#83 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a5168) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
86#84 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a5168) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
87#85 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
88#86 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a5168) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
89#87 0x00000000006dfdc1 in zend_call_function (fci=fci@entry=0x7ffe3f70de40, fci_cache=<optimized out>, fci_cache@entry=0x7ffe3f70de10) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_execute_API.c:939
90#88 0x0000000000580676 in zim_reflection_method_invokeArgs (ht=<optimized out>, return_value=0x246ee468, return_value_ptr=<optimized out>, this_ptr=<optimized out>, return_value_used=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/ext/reflection/php_reflection.c:3015
91#89 0x00000000006ddbcb in dtrace_execute_internal (execute_data_ptr=<optimized out>, fci=<optimized out>, return_value_used=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:97
92#90 0x00007fe61af03476 in xdebug_execute_internal (current_execute_data=0x7fe61d9a4e08, fci=0x0, return_value_used=1) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1551
93#91 0x000000000079cf60 in zend_do_fcall_common_helper_SPEC (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:552
94#92 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a4e08) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
95#93 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
96#94 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a4e08) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
97#95 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a4170) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
98#96 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a4170) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
99#97 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
100#98 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a4170) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
101#99 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a2ca8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
102#100 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a2ca8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
103#101 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
104#102 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a2ca8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
105#103 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9a10e8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
106#104 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9a10e8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
107#105 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
108#106 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9a10e8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
109#107 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d99fa68) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
110#108 0x0000000000717898 in execute_ex (execute_data=0x7fe61d99fa68) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
111#109 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
112#110 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d99fa68) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
113#111 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d99f1b8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
114#112 0x0000000000717898 in execute_ex (execute_data=0x7fe61d99f1b8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
115#113 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
116#114 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d99f1b8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
117#115 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d99e148) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
118#116 0x0000000000717898 in execute_ex (execute_data=0x7fe61d99e148) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
119#117 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
120#118 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d99e148) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
121#119 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d99d0d8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
122#120 0x0000000000717898 in execute_ex (execute_data=0x7fe61d99d0d8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
123#121 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
124#122 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d99d0d8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
125#123 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d99c068) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
126#124 0x0000000000717898 in execute_ex (execute_data=0x7fe61d99c068) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
127#125 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
128#126 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d99c068) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
129#127 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d99afb8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
130#128 0x0000000000717898 in execute_ex (execute_data=0x7fe61d99afb8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
131#129 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
132#130 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d99afb8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
133#131 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d9976d0) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
134#132 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9976d0) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
135#133 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
136#134 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9976d0) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
137#135 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d996d98) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
138#136 0x0000000000717898 in execute_ex (execute_data=0x7fe61d996d98) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
139#137 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
140#138 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d996d98) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
141#139 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d996b70) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
142#140 0x0000000000717898 in execute_ex (execute_data=0x7fe61d996b70) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
143#141 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
144#142 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d996b70) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
145#143 0x000000000079d5b0 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe61d995ca8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:584
146#144 0x0000000000717898 in execute_ex (execute_data=0x7fe61d995ca8) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
147#145 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
148#146 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d995ca8) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
149#147 0x000000000079c4e1 in ZEND_INCLUDE_OR_EVAL_SPEC_TMP_HANDLER (execute_data=0x7fe61d9951a0) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:8053
150#148 0x0000000000717898 in execute_ex (execute_data=0x7fe61d9951a0) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_vm_execute.h:363
151#149 0x00000000006ddac9 in dtrace_execute_ex (execute_data=<optimized out>) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_dtrace.c:73
152#150 0x00007fe61af02a7c in xdebug_execute_ex (execute_data=0x7fe61d9951a0) at /build/buildd/xdebug-2.2.3/xdebug-2.2.3/xdebug.c:1437
153#151 0x00000000006ef150 in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend.c:1316
154#152 0x000000000068f9b5 in php_execute_script (primary_file=primary_file@entry=0x7ffe3f712410) at /build/php5-i8TFRg/php5-5.5.9+dfsg/main/main.c:2506
155#153 0x000000000079ef3e in do_cli (argc=6, argv=0x128c310) at /build/php5-i8TFRg/php5-5.5.9+dfsg/sapi/cli/php_cli.c:994
156#154 0x0000000000461dc0 in main (argc=6, argv=0x128c310) at /build/php5-i8TFRg/php5-5.5.9+dfsg/sapi/cli/php_cli.c:1378

Might be worth trying disabling the gc again such as with https://gerrit.wikimedia.org/r/#/c/127485/1/tests/phpunit/phpunit.php

Prepare a patch for mediawiki/core phpunit.php with:

	register_shutdown_function( function() {
		gc_collect_cycles();
		gc_disable();
	} );

Create a change for Wikibase that Depends-On. Rebuild a dozen a time via the Jenkins UI. Check whether there are still segfaults.

And here is a night worth of core files:

`
root@integration-saltmaster:~# salt -v '*trusty*' cmd.run 'ls -hlrta /var/tmp/core/core.*'

integration-slave-trusty-1011.integration.eqiad.wmflabs:
 595M Sep 14 21:49 /var/tmp/core/core.integration-slave-trusty-1011.php5.21391.1473889766
 595M Sep 15 04:18 /var/tmp/core/core.integration-slave-trusty-1011.php5.15988.1473913130
integration-slave-trusty-1018.integration.eqiad.wmflabs:
 596M Sep 15 04:08 /var/tmp/core/core.integration-slave-trusty-1018.php5.15464.1473912526

integration-slave-trusty-1004.integration.eqiad.wmflabs:
 596M Sep 15 04:40 /var/tmp/core/core.integration-slave-trusty-1004.php5.28493.1473914446

integration-slave-trusty-1013.integration.eqiad.wmflabs:
595M Sep 14 21:49 /var/tmp/core/core.integration-slave-trusty-1013.php5.27576.1473889797
`
hashar added a comment.EditedSep 15 2016, 9:58 AM

I have added a gdb script from puppet modules/admin/files/home/hashar/gdbinit . Put it to /mnt/home/jenkins-deploy

Then start gdb and source gdbinit. That gives a bunch of commands that are listed with help user-defined.

So:

integration-slave-trusty-1018:~$ gdb /usr/bin/php5 /var/tmp/core/core.integration-slave-trusty-1018.php5.15464.1473912526

Core was generated by `php5 phpunit.php --log-junit /mnt/jenkins-workspace/workspace/mwext-Wikibase-re'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000070c450 in gc_remove_from_buffer (root=0x224926c0) at /build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h:189
189	/build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h: No such file or directory.

(gdb) source gbinit
(gdb) set logging on
Copying output to gdb.txt.
(gdb) zbacktrace

End result is pages of stack traces with the PHP file and line number. Will paste soonish

And the PHP aware stacktrace is P4052

1Relative to /mnt/jenkins-workspace/workspace/mwext-Wikibase-repo-tests-sqlite-php55/src/
2
3[0x7f87a6cf7ff0] preSetElement() includes/site/SiteList.php:89
4[0x7f87a6cf7b98] SiteList->preSetElement(2, object[0x22421988]) includes/libs/GenericArrayObject.php:155
5[0x7f87a6cf7800] GenericArrayObject->setElement(NULL, object[0x22421988]) includes/libs/GenericArrayObject.php:109
6[0x7ffd1722ae80] GenericArrayObject->offsetSet(NULL, object[0x22421988])
7[0x7f87a6cf7708] setSite() includes/site/SiteList.php:264
8[0x7f87a6cf75a8] SiteList->setSite(object[0x22421988]) extensions/Wikibase/view/src/SiteLinksView.php:216
9[0x7f87a6cf7188] Wikibase\View\SiteLinksView->getSitesForGroup("wikipedia") extensions/Wikibase/view/src/SiteLinksView.php:164
10[0x7f87a6cf6aa8] Wikibase\View\SiteLinksView->getHtmlForSiteLinkGroup(array(0)[0xe20ba70], object[0x246ec9a0], "wikipedia") extensions/Wikibase/view/src/SiteLinksView.php:123
11[0x7f87a6cf6758] Wikibase\View\SiteLinksView->getHtml(array(0)[0xe20ba70], object[0x246ec9a0], array(6)[0x7f8796d85f18]) extensions/Wikibase/view/src/ItemView.php:124
12[0x7f87a6cf6558] Wikibase\View\ItemView->getHtmlForSiteLinks(object[0x2477f280]) extensions/Wikibase/view/src/ItemView.php:107
13[0x7f87a6cf62c8] Wikibase\View\ItemView->getSideHtml(object[0x2477f280]) extensions/Wikibase/view/src/EntityView.php:84
14[0x7f87a6cf5e88] Wikibase\View\EntityView->getHtml(object[0x2477f280]) extensions/Wikibase/repo/includes/ParserOutput/EntityParserOutputGenerator.php:351
15[0x7f87a6cf5088] Wikibase\Repo\ParserOutput\EntityParserOutputGenerator->addHtmlToParserOutput(object[0x241afba8], object[0x2477f280], object[0x24612eb8]) extensions/Wikibase/repo/includes/ParserOutput/EntityParserOutputGenerator.php:168
16[0x7f87a6cf4a70] Wikibase\Repo\ParserOutput\EntityParserOutputGenerator->getParserOutput(object[0x2477f280], true) extensions/Wikibase/repo/includes/Content/EntityContent.php:294
17[0x7f87a6cf4648] Wikibase\EntityContent->getParserOutputFromEntityView(object[0x246ab970], NULL, object[0x2483b458], true) extensions/Wikibase/repo/includes/Content/EntityContent.php:232
18[0x7f87a6cf42e8] Wikibase\EntityContent->getParserOutput(object[0x246ab970], NULL, object[0x2483b458]) includes/page/WikiPage.php:2193
19[0x7f87a6cf3218] WikiPage->prepareContentForEdit(object[0x11367d08], NULL, object[0x11367d98], NULL, true) includes/page/WikiPage.php:1712
20[0x7f87a6cf2278] WikiPage->doEditContent(object[0x24865cc0], "test", 64, false, object[0x248cf418]) extensions/Wikibase/repo/includes/Store/Sql/WikiPageEntityStore.php:266
21[0x7f87a6cf1be8] Wikibase\Repo\Store\WikiPageEntityStore->saveEntityContent(object[0x24865cc0], "test", object[0x248cf418], 0, false) extensions/Wikibase/repo/includes/Store/Sql/WikiPageEntityStore.php:166
22[0x7f87a6cf16c8] Wikibase\Repo\Store\WikiPageEntityStore->saveEntity(object[0x2477f280], "test", object[0x248cf418]) extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php:407
23[0x7f87a6cf1168] Wikibase\Test\EntityContentTest->createTitleForEntity(object[0x2477f280]) extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php:431
24[0x7ffd1722d500] Wikibase\Test\EntityContentTest->testGetDeletionUpdates()
25[0x7f87a6cf0e08] ReflectionMethod->invokeArgs(object[0x246e4d78], array(0)[0x24810200]) vendor/phpunit/phpunit/src/Framework/TestCase.php:909
26[0x7f87a6cf0170] PHPUnit_Framework_TestCase->runTest() vendor/phpunit/phpunit/src/Framework/TestCase.php:768
27[0x7f87a6ceeca8] PHPUnit_Framework_TestCase->runBare() vendor/phpunit/phpunit/src/Framework/TestResult.php:612
28[0x7f87a6ced0e8] PHPUnit_Framework_TestResult->run(object[0x246e4d78]) vendor/phpunit/phpunit/src/Framework/TestCase.php:724
29[0x7f87a6ceba68] PHPUnit_Framework_TestCase->run(object[0x146520b0]) tests/phpunit/MediaWikiTestCase.php:388
30[0x7f87a6ceb1b8] MediaWikiTestCase->run(object[0x146520b0]) vendor/phpunit/phpunit/src/Framework/TestSuite.php:747
31[0x7f87a6cea148] PHPUnit_Framework_TestSuite->run(object[0x146520b0]) vendor/phpunit/phpunit/src/Framework/TestSuite.php:747
32[0x7f87a6ce90d8] PHPUnit_Framework_TestSuite->run(object[0x146520b0]) vendor/phpunit/phpunit/src/Framework/TestSuite.php:747
33[0x7f87a6ce8068] PHPUnit_Framework_TestSuite->run(object[0x146520b0]) vendor/phpunit/phpunit/src/Framework/TestSuite.php:747
34[0x7f87a6ce6fb8] PHPUnit_Framework_TestSuite->run(object[0x146520b0]) vendor/phpunit/phpunit/src/TextUI/TestRunner.php:440
35[0x7f87a6ce36d0] PHPUnit_TextUI_TestRunner->doRun(object[0x21f10d0], array(7)[0x16883d50]) vendor/phpunit/phpunit/src/TextUI/Command.php:149
36[0x7f87a6ce2d98] PHPUnit_TextUI_Command->run(array(7)[0x373a2a0], true) vendor/phpunit/phpunit/src/TextUI/Command.php:100
37[0x7f87a6ce2b70] PHPUnit_TextUI_Command::main() tests/phpunit/phpunit.php:133
38[0x7f87a6ce1ca8] PHPUnitMaintClass->execute() maintenance/doMaintenance.php:110
39[0x7f87a6ce11a0] ??? tests/phpunit/phpunit.php:163

Looks like it is whatever test defined at extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php:431

It eventually reach a code path in includes/site/SiteList.php which is class SiteList extends GenericArrayObject. And that explodes on line 89:

79     protected function preSetElement( $index, $site ) {
80         if ( $this->hasSite( $site->getGlobalId() ) ) {
81             $this->removeSite( $site->getGlobalId() );
82         }
83 
84         $this->byGlobalId[$site->getGlobalId()] = $index;
85         $this->byInternalId[$site->getInternalId()] = $index;
86         
87         $ids = $site->getNavigationIds();
88         foreach ( $ids as $navId ) {
89             $this->byNavigationId[$navId] = $index;   # SEGFAULT!!!!!!
90         }
91         
92         return true;
93     }

No clue how to reliably reproduce it though :-(

aude added a subscriber: aude.EditedSep 19 2016, 12:42 PM

at the moment, i am running php 5.5.9-1ubuntu4.19 and am getting segmentation fault when I run wikibase repo tests locally (but not always)

aude added a comment.Sep 19 2016, 1:17 PM

seems I am able to reproduce only when running the tests on a fresh checkout of a branch (of core + wikibase)

on jenkins, it could be sometimes that a jenkins slave is using an existing checkout of mediawiki + wikibase and sometimes has to do fresh checkout.

maybe the segmentation fault is happening when there is garbage collection

aude added a comment.EditedSep 19 2016, 1:51 PM

with set env MALLOC_CHECK_=3 in gdb, I get:

Wikibase\Test\StoreTest::testRebuild with data set #0 (Wikibase\SqlStore Object (...))
Cannot use a scalar value as an array

/var/www/wiki/w/includes/site/SiteList.php:89
/var/www/wiki/w/includes/libs/GenericArrayObject.php:155
/var/www/wiki/w/includes/libs/GenericArrayObject.php:109
/var/www/wiki/w/includes/site/SiteList.php:264
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/SiteLinksView.php:216
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/SiteLinksView.php:164
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/SiteLinksView.php:123
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/ItemView.php:124
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/ItemView.php:107
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/view/src/EntityView.php:84
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/includes/ParserOutput/EntityParserOutputGenerator.php:351
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/includes/ParserOutput/EntityParserOutputGenerator.php:168
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php:294
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php:232
/var/www/wiki/w/includes/page/WikiPage.php:2182
/var/www/wiki/w/includes/page/WikiPage.php:2265
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php:258
/var/www/wiki/w/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Store/StoreTest.php:60
/var/www/wiki/w/tests/phpunit/MediaWikiTestCase.php:388
/var/www/wiki/w/maintenance/doMaintenance.php:110
greg added a comment.Sep 26 2016, 10:58 PM

@hashar you set this to UBN! back on Sept 14th. Is that still the case? (I know I'm not suppose to distract you from your other work ;) ).

hashar lowered the priority of this task from Unbreak Now! to High.Sep 27 2016, 11:05 AM

That has been worked around by skipping the faulty test whenever it is found to run via a specific Jenkins job. So at least is no more randomly blocking everyone (since the change got merged).

repo/tests/phpunit/includes/Notifications/JobQueueChangeNotificationSenderTest.php
class JobQueueChangeNotificationSenderTest extends PHPUnit_Framework_TestCase {

	public function setUp() {
		parent::setUp();

		$isFaultyJob = strpos(
			__DIR__,
			'mwext-Wikibase-repo-tests-sqlite-php55'
		) > 1;
		if ( $isFaultyJob ) {
			$this->markTestSkipped(
				'This test causes segfaults in mwext-Wikibase-repo-tests-sqlite-php55 jobs, skipping.'
			);
		}
	}

Still have to figure out the reason for the segfault but I would blame the Zend garbage collector there. We used to have it disabled (see above T142158#2638684 ). Might also look at Zend 5.5 patches which we might want to cherry pick in our Debian package.

Paladox added a comment.EditedSep 27 2016, 4:35 PM

We could do https://bugs.php.net/bugs-generating-backtrace.php and run it on the specific file that causes the fault to get a trace, and then possibly try and find similar errors to that and if we doint find any similar, report it.

I'm wondering could it be this bug https://bugs.php.net/bug.php?id=72286

Since that also affects 5.5 but not 7.0?

Apparently it is fixed in one of the php 5.5 releases but still doesn't work with php 5.6.

Also they are saying the problem is reproducible if we have opcache disabled.

Or we can add zend.enable_gc=0 to php.ini

or in phpunit do

phpunit -d zend.enable_gc=0

dmesg on integration-slave-trusty-1001 reports we are consistently segfaulting at the same instruction pointer:

[12600829.516010] traps: php5[3756] general protection ip:70c450 sp:7ffd02145a28 error:0 in php5[400000+7f3000]
[13109969.662961] php5[31621]: segfault at 3 ip 000000000070c450 sp 00007fff7486b2b8 error 6 in php5[400000+7f3000]
[13121365.417743] php5[29445]: segfault at 3 ip 000000000070c450 sp 00007ffd296d1a68 error 6 in php5[400000+7f3000]
[13126836.791557] traps: php5[2279] general protection ip:70c450 sp:7fff7281d078 error:0 in php5[400000+7f3000]
[13138327.663403] php5[28120]: segfault at 3 ip 000000000070c450 sp 00007ffe2d34be48 error 6 in php5[400000+7f3000]
[13139900.720581] php5[1421]: segfault at 3 ip 000000000070c450 sp 00007ffca4008a98 error 6 in php5[400000+7f3000]

Using addr2line we can see this is in the garbage collector

ebernhardson@integration-slave-trusty-1001:~$ ~/addr2line -e $(which php5) 70c450
/build/php5-i8TFRg/php5-5.5.9+dfsg/Zend/zend_gc.h:189

Specifically that bit of code is the first line of this function (from php 5.5.9 sources):

static zend_always_inline void gc_remove_from_buffer(gc_root_buffer *root TSRMLS_DC)
{
    root->next->prev = root->prev;
    root->prev->next = root->next;
    root->prev = GC_G(unused);
    GC_G(unused) = root;
    GC_BENCH_DEC(root_buf_length);
}

Checking the php bugtracker there are a few different fatals/errors related to garbage collection and gc_remove_from_buffer. We could try a few of them out, but it would take a bit of effort to test the various patches until we find the relevant patch. As a workaround, i believe we can disable the garbage collector when running tests and things might "just work". Incoming patch that will set this for the mediawiki-extensions-php55 job, and if it does the trick we can also set it elsewhere. refcounted GC should still work with this disabled, only the cycle collector will be disabled.

Other users seem to be having this problem

http://www.serverphorums.com/read.php?7,834007 looks very similar to the line you pointed at.

Maybe

  • "(Segmentation fault in gc_remove_zval_from_buffer)."

https://bugs.php.net/bug.php?id=67314

?

Paladox added a comment.EditedSep 27 2016, 5:18 PM

I belive I have found the fixes for this.

Fixed in php 5.6.15, per changelog

1•Opcache: •Fixed bug #70632 (Third one of segfault in gc_remove_from_buffer).
2•Fixed bug #70631 (Another Segfault in gc_remove_from_buffer()).
3•Fixed bug #70601 (Segfault in gc_remove_from_buffer()).
4•Fixed compatibility with Windows 10 (see also #70652).

We should setup a php 5.6 test for the test that fails and try it in there using php 5.6.15.

Change 313048 had a related patch set (by Paladox) published:
Correctly disable garbage collection

https://gerrit.wikimedia.org/r/313048

Change 313051 had a related patch set uploaded (by Paladox):
Disable garbage collection for mw-phpunit.sh too

https://gerrit.wikimedia.org/r/313051

Change 313048 abandoned by Hashar:
Correctly disable garbage collection

Reason:
The space does not matter:

Default:

$ php -r 'print ini_get("zend.enable_gc") . "\n";'
1

Without a space:

$ php -dzend.enable_gc=0 -r 'print ini_get("zend.enable_gc") . "\n";'
0

With a space:

$ php -d zend.enable_gc=0 -r 'print ini_get("zend.enable_gc") . "\n";'
0

https://gerrit.wikimedia.org/r/313048

Change 313056 had a related patch set uploaded (by Paladox):
Disable garbage collection for wikibase tests

https://gerrit.wikimedia.org/r/313056

Could be unrelated but just noticed another one of these in a similar (?) job:

https://gerrit.wikimedia.org/r/#/c/307729/
https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-php55/900/console

............................................................. 2135 / 5399 ( 39%)
................................................./tmp/hudson4524155360135912232.sh: line 6: 
  9997 Segmentation fault      php phpunit.php --log-junit "$WORKSPACE/log/junit-wikidata.xml" --group Wikibase,WikibaseAPI,Purtle
Build step 'Execute shell' marked build as failure

That's because https://gerrit.wikimedia.org/r/#/c/306072/ never applied to wikibase, but this is now fixed in https://gerrit.wikimedia.org/r/#/c/313056/

Thank you for looking into this @Paladox!

Your welcome :)

Change 313051 merged by jenkins-bot:
Disable garbage collection for mw-phpunit.sh too

https://gerrit.wikimedia.org/r/313051

Mentioned in SAL (#wikimedia-releng) [2016-09-28T10:43:23Z] <hashar> Updating slave scripts for "Disable garbage collection for mw-phpunit.sh" https://gerrit.wikimedia.org/r/313051 T142158

Mentioned in SAL (#wikimedia-releng) [2016-09-28T10:44:54Z] <hashar> CI updating all mwext-Wikibase* jenkins jobs for https://gerrit.wikimedia.org/r/#/c/313056/ T142158

Change 313056 merged by jenkins-bot:
Disable garbage collection for wikibase tests

https://gerrit.wikimedia.org/r/313056

Paladox added a comment.EditedSep 28 2016, 10:51 AM

@daniel and @hoo the wikibase tests should work now. With the patches that @hashar has merged now :)

@hoo could you revert your hack to see if the segfault has gone with those patches please?

Krinkle removed a subscriber: Krinkle.Sep 29 2016, 7:00 PM
greg added a comment.May 20 2017, 12:21 PM

@daniel and @hoo the wikibase tests should work now. With the patches that @hashar has merged now :)

@hoo could you revert your hack to see if the segfault has gone with those patches please?

Status?

greg added a comment.Jun 13 2017, 11:56 PM

@daniel and @hoo the wikibase tests should work now. With the patches that @hashar has merged now :)

@hoo could you revert your hack to see if the segfault has gone with those patches please?

Status?

.... :)

Addshore removed a subscriber: Addshore.Jun 16 2017, 8:42 AM
Krinkle moved this task from Inbox to PHPUnit on the MediaWiki-Core-Testing board.Jul 7 2017, 5:11 AM

Change 427330 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: disable garbage collector on Zend php5.5

https://gerrit.wikimedia.org/r/427330

Change 427330 merged by jenkins-bot:
[integration/config@master] docker: disable garbage collector on Zend php5.5

https://gerrit.wikimedia.org/r/427330