MediaWiki phpunit tests SIGSEGV under php5.5 on Jessie
Closed, ResolvedPublic

Description

Running MediaWiki PHPUnit Database tests with docker-registry.wikimedia.org/releng/quibble-jessie-php55:0.0.8 ends up exiting due to a SIGSEGV:

php tests/phpunit/phpunit.php --debug-tests --group Database --exclude-group Broken,ParserFuzz,Stub
Using PHP 5.5.38-4+wmf1+jessie
<...>
................Traceback (most recent call last):
  File "/usr/local/bin/quibble", line 9, in <module>
    load_entry_point('quibble==0.0.0', 'console_scripts', 'quibble')()
  File "/usr/local/lib/python3.4/dist-packages/quibble/cmd.py", line 402, in main
    cmd.execute()
  File "/usr/local/lib/python3.4/dist-packages/quibble/cmd.py", line 397, in execute
    quibble.test.run_phpunit_database(mwdir=self.mw_install_path)
  File "/usr/local/lib/python3.4/dist-packages/quibble/test.py", line 122, in run_phpunit_database
    run_phpunit(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/quibble/test.py", line 117, in run_phpunit
    subprocess.check_call(cmd, cwd=mwdir, env={'LANG': 'C.UTF-8'})
  File "/usr/lib/python3.4/subprocess.py", line 561, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['php', 'tests/phpunit/phpunit.php', '--debug-tests', '--group', 'Database', '--exclude-group', 'Broken,ParserFuzz,Stub']' returned non-zero exit status -11

-11 is Python subprocess.check_call() setting a negative exit code https://bugs.python.org/issue27167 Signal 11 is SIGSEGV: Invalid memory reference / segmentation fault which produces a core dump.

A build with sqlite is not affected, and REL1_30 pass. So something introduced between REL1_30 and master causes the issue. Most probably in the MySQL php5.5 extension.

Found via https://integration.wikimedia.org/ci/job/quibble-integration/ and specially the triggered job https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php55-docker/

Reproduction

docker run --rm -it -v "$(pwd)/ref:/srv/git" -v "$(pwd)/cache:/cache" -v "$(pwd)/workspace:/workspace" docker-registry.wikimedia.org/releng/quibble-jessie-php55:0.0.8 --log-dir=/workspace/log  --run phpunit

I even got a core file!

hashar created this task.Apr 18 2018, 7:55 AM
hashar updated the task description. (Show Details)Apr 18 2018, 7:58 AM
hashar updated the task description. (Show Details)

Change 427326 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Allow selection of builds for quibble-integration

https://gerrit.wikimedia.org/r/427326

Change 427326 merged by jenkins-bot:
[integration/config@master] Allow selection of builds for quibble-integration

https://gerrit.wikimedia.org/r/427326

hashar updated the task description. (Show Details)Apr 18 2018, 8:40 AM

Looking at the core file. Self notes:

docker run -it --user root -v "$(pwd)":/workspace --entrypoint /bin/bash docker-registry.wikimedia.org/releng/quibble-jessie-php55:0.0.8
apt update
apt install gdb
gdb -ex bt ./core /usr/bin/php5.5

Which yields:

Reading symbols from /usr/bin/php5.5...(no debugging symbols found)...done.

warning: core file may not match specified executable file.

warning: [New LWP 748]
.dynamic section for "/lib/x86_64-linux-gnu/libpthread.so.0" is not at the expected address (wrong library or version mismatch?)
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error

warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
To enable execution of this file add
	add-auto-load-safe-path /lib/x86_64-linux-gnu/libthread_db-1.0.so
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error

warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `php tests/phpunit/phpunit.php --debug-tests --group Database --exclude-group Br'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055a2d4c3be48 in zend_std_object_get_class ()
#0  0x000055a2d4c3be48 in zend_std_object_get_class ()
#1  0x000055a2d4c5c669 in ?? ()
#2  0x000055a2d4c43d20 in execute_ex ()
#3  0x000055a2d4c06d88 in dtrace_execute_ex ()
#4  0x000055a2d4cd38bc in ?? ()
#5  0x000055a2d4c43d20 in execute_ex ()
#6  0x000055a2d4c06d88 in dtrace_execute_ex ()
#7  0x000055a2d4cd38bc in ?? ()
#8  0x000055a2d4c43d20 in execute_ex ()
#9  0x000055a2d4c06d88 in dtrace_execute_ex ()
#10 0x000055a2d4c09218 in zend_call_function ()
#11 0x000055a2d4b50953 in zif_call_user_func_array ()
#12 0x000055a2d4c06eea in dtrace_execute_internal ()
#13 0x000055a2d4cd34de in ?? ()
#14 0x000055a2d4c43d20 in execute_ex ()
#15 0x000055a2d4c06d88 in dtrace_execute_ex ()
#16 0x000055a2d4cd38bc in ?? ()
#17 0x000055a2d4c43d20 in execute_ex ()
#18 0x000055a2d4c06d88 in dtrace_execute_ex ()
#19 0x000055a2d4cd38bc in ?? ()
#20 0x000055a2d4c43d20 in execute_ex ()
#21 0x000055a2d4c06d88 in dtrace_execute_ex ()
#22 0x000055a2d4c09218 in zend_call_function ()
#23 0x000055a2d4b50953 in zif_call_user_func_array ()
#24 0x000055a2d4c06eea in dtrace_execute_internal ()
#25 0x000055a2d4cd34de in ?? ()
#26 0x000055a2d4c43d20 in execute_ex ()
#27 0x000055a2d4c06d88 in dtrace_execute_ex ()
#28 0x000055a2d4c09218 in zend_call_function ()
#29 0x000055a2d4b50953 in zif_call_user_func_array ()
#30 0x000055a2d4c06eea in dtrace_execute_internal ()
#31 0x000055a2d4cd34de in ?? ()
#32 0x000055a2d4c43d20 in execute_ex ()
#33 0x000055a2d4c06d88 in dtrace_execute_ex ()
#34 0x000055a2d4cd38bc in ?? ()
#35 0x000055a2d4c43d20 in execute_ex ()
#36 0x000055a2d4c06d88 in dtrace_execute_ex ()
#37 0x000055a2d4cd38bc in ?? ()
#38 0x000055a2d4c43d20 in execute_ex ()
#39 0x000055a2d4c06d88 in dtrace_execute_ex ()
#40 0x000055a2d4cd38bc in ?? ()
#41 0x000055a2d4c43d20 in execute_ex ()
#42 0x000055a2d4c06d88 in dtrace_execute_ex ()
#43 0x000055a2d4c09218 in zend_call_function ()
#44 0x000055a2d4c2fac7 in zend_call_method ()
#45 0x000055a2d4c3b8db in zend_objects_destroy_object ()
#46 0x000055a2d4c3847b in gc_collect_cycles ()
#47 0x000055a2d4c388df in gc_zobj_possible_root ()
#48 0x000055a2d4cd3800 in ?? ()
#49 0x000055a2d4c43d20 in execute_ex ()
#50 0x000055a2d4c06d88 in dtrace_execute_ex ()
#51 0x000055a2d4cd38bc in ?? ()
#52 0x000055a2d4c43d20 in execute_ex ()
#53 0x000055a2d4c06d88 in dtrace_execute_ex ()
#54 0x000055a2d4cd38bc in ?? ()
#55 0x000055a2d4c43d20 in execute_ex ()
#56 0x000055a2d4c06d88 in dtrace_execute_ex ()
#57 0x000055a2d4cd38bc in ?? ()
#58 0x000055a2d4c43d20 in execute_ex ()
#59 0x000055a2d4c06d88 in dtrace_execute_ex ()
#60 0x000055a2d4cd38bc in ?? ()
#61 0x000055a2d4c43d20 in execute_ex ()
#62 0x000055a2d4c06d88 in dtrace_execute_ex ()
#63 0x000055a2d4cd38bc in ?? ()
#64 0x000055a2d4c43d20 in execute_ex ()
#65 0x000055a2d4c06d88 in dtrace_execute_ex ()
#66 0x000055a2d4cd38bc in ?? ()
#67 0x000055a2d4c43d20 in execute_ex ()
#68 0x000055a2d4c06d88 in dtrace_execute_ex ()
#69 0x000055a2d4cd38bc in ?? ()
#70 0x000055a2d4c43d20 in execute_ex ()
#71 0x000055a2d4c06d88 in dtrace_execute_ex ()
#72 0x000055a2d4cd38bc in ?? ()
#73 0x000055a2d4c43d20 in execute_ex ()
#74 0x000055a2d4c06d88 in dtrace_execute_ex ()
#75 0x000055a2d4cd38bc in ?? ()
#76 0x000055a2d4c43d20 in execute_ex ()
#77 0x000055a2d4c06d88 in dtrace_execute_ex ()
#78 0x000055a2d4cd38bc in ?? ()
#79 0x000055a2d4c43d20 in execute_ex ()
#80 0x000055a2d4c06d88 in dtrace_execute_ex ()
#81 0x000055a2d4cd38bc in ?? ()
#82 0x000055a2d4c43d20 in execute_ex ()
#83 0x000055a2d4c06d88 in dtrace_execute_ex ()
#84 0x000055a2d4cd38bc in ?? ()
#85 0x000055a2d4c43d20 in execute_ex ()
#86 0x000055a2d4c06d88 in dtrace_execute_ex ()
#87 0x000055a2d4cd38bc in ?? ()
#88 0x000055a2d4c43d20 in execute_ex ()
#89 0x000055a2d4c06d88 in dtrace_execute_ex ()
#90 0x000055a2d4cd38bc in ?? ()
#91 0x000055a2d4c43d20 in execute_ex ()
#92 0x000055a2d4c06d88 in dtrace_execute_ex ()
#93 0x000055a2d4c09218 in zend_call_function ()
#94 0x000055a2d4b0855b in zim_reflection_method_invokeArgs ()
#95 0x000055a2d4c06eea in dtrace_execute_internal ()
#96 0x000055a2d4cd34de in ?? ()
#97 0x000055a2d4c43d20 in execute_ex ()
#98 0x000055a2d4c06d88 in dtrace_execute_ex ()
#99 0x000055a2d4cd38bc in ?? ()
#100 0x000055a2d4c43d20 in execute_ex ()
#101 0x000055a2d4c06d88 in dtrace_execute_ex ()
#102 0x000055a2d4cd38bc in ?? ()
#103 0x000055a2d4c43d20 in execute_ex ()
#104 0x000055a2d4c06d88 in dtrace_execute_ex ()
#105 0x000055a2d4cd38bc in ?? ()
#106 0x000055a2d4c43d20 in execute_ex ()
#107 0x000055a2d4c06d88 in dtrace_execute_ex ()
#108 0x000055a2d4cd38bc in ?? ()
#109 0x000055a2d4c43d20 in execute_ex ()
#110 0x000055a2d4c06d88 in dtrace_execute_ex ()
#111 0x000055a2d4cd38bc in ?? ()
#112 0x000055a2d4c43d20 in execute_ex ()
#113 0x000055a2d4c06d88 in dtrace_execute_ex ()
#114 0x000055a2d4cd38bc in ?? ()
#115 0x000055a2d4c43d20 in execute_ex ()
#116 0x000055a2d4c06d88 in dtrace_execute_ex ()
#117 0x000055a2d4cd38bc in ?? ()
#118 0x000055a2d4c43d20 in execute_ex ()
#119 0x000055a2d4c06d88 in dtrace_execute_ex ()
#120 0x000055a2d4cd38bc in ?? ()
#121 0x000055a2d4c43d20 in execute_ex ()
#122 0x000055a2d4c06d88 in dtrace_execute_ex ()
#123 0x000055a2d4cd38bc in ?? ()
#124 0x000055a2d4c43d20 in execute_ex ()
#125 0x000055a2d4c06d88 in dtrace_execute_ex ()
#126 0x000055a2d4cd38bc in ?? ()
#127 0x000055a2d4c43d20 in execute_ex ()
#128 0x000055a2d4c06d88 in dtrace_execute_ex ()
#129 0x000055a2d4cd38bc in ?? ()
#130 0x000055a2d4c43d20 in execute_ex ()
#131 0x000055a2d4c06d88 in dtrace_execute_ex ()
#132 0x000055a2d4cd38bc in ?? ()
#133 0x000055a2d4c43d20 in execute_ex ()
#134 0x000055a2d4c06d88 in dtrace_execute_ex ()
#135 0x000055a2d4cd28bc in ?? ()
#136 0x000055a2d4c43d20 in execute_ex ()
#137 0x000055a2d4c06d88 in dtrace_execute_ex ()
#138 0x000055a2d4c18dc0 in zend_execute_scripts ()
#139 0x000055a2d4bb62a0 in php_execute_script ()
#140 0x000055a2d4cd5551 in ?? ()
#141 0x000055a2d4a9743d in main ()

There is the garbage collector mentioned at some point. We had to disable it for php5.5 due to constant segfaults ( T142158 ) by passing: -d zend.enable_gc=0

The parameter is present in the slave scripts bin/mw-phpunit.sh and bin/mw-run-phpunit-allexts.sh but hasn't been ported in Quibble

Change 427330 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: disable garbage collector on Zend php5.5

https://gerrit.wikimedia.org/r/427330

Change 427332 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Update qiubble php55 job to disable zend gc

https://gerrit.wikimedia.org/r/427332

Change 427330 merged by jenkins-bot:
[integration/config@master] docker: disable garbage collector on Zend php5.5

https://gerrit.wikimedia.org/r/427330

hashar closed this task as Resolved.Apr 18 2018, 9:39 AM
hashar claimed this task.
hashar added a subscriber: Legoktm.

php55 with mysql ended up segfaulting due to the Zend garbage collector. A known issue T142158.

https://gerrit.wikimedia.org/r/427330 adds a php ini file to set zend.enable_gc = Off which fix it.

https://gerrit.wikimedia.org/r/#/c/427332/ updates the jenkins jobs

INFO:jenkins_jobs.builder:Reconfiguring jenkins job quibble-composer-mysql-php55-docker
INFO:jenkins_jobs.builder:Reconfiguring jenkins job quibble-composer-sqlite-php55-docker
INFO:jenkins_jobs.builder:Reconfiguring jenkins job quibble-vendor-mysql-php55-docker
INFO:jenkins_jobs.builder:Reconfiguring jenkins job quibble-vendor-sqlite-php55-docker

quibble-composer-mysql-php55-docker passed \o/

Maybe later Quibble could be made to recognizes it is running php5.5 and inject the setting automatically. Shipping it in the CI Docker env is good enough though.

Change 427332 merged by jenkins-bot:
[integration/config@master] Update quibble php55 job to disable zend gc

https://gerrit.wikimedia.org/r/427332

Mentioned in SAL (#wikimedia-releng) [2018-04-18T09:40:26Z] <hashar> Buiding quibble jobs with (master, mysql, php5.5) https://integration.wikimedia.org/ci/job/quibble-integration/7/ T192432