xdebug causes segmentation fault in PHP when hitting a breakpoint in a phpunit test
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	daniel
	Jun 6 2023, 8:12 AM

Description

Steps to reproduce:

In PhpStorm, set a break point in a phpunit test like TitleTest.php. Doesn't matter where, setUp() will do.
Execute the test case in a Docker container (per MediaWiki's docker-compose file), with Xdebug enabled. You can use PhpStorm integration run it from the command line, it doesn't matter.
Observe the php process crashing with a segmentation fault.

Observations:

Any test class based on MediaWikiUnitTestCase or MediaWikiIntegrationTestCase will trigger the error.
Test classes based directly on phpunit's TestCase do not, even when adding the MediaWikiCoversValidator and MediaWikiTestCaseTrait traits.
Setting a breakpoint in the code executed from the test case works most of the time.
Attempts to examine the stack frame for a method in the test class will cause the segmentation fault.
Disabling the __toString object view in the debugger settings does not help.

Environment:

OS	Linux 5.15.90.1-microsoft-standard-WSL2 (WSL-Version: 1.2.5.0)
PhpStorm	2022.2.3, Build #PS-222.4345.15
Docker Desktop	v4.17.0
Image	docker-registry.wikimedia.org/dev/buster-php81-fpm 1.0.0 (549543a10e3d)
PHP	8.1.13 (cli) (built: Nov 26 2022 14:29:42) (NTS)
Zend Engine	v4.1.13
Zend OPcache	v8.1.13
Xdebug	v3.0.4
PHPUnit	9.5.28

Core dump at /home/daniel/core-xdebug-T338208 on mwdebug1001.eqiad.wmnet

Details

	Subject	Repo	Branch	Lines +/-
	docker: update to latest published images	mediawiki/core	master	+3 -3
	Docker: Update dev images to 1.0.1-s2 images with fixed xdebug	mediawiki/core	master	+2 -2

Customize query in gerrit

Title	Reference	Author	Source Branch	Dest Branch
Fix buster-php74-fpm which was based on buster-php72	repos/releng/dev-images!40	hashar	php72-used-in-buster-php74-fpm	main
Install Xdebug from Debian package, update source of php72 and php74	repos/releng/dev-images!36	hashar	xdebug-and-php74	main
Refresh buster-php81-* images to upgrade php	repos/releng/dev-images!33	hashar	php81-upgrade	main

Customize query in GitLab

Related Objects

Mentioned Here: T216689: Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11)

Event Timeline

daniel created this task.Jun 6 2023, 8:12 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 6 2023, 8:12 AM

daniel updated the task description. (Show Details)Jun 6 2023, 8:21 AM

daniel added subscribers: CCicalese_WMF, tstarling, kostajh and 2 others.

daniel updated the task description. (Show Details)Jun 6 2023, 8:27 AM

daniel updated the task description. (Show Details)Jun 6 2023, 8:32 AM

tstarling added a project: php-segfault.Jun 6 2023, 10:39 AM

You could try setting

xdebug.log = <some file>
xdebug.log_level = 10

to dump the exact commands that PHPStorm sends to XDebug. Although presumably the one that causes the segfault won't make it into the log.

hashar updated the task description. (Show Details)Jun 6 2023, 1:46 PM

From the Slack thread:

Xdebug changelog: https://xdebug.org/updates

J152 is a post I wrote when I had to investigate HHVM core dumping in a CI Docker image. I went to install debug symbols inside the image and run gbd from inside it. So tentatively the core dump can be analyzed form within a container based on docker-registry.wikimedia.org/dev/buster-php81-fpm:1.0.0.

There are some steps I have skip in the blog post but can be found in the original task such as:

doing a busy loop to catch intermittent segfaults when running the full testsuite (T216689#5022384)
Installing gdb and the debug symbols and capturing the backtrace T216689#5024380

Retrieve the core dump and Docker image:

scp mwdebug1001.eqiad.wmnet:/home/daniel/core-xdebug-T338208 .
docker pull docker-registry.wikimedia.org/dev/buster-php81-fpm:1.0.0

Run a container based on that image with bind mount and running as root:

docker run -it -v .:/coredump --workdir /coredump --user=root --entrypoint=bash docker-registry.wikimedia.org/dev/buster-php81-fpm:1.0.0

Then inside the container:

apt update
apt install -y gdb php8.1-dbgsym

Note, it might be required to install some other php extension debug symbols.

The trouble is doing that will upgrade PHP cause the Docker image has outdated packages:

php8.1:
  Installed: 8.1.13-1+0~20221126.29+debian10~1.gbpfee7cc
  Candidate: 8.1.18-1+0~20230517.40+debian10~1.gbpad774c
  Version table:
     8.1.18-1+0~20230517.40+debian10~1.gbpad774c 500
        500 https://packages.sury.org/php buster/main amd64 Packages
 *** 8.1.13-1+0~20221126.29+debian10~1.gbpfee7cc 100
        100 /var/lib/dpkg/status

So installing php8.1-dbgsym gives the one from PHP 8.1.18 rather than 8.1.13 and I don't think sury keeps an archive of old packages. So the debug symbols are useless:

$ gdb /usr/bin/php core-xdebug-T338208
...
Core was generated by `php tests/phpunit/phpunit.php tests/phpunit/includes/TitleTest.php'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fbe1d825404 in ?? ()
(gdb) bt
#0  0x00007fbe1d825404 in ?? ()
#1  0x00007fbe1d82684a in ?? ()
#2  0x000000001d848080 in ?? ()
#3  0x000055d5b4286dc0 in ?? ()
#4  0x00007fbe1566fdc8 in ?? ()
#5  0x00007ffd2a4f9960 in ?? ()
#6  0x00007fbe155df8e8 in ?? ()
#7  0x00007fbe17f37a18 in ?? ()
#8  0x000055d5b42df060 in ?? ()
#9  0x000055d5b42e3070 in ?? ()
#10 0x000055d5b3ee9514 in ?? ()
#11 0x000055d5b42e2460 in ?? ()
#12 0x00007fbe1568a460 in ?? ()
#13 0x000055d5b42e2460 in ?? ()
#14 0x000055d5b4286dc0 in ?? ()
#15 0x00007fbe1d84535e in ?? ()
#16 0x0000000000000008 in ?? ()
#17 0x00007fbe1d84472e in ?? ()
#18 0x000055d5b42e88c0 in ?? ()
#19 0x00007fbe0000000d in ?? ()
#20 0x00007fbe17f37a18 in ?? ()
#21 0x000055d5b3ee9500 in ?? ()
#22 0x0000000000000000 in ?? ()
(gdb)

Note: the image does not have Xdebug installed so I am not sure from where the v3.0.4 version comes from.

Given the image has PHP 8.1.13 while sury has 8.1.18 I propose we refresh the image to grab the updates then see whether the issue still occurs.

It might be beneficial to have child images with debug symbols installed :]

hashar opened https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/33

Refresh buster-php81-* images to upgrade php

CodeReviewBot added a project: Patch-For-Review.Jun 6 2023, 3:24 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/33

Refresh buster-php81-* images to upgrade php

Maintenance_bot removed a project: Patch-For-Review.Jun 6 2023, 4:10 PM

Xdebug 3.1.0 is the first version that supports PHP 8.1, according to the changelog. It shouldn't be possible to run Xdebug 3.0.4 with PHP 8.1.13.

I'm having the same issue on macOS (Ventura 13.4) using MediaWiki Cli. This uses the same versions of PHP and Xdebug reported above.

dancy merged https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/33
Refresh buster-php81-* images to upgrade php

Note that although merged, the images have not been build cause we have an unrelated trouble with the build script :/

In T338208#8906292, @hashar wrote:

Note: the image does not have Xdebug installed so I am not sure from where the v3.0.4 version comes from.

I am similarly confused by that.

That being said, I experimented with updating my container to Xdebug v3.2.1 and can confirm that it fixes the segfault. Prior to that, I tried updating to PHP 8.1.18 (did not fix the segfault) and PHP 8.2 (which was actually a by-product of running "apt install php-xdebug", but it generated enough problems initially that I quickly abandoned that path). The commands that I used to get to a working configuration were:

apt update
apt install php-pear
pecl install xdebug

This results in the following version report:

PHP 8.1.13 (cli) (built: Nov 26 2022 14:29:42) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.13, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.13, Copyright (c), by Zend Technologies
    with Xdebug v3.2.1, Copyright (c) 2002-2023, by Derick Rethans

I ran phpunit immediately before and after that update and received a segfault before but not after.

So the image had the wrong version xdebug? That's the solution?

~~I'm trying to confirm Cindy's fix, but for some reason pecl install xdebug just sits and does nothing...~~ It succeeded eventually, after sitting with no output for half an hour.

I can confirm that manually updating xdebug inside the container works! Thank you Cindy!

Mentioned in SAL (#wikimedia-releng) [2023-06-08T08:11:30Z] <hashar> Manually building php8.1 dev images to update php 8.1 to the latest provided by sury.org. Change is https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/33 for T338208

I have manually build the images from https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/33

== Step 0: scanning /srv/dev-images/dockerfiles ==
Will build the following images:
* docker-registry.discovery.wmnet/dev/buster-php81:1.0.1
* docker-registry.discovery.wmnet/dev/buster-php81-jobrunner:1.0.1
* docker-registry.discovery.wmnet/dev/buster-swift53:0.0.1-s2
* docker-registry.discovery.wmnet/dev/buster-php81-fpm:1.0.1
== Step 1: building images ==
* Built image docker-registry.discovery.wmnet/dev/buster-php81:1.0.1
* Built image docker-registry.discovery.wmnet/dev/buster-php81-jobrunner:1.0.1
 ERROR: image docker-registry.discovery.wmnet/dev/buster-swift53 failed to build, see logs for details
* Built image docker-registry.discovery.wmnet/dev/buster-php81-fpm:1.0.1
== Step 2: publishing ==
Successfully published image docker-registry.discovery.wmnet/dev/buster-php81-jobrunner:1.0.1
Successfully published image docker-registry.discovery.wmnet/dev/buster-php81-fpm:1.0.1
Successfully published image docker-registry.discovery.wmnet/dev/buster-php81:1.0.1
== Build done! ==

May you try again with docker-registry.wikimedia.org/dev/buster-php81:1.0.1. It comes with php 8.1.18.

Looking at the build log it seems to compile Xdebug 3.0.4 from source rather than relying on the Debian package which is probably a bug in itself.

docker-registry.discovery.wmnet/dev/buster-swift53 failed cause of an outdated gpg key.

Daniel confirmed upgrading Xdebug fixed it. Thank you Tim and Cindy!

I will dig into the dev-images repository, we have a base buster images which downloads the xdebug 3.0.4 tarball and then all the flavored php images are inheriting it. We should use sury.org Debian package instead, I don't know why we went to compile from source.

hashar opened https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/36

Install Xdebug from Debian package, update source of php72 and php74

CodeReviewBot added a project: Patch-For-Review.Jun 8 2023, 10:18 AM

Any progress on fixing this in the image?

After I had successfully fixed this by manually updating xdebug in the docker container, it seems like it has now reverted itself. Presumably because docker-compose decided to pull a new image.

In T338208#8976091, @daniel wrote:

Any progress on fixing this in the image?

After I had successfully fixed this by manually updating xdebug in the docker container, it seems like it has now reverted itself. Presumably because docker-compose decided to pull a new image.

As hashar said on GitLab, this is blocked by git changes, and should be unblocked by https://gerrit.wikimedia.org/r/c/operations/puppet/+/927975

Yes that is the fix. Sorry it has been a long tail and I kind of loose track of this task while doing all the changes I had to create. Some patches got merged this week at least so it is progressing.

(also I haven't tried building the images proposed by https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/36 , much less tested them.

hashar merged https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/36

Install Xdebug from Debian package, update source of php72 and php74

Maintenance_bot removed a project: Patch-For-Review.Jul 5 2023, 12:10 PM

Mentioned in SAL (#wikimedia-releng) [2023-07-05T12:11:16Z] <hashar> Building dev-images for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/36 | "Install Xdebug from Debian package, update source of php72 and php74" | T338208

hashar opened https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/40

Fix buster-php74-fpm which was based on buster-php72

hashar merged https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/40

Fix buster-php74-fpm which was based on buster-php72

Maintenance_bot removed a project: Patch-For-Review.Jul 5 2023, 1:10 PM

I think I have successfully rebuild all images to include XDebug from a Debian package instead of compiling our own (outdated) version. I might have broken something while doing it though :-\

Checking the Xdebug version on the buster-php?? images:

$ for version in 72 73 74 80 81; do  echo -n "buster-php-$version: "; docker run --rm -it --entrypoint=php "docker-registry.discovery.wmnet/dev/buster-php$version" -r 'print phpversion("xdebug") . "\n";'; done;
buster-php-72: 3.1.6
buster-php-73: 3.1.6
buster-php-74: 3.0.3
buster-php-80: 3.2.1
buster-php-81: 3.2.1

The php 8.x version get Xdebug version 3.2.1 which I guess addresses the issue.

I am not sure whether the Xdebug configuration is correct though, I might have screwed that one. Please reopen if there is any issue!

How would I make use of the new images?

docker-compose pull doesn't seem to have pulled the new image, I'm still getting xdebug 3.0.4...

docker-compose images gives me:

           Container                                     Repository                            Tag        Image Id       Size  
-------------------------------------------------------------------------------------------------------------------------------
mediawiki_database_1              mariadb                                                    latest     011343cf3ec3   403.3 MB
mediawiki_mediawiki-jobrunner_1   docker-registry.wikimedia.org/dev/buster-php81-jobrunner   1.0.0      75e2399c9bdd   821.1 MB
mediawiki_mediawiki-web_1         docker-registry.wikimedia.org/dev/buster-apache2           2.0.0-s1   c6aec360894e   320.6 MB
mediawiki_mediawiki_1             docker-registry.wikimedia.org/dev/buster-php81-fpm         1.0.0      549543a10e3d   653 MB

Change 935780 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@master] Docker: Update dev images to 1.0.1-s2 images with fixed xdebug

https://gerrit.wikimedia.org/r/935780

gerritbot added a project: Patch-For-Review.Jul 5 2023, 5:01 PM

Note: when adding Xdebug with https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/36 I have also removed xhprof since the script to install Xdebug also cloned https://github.com/tideways/php-xhprof-extension.git and installed from source :/

So those images would need the xhprof Debian package to be installed (if available) or restore the source compilation.

Change 936327 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[mediawiki/core@master] docker: update to latest published images; tidy DEVELOPERS.md

https://gerrit.wikimedia.org/r/936327

Change 935780 abandoned by Hashar:

[mediawiki/core@master] Docker: Update dev images to 1.0.1-s2 images with fixed xdebug

Reason:

In favor of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/936327

https://gerrit.wikimedia.org/r/935780

Change 936327 merged by jenkins-bot:

[mediawiki/core@master] docker: update to latest published images

https://gerrit.wikimedia.org/r/936327

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.17; 2023-07-11).Jul 10 2023, 3:00 PM

Maintenance_bot removed a project: Patch-For-Review.Jul 10 2023, 3:10 PM