Page MenuHomePhabricator

Upgrade our php-xdebug package for php7.2
Closed, ResolvedPublic

Description

From Stretch component/php72 we have php-xdebug 2.7.0-1+wmf1 and it has some performance issue doing way too many getpid calls.

Upstream issue: https://bugs.xdebug.org/bug_view_page.php?bug_id=00001641

I thus could use an upgrade to xdebug 2.7.1, or at least the inclusion of upstream commit https://github.com/xdebug/xdebug/commit/df17570fd52846baa7134f350547970644a8d1d5 Potentially even just upgrade to 2.7.2

target

The php-xdebug package is for the CI Docker containers

There are a couple production hosts using it releases* but the package should be removed from them: T236774

Xdebug Changelog

https://xdebug.org/updates

[2019-05-06] — Xdebug 2.7.2

Fixed Bugs
Fixed bug #1488: Rewrite DBGp 'property_set' to always use eval
Fixed bug #1586: error_reporting()'s return value is incorrect during debugger's 'eval' command
Fixed bug #1615: Turn off Zend OPcache when remote debugger is turned on
Fixed bug #1656: remote_connect_back alters header if multiple values are present
Fixed bug #1662: __debugInfo should not be used for user-defined classes

[2019-04-05] — Xdebug 2.7.1

Fixed Bugs
Fixed bug #1646: Missing newline in error message
Fixed bug #1647: Memory corruption when a conditional breakpoint is used
Fixed bug #1641: Perfomance degradation with getpid syscall (Kees Hoekzema)

Xdebug is used for generating PHPUnit code coverage report. I noticed the issue while doing a system wide perf showing:

Samples: 34K of event 'cpu-clock', Event count (approx.): 346343430880
Overhead  Command          Shared Object             Symbol
  85.31%  swapper          [kernel.kallsyms]         [k] native_safe_halt
   3.77%  php              [kernel.kallsyms]         [k] __bpf_prog_run
   2.04%  php              libc-2.24.so              [.] __getpid            # <<------ OOPS
   0.84%  php              xdebug.so                 [.] 0x00000000000165d5
   0.84%  php              xdebug.so                 [.] 0x000000000001665b

And found the upstream bug report by pure luck :]

Details

Related Gerrit Patches:

Event Timeline

hashar created this task.Oct 2 2019, 9:13 AM
hashar added a comment.Oct 2 2019, 9:21 AM

perf for the php process (which runs under a Docker stretch container so symbols are not available):

Samples: 192K of event 'cpu-clock', Event count (approx.): 48129000000
Overhead  Command  Shared Object        Symbol
  22.69%  php      xdebug.so            [.] 0x000000000001665b
  17.47%  php      xdebug.so            [.] 0x00000000000165d5
  10.63%  php      [kernel.kallsyms]    [k] __bpf_prog_run
   5.79%  php      xdebug.so            [.] 0x00000000000165c0
   5.42%  php      libc-2.24.so         [.] __getpid            # <<--------
   2.98%  php      xdebug.so            [.] 0x0000000000016661
   2.15%  php      [kernel.kallsyms]    [k] do_syscall_64
   1.33%  php      libc-2.24.so         [.] strlen
Krinkle moved this task from Untriaged to Other on the PHP 7.2 support board.Oct 10 2019, 4:34 AM
hashar triaged this task as High priority.Oct 22 2019, 11:40 AM
hashar added subscribers: jijiki, MoritzMuehlenhoff, Joe.

Summary: the Xdebug package for stretch-wikimedia component/php72 is version 2.7.0 which is crippled by a serious performance bug. That causes PHPUnit coverage reports to be way slower than they should.

Our package is rebuild of the Buster one:

xdebug (2.7.0-1+wmf1) stretch-wikimedia; urgency=medium

  * Rebuild for component/php7.2

 -- Moritz Muehlenhoff <moritz@wikimedia.org>  Fri, 26 Apr 2019 08:05:44 +0000

xdebug (2.7.0-1) unstable; urgency=medium

  * New upstream version 2.7.0

 -- Ondřej Surý <ondrej@debian.org>  Thu, 07 Mar 2019 19:45:49 +0000

Apparently build from https://salsa.debian.org/php-team/pecl/xdebug

I would need it to be rebuild with patch https://github.com/xdebug/xdebug/commit/df17570fd52846baa7134f350547970644a8d1d5 or directly upgraded to 2.7.2.

Thank you!

@hashar I will look into it, sorry for not getting to it sooner

Where does this error show up, in some CI instance or in prod? The only prod hosts with xdebug installed are releases* and those use PHP 7.0.

That is for the CI Docker containers running php7.2.

It seems the production releases* hosts have borrowed their php configuration from the CI hosts (d9388d6c8be9a1e879a3dfea232a87b7aada8508). Should be refactored + they are stuck to php7.0 :/

Where does this error show up, in some CI instance or in prod? The only prod hosts with xdebug installed are releases* and those use PHP 7.0.

I will get xdebug removed from the releases* hosts via T236774

jijiki claimed this task.Tue, Oct 29, 11:31 AM
hashar removed jijiki as the assignee of this task.Tue, Oct 29, 11:33 AM
hashar updated the task description. (Show Details)
hashar assigned this task to jijiki.Thu, Oct 31, 11:34 AM

Phabricator edit conflict. @jijiki is indeed working on it :]

Cherry picked and packaged. Please ping when you test it so I can upload it:)

So I have downloaded https://people.wikimedia.org/~jiji/php-xdebug_2.7.0-1+wmf2_amd64.deb

Ran mediawiki/core coverage using:

$ install --directory --mode 777 cache workspace
$ docker run --rm -it \
  -v "$HOME"/projects:/srv/git:ro \
  -v "$(pwd)/cache:/cache" \
  -v "$(pwd)/workspace:/workspace"  \
  docker-registry.wikimedia.org/releng/quibble-stretch-php72:latest --commands=mediawiki-coverage

Let it run then took a trace with sudo perf trace -p <pidof php process>:

Summary of events:

php (29386), 1832067 events, 100.0%

  syscall            calls    total       min       avg       max      stddev
                              (msec)    (msec)    (msec)    (msec)        (%)
  --------------- -------- --------- --------- --------- ---------     ------
  getpid            910806  1344.784     0.001     0.001     0.081      0.03%
  poll                 145    60.181     0.001     0.415     6.798     22.13%
  access              2790     9.794     0.002     0.004     0.052      1.18%
  write                215     2.088     0.003     0.010     0.058      4.19%
  open                 216     1.814     0.003     0.008     0.050      4.61%
  sendto               131     1.171     0.005     0.009     0.059      6.93%
  fstat                222     0.531     0.001     0.002     0.019      3.87%

Then I have build a new container:

Dockerfile
FROM docker-registry.wikimedia.org/releng/quibble-stretch-php72:latest

USER root

RUN curl -ophp-xdebug_2.7.0-1+wmf2_amd64.deb https://people.wikimedia.org/~jiji/php-xdebug_2.7.0-1+wmf2_amd64.deb
RUN dpkg -i php-xdebug_2.7.0-1+wmf2_amd64.deb

USER nobody

Ran it:

$ docker build -t newxdebug
...
$ docker run --rm -it \
  -v "$HOME"/projects:/srv/git:ro \
  -v "$(pwd)/cache:/cache" \
  -v "$(pwd)/workspace:/workspace"  \
  newxdebug --commands=mediawiki-coverage

Took trace with sudo perf trace -p <pid of php process>:

Summary of events:

php (31082), 48371 events, 100.0%

  syscall            calls    total       min       avg       max      stddev
                              (msec)    (msec)    (msec)    (msec)        (%)
  --------------- -------- --------- --------- --------- ---------     ------
  poll                 562   486.498     0.001     0.866     9.244      8.68%
  access             12946    40.707     0.002     0.003     0.057      0.67%
  open                1234     7.616     0.002     0.006     0.081      2.49%
  write                619     5.884     0.003     0.010     0.052      2.65%
  fstat               1909     3.781     0.001     0.002     0.036      2.26%
  sendto               381     3.731     0.004     0.010     0.042      2.78%
  read                2455     3.583     0.001     0.001     0.028      1.70%
  lseek               1817     2.440     0.001     0.001     0.004      1.21%
  recvfrom             562     2.416     0.001     0.004     0.018      2.78%
  close               1234     1.930     0.001     0.002     0.058      3.67%
  stat                 334     1.198     0.002     0.004     0.020      2.20%
  munmap                39     0.977     0.006     0.025     0.096     10.95%
  mmap                  44     0.589     0.006     0.013     0.062      9.56%
  getcwd                22     0.080     0.003     0.004     0.006      5.73%
  getrandom             12     0.062     0.003     0.005     0.007      8.17%
  lstat                  7     0.027     0.002     0.004     0.009     24.34%

getpid is gone entirely \o/

Mentioned in SAL (#wikimedia-operations) [2019-10-31T13:40:47Z] <effie> upload xdebug 2.7.0-1+wmf2 to component/php72 - T234418

jijiki closed this task as Resolved.Thu, Oct 31, 1:43 PM

cool :)

hashar reopened this task as Open.Thu, Oct 31, 2:26 PM

Gotta rebuild the CI containers and switch jobs to it ;]

jijiki reassigned this task from jijiki to hashar.Thu, Oct 31, 2:27 PM
jijiki removed a subscriber: jijiki.

Change 547542 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: rebuild for php-xdebug upgrade on php7.2

https://gerrit.wikimedia.org/r/547542

Change 547542 merged by jenkins-bot:
[integration/config@master] docker: rebuild for php-xdebug upgrade on php7.2

https://gerrit.wikimedia.org/r/547542

Mentioned in SAL (#wikimedia-releng) [2019-10-31T14:46:14Z] <hashar> Building CI containers for php7.2 php-xdebug upgrade #T234418

Change 547551 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Update coverage jobs for php-xdebug update

https://gerrit.wikimedia.org/r/547551

Change 547551 merged by jenkins-bot:
[integration/config@master] Update coverage jobs for php-xdebug update

https://gerrit.wikimedia.org/r/547551

hashar closed this task as Resolved.Thu, Oct 31, 3:10 PM
hashar added a subscriber: jijiki.

All good thank you @jijiki !

I have rebuild the container, updated a job, ran perf top -g -p on the php process. getpid() is gone \o/