Page MenuHomePhabricator
Create Task
Maniphest T234418

Upgrade our php-xdebug package for php7.2
Closed, ResolvedPublic
Actions

Description

From Stretch component/php72 we have php-xdebug 2.7.0-1+wmf1 and it has some performance issue doing way too many getpid calls.

Upstream issue: https://bugs.xdebug.org/bug_view_page.php?bug_id=00001641

I thus could use an upgrade to xdebug 2.7.1, or at least the inclusion of upstream commit https://github.com/xdebug/xdebug/commit/df17570fd52846baa7134f350547970644a8d1d5 Potentially even just upgrade to 2.7.2

target

The php-xdebug package is for the CI Docker containers

There are a couple production hosts using it releases* but the package should be removed from them: T236774

Xdebug Changelog

https://xdebug.org/updates

[2019-05-06] — Xdebug 2.7.2

Fixed Bugs
Fixed bug #1488: Rewrite DBGp 'property_set' to always use eval
Fixed bug #1586: error_reporting()'s return value is incorrect during debugger's 'eval' command
Fixed bug #1615: Turn off Zend OPcache when remote debugger is turned on
Fixed bug #1656: remote_connect_back alters header if multiple values are present
Fixed bug #1662: __debugInfo should not be used for user-defined classes

[2019-04-05] — Xdebug 2.7.1

Fixed Bugs
Fixed bug #1646: Missing newline in error message
Fixed bug #1647: Memory corruption when a conditional breakpoint is used
Fixed bug #1641: Perfomance degradation with getpid syscall (Kees Hoekzema)

Xdebug is used for generating PHPUnit code coverage report. I noticed the issue while doing a system wide perf showing:

Samples: 34K of event 'cpu-clock', Event count (approx.): 346343430880
Overhead  Command          Shared Object             Symbol
  85.31%  swapper          [kernel.kallsyms]         [k] native_safe_halt
   3.77%  php              [kernel.kallsyms]         [k] __bpf_prog_run
   2.04%  php              libc-2.24.so              [.] __getpid            # <<------ OOPS
   0.84%  php              xdebug.so                 [.] 0x00000000000165d5
   0.84%  php              xdebug.so                 [.] 0x000000000001665b

And found the upstream bug report by pure luck :]

Details

SubjectRepoBranchLines +/-
Update coverage jobs for php-xdebug updateintegration/configmaster+5 -5
docker: rebuild for php-xdebug upgrade on php7.2integration/configmaster+88 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Oct 2 2019, 9:13 AM
hashar added a comment.Oct 2 2019, 9:21 AM

perf for the php process (which runs under a Docker stretch container so symbols are not available):

Samples: 192K of event 'cpu-clock', Event count (approx.): 48129000000
Overhead  Command  Shared Object        Symbol
  22.69%  php      xdebug.so            [.] 0x000000000001665b
  17.47%  php      xdebug.so            [.] 0x00000000000165d5
  10.63%  php      [kernel.kallsyms]    [k] __bpf_prog_run
   5.79%  php      xdebug.so            [.] 0x00000000000165c0
   5.42%  php      libc-2.24.so         [.] __getpid            # <<--------
   2.98%  php      xdebug.so            [.] 0x0000000000016661
   2.15%  php      [kernel.kallsyms]    [k] do_syscall_64
   1.33%  php      libc-2.24.so         [.] strlen
Mainframe98 subscribed.Oct 2 2019, 9:46 AM
Krinkle moved this task from Untriaged to Other on the PHP 7.2 support board.Oct 10 2019, 4:34 AM
Jdforrester-WMF removed a project: PHP 7.2 support.Oct 10 2019, 10:53 PM
hashar triaged this task as High priority.Oct 22 2019, 11:40 AM
hashar added subscribers: jijiki, MoritzMuehlenhoff, Joe.

Summary: the Xdebug package for stretch-wikimedia component/php72 is version 2.7.0 which is crippled by a serious performance bug. That causes PHPUnit coverage reports to be way slower than they should.

Our package is rebuild of the Buster one:

xdebug (2.7.0-1+wmf1) stretch-wikimedia; urgency=medium

  * Rebuild for component/php7.2

 -- Moritz Muehlenhoff <moritz@wikimedia.org>  Fri, 26 Apr 2019 08:05:44 +0000

xdebug (2.7.0-1) unstable; urgency=medium

  * New upstream version 2.7.0

 -- Ondřej Surý <ondrej@debian.org>  Thu, 07 Mar 2019 19:45:49 +0000

Apparently build from https://salsa.debian.org/php-team/pecl/xdebug

I would need it to be rebuild with patch https://github.com/xdebug/xdebug/commit/df17570fd52846baa7134f350547970644a8d1d5 or directly upgraded to 2.7.2.

Thank you!

jijiki added a comment.Oct 22 2019, 11:50 AM

@hashar I will look into it, sorry for not getting to it sooner

MoritzMuehlenhoff added a comment.Oct 22 2019, 1:00 PM

Where does this error show up, in some CI instance or in prod? The only prod hosts with xdebug installed are releases* and those use PHP 7.0.

hashar added a comment.Oct 28 2019, 11:04 AM

That is for the CI Docker containers running php7.2.

It seems the production releases* hosts have borrowed their php configuration from the CI hosts (d9388d6c8be9a1e879a3dfea232a87b7aada8508). Should be refactored + they are stuck to php7.0 :/

hashar added a comment.Oct 29 2019, 11:19 AM
In T234418#5594961, @MoritzMuehlenhoff wrote:

Where does this error show up, in some CI instance or in prod? The only prod hosts with xdebug installed are releases* and those use PHP 7.0.

I will get xdebug removed from the releases* hosts via T236774

jijiki claimed this task.Oct 29 2019, 11:31 AM
hashar removed jijiki as the assignee of this task.Oct 29 2019, 11:33 AM
hashar updated the task description. (Show Details)
hashar assigned this task to jijiki.Oct 31 2019, 11:34 AM

Phabricator edit conflict. @jijiki is indeed working on it :]

jijiki added a comment.Oct 31 2019, 11:36 AM

Cherry picked and packaged. Please ping when you test it so I can upload it:)

hashar added a comment.Oct 31 2019, 1:40 PM

So I have downloaded https://people.wikimedia.org/~jiji/php-xdebug_2.7.0-1+wmf2_amd64.deb

Ran mediawiki/core coverage using:

$ install --directory --mode 777 cache workspace
$ docker run --rm -it \
  -v "$HOME"/projects:/srv/git:ro \
  -v "$(pwd)/cache:/cache" \
  -v "$(pwd)/workspace:/workspace"  \
  docker-registry.wikimedia.org/releng/quibble-stretch-php72:latest --commands=mediawiki-coverage

Let it run then took a trace with sudo perf trace -p <pidof php process>:

Summary of events:

php (29386), 1832067 events, 100.0%

  syscall            calls    total       min       avg       max      stddev
                              (msec)    (msec)    (msec)    (msec)        (%)
  --------------- -------- --------- --------- --------- ---------     ------
  getpid            910806  1344.784     0.001     0.001     0.081      0.03%
  poll                 145    60.181     0.001     0.415     6.798     22.13%
  access              2790     9.794     0.002     0.004     0.052      1.18%
  write                215     2.088     0.003     0.010     0.058      4.19%
  open                 216     1.814     0.003     0.008     0.050      4.61%
  sendto               131     1.171     0.005     0.009     0.059      6.93%
  fstat                222     0.531     0.001     0.002     0.019      3.87%

Then I have build a new container:

Dockerfile
FROM docker-registry.wikimedia.org/releng/quibble-stretch-php72:latest

USER root

RUN curl -ophp-xdebug_2.7.0-1+wmf2_amd64.deb https://people.wikimedia.org/~jiji/php-xdebug_2.7.0-1+wmf2_amd64.deb
RUN dpkg -i php-xdebug_2.7.0-1+wmf2_amd64.deb

USER nobody

Ran it:

$ docker build -t newxdebug
...
$ docker run --rm -it \
  -v "$HOME"/projects:/srv/git:ro \
  -v "$(pwd)/cache:/cache" \
  -v "$(pwd)/workspace:/workspace"  \
  newxdebug --commands=mediawiki-coverage

Took trace with sudo perf trace -p <pid of php process>:

Summary of events:

php (31082), 48371 events, 100.0%

  syscall            calls    total       min       avg       max      stddev
                              (msec)    (msec)    (msec)    (msec)        (%)
  --------------- -------- --------- --------- --------- ---------     ------
  poll                 562   486.498     0.001     0.866     9.244      8.68%
  access             12946    40.707     0.002     0.003     0.057      0.67%
  open                1234     7.616     0.002     0.006     0.081      2.49%
  write                619     5.884     0.003     0.010     0.052      2.65%
  fstat               1909     3.781     0.001     0.002     0.036      2.26%
  sendto               381     3.731     0.004     0.010     0.042      2.78%
  read                2455     3.583     0.001     0.001     0.028      1.70%
  lseek               1817     2.440     0.001     0.001     0.004      1.21%
  recvfrom             562     2.416     0.001     0.004     0.018      2.78%
  close               1234     1.930     0.001     0.002     0.058      3.67%
  stat                 334     1.198     0.002     0.004     0.020      2.20%
  munmap                39     0.977     0.006     0.025     0.096     10.95%
  mmap                  44     0.589     0.006     0.013     0.062      9.56%
  getcwd                22     0.080     0.003     0.004     0.006      5.73%
  getrandom             12     0.062     0.003     0.005     0.007      8.17%
  lstat                  7     0.027     0.002     0.004     0.009     24.34%

getpid is gone entirely \o/

Stashbot added a comment.Oct 31 2019, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-31T13:40:47Z] <effie> upload xdebug 2.7.0-1+wmf2 to component/php72 - T234418

jijiki closed this task as Resolved.Oct 31 2019, 1:43 PM

cool :)

hashar reopened this task as Open.Oct 31 2019, 2:26 PM

Gotta rebuild the CI containers and switch jobs to it ;]

jijiki reassigned this task from jijiki to hashar.Oct 31 2019, 2:27 PM
jijiki unsubscribed.
gerritbot added a comment.Oct 31 2019, 2:36 PM

Change 547542 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: rebuild for php-xdebug upgrade on php7.2

https://gerrit.wikimedia.org/r/547542

gerritbot added a project: Patch-For-Review.Oct 31 2019, 2:36 PM
gerritbot added a comment.Oct 31 2019, 2:42 PM

Change 547542 merged by jenkins-bot:
[integration/config@master] docker: rebuild for php-xdebug upgrade on php7.2

https://gerrit.wikimedia.org/r/547542

Stashbot added a comment.Oct 31 2019, 2:46 PM

Mentioned in SAL (#wikimedia-releng) [2019-10-31T14:46:14Z] <hashar> Building CI containers for php7.2 php-xdebug upgrade #T234418

gerritbot added a comment.Oct 31 2019, 3:05 PM

Change 547551 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Update coverage jobs for php-xdebug update

https://gerrit.wikimedia.org/r/547551

gerritbot added a comment.Oct 31 2019, 3:09 PM

Change 547551 merged by jenkins-bot:
[integration/config@master] Update coverage jobs for php-xdebug update

https://gerrit.wikimedia.org/r/547551

hashar closed this task as Resolved.Oct 31 2019, 3:10 PM
hashar added a subscriber: jijiki.

All good thank you @jijiki !

hashar added a comment.Oct 31 2019, 3:21 PM

I have rebuild the container, updated a job, ran perf top -g -p on the php process. getpid() is gone \o/

hashar mentioned this in T247587: XDebug makes requiring files unreasonably slow and affects open file limit.Apr 1 2020, 5:51 PM
hashar added a subtask: T220406: Package XDebug 2.9 for apt.wikimedia.org.
hashar mentioned this in T220406: Package XDebug 2.9 for apt.wikimedia.org.Apr 1 2020, 6:04 PM
hashar mentioned this in T243847: Add pcov PHP extension to wikimedia apt (and upgrade from 1.0.6-4+wmf1~buster1 to 1.0.11) so it can be used in Wikimedia CI.
hashar added a comment.May 6 2020, 12:08 PM

I have filed it to Debian upstream since Buster is affected: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959873

Tgr closed subtask T220406: Package XDebug 2.9 for apt.wikimedia.org as Invalid.Oct 27 2022, 4:00 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 27 2022, 4:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits