Page MenuHomePhabricator

Set up request debug profiling for PHP 7
Closed, ResolvedPublic

Description

For request profiling in debug requests we currently use HHVM's built-in XHProf extension in a number of ways:

  • Capture one-off profiles and send to XHGui at https://performance.wikimedia.org/.
  • Capture one-off profiles and send to stdout, via MediaWiki's Profiler and ProfilerOutput interfaces.

See also https://wikitech.wikimedia.org/wiki/X-Wikimedia-Debug for details.

The logic for XHGui is in wmf-config/profiler.php. (Basically, just xhprof_enable/disable.)

The logic for MediaWiki is in core, and currently supports php-xhprof (PHP 5.x), php-tideways (PHP 5.x and PHP 7.x) and HHVM's XHProf.

Links:

Before we can switch from HHVM to PHP 7, we need to decide which extension to use for PHP debug profiling.

Options:

Event Timeline

Krinkle created this task.Oct 3 2018, 6:12 PM
Krinkle triaged this task as Normal priority.
Joe added a comment.Dec 3 2018, 9:13 PM

php-tideways is packaged in debian stretch, and although we will need to rebuild it (to add php 7.2 support), it should be pretty straightforward. AIUI, it should allow us to continue using the same tooling as before.

Krinkle claimed this task.Dec 5 2018, 4:43 PM

Please install tideways, but it should only be enabled in php.ini on the debug servers, since it will cause a performance degradation even without being used. Also, please install php-mongodb, the PHP driver for MongoDB, since this is recommended for XHGui saving on PHP 7. I am working on the mediawiki-config patch which will use these extensions.

Joe added a comment.EditedDec 7 2018, 7:15 AM

Please install tideways, but it should only be enabled in php.ini on the debug servers, since it will cause a performance degradation even without being used. Also, please install php-mongodb, the PHP driver for MongoDB, since this is recommended for XHGui saving on PHP 7. I am working on the mediawiki-config patch which will use these extensions.

For now I'll just install/enable them on the debug servers, which will then merit their own separate role.

Mentioned in SAL (#wikimedia-operations) [2018-12-07T17:15:50Z] <_joe_> uploading php-tideways (rebuilt with php 7.2 support) to stretch-wikimedia thirdparty/php72 T206152

Change 478594 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::php: install php-tideways, php-mongodb

https://gerrit.wikimedia.org/r/478594

Joe added a comment.Dec 10 2018, 1:38 PM

I did some benchmarks , using the same setup I used for T206341, with tideways enabled and disabled. I could not notice any clear trend besides a slight variance due probably to external factors.

See the graphs collected here https://people.wikimedia.org/~oblivian/T206341/tideways/images/ for reference. I am in favour of enabling it everywhere, unless there is some non-performance-related concern.

Mentioned in SAL (#wikimedia-operations) [2018-12-10T14:50:29Z] <_joe_> uploading php-mongodb 1.5.3 to stretch-wikimedia thirdparty/php72 T206152

Change 478594 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::php: install php-tideways, php-mongodb

https://gerrit.wikimedia.org/r/478594

Assuming the above means tideways is now installed and available with PHP7 on mwdebug servers, next step is to make it work with our X-Wikimedia-Debug "Profiling" option so that the profiles are captured and send to XHGui.

Upon initially trying this to day, nothing happened (no errors, either). Probably because of an HHVM-conditional somewhere, will look again tomorrow.

Krinkle moved this task from Doing to Blocked on the Performance-Team board.Jan 23 2019, 12:28 AM
Krinkle moved this task from Blocked to Doing on the Performance-Team board.

With 3545a906d6b1 deployed (by @tstarling). I'll try this again tomorrow.

Krinkle added a comment.EditedJan 25 2019, 10:26 PM

Tried both via forceprofile=1 (GET parameter) and via the profile (XWD attribute), and neither seems to work yet on PHP 7.

They do still work on HHVM.

Krinkle moved this task from Doing to Blocked on the Performance-Team board.Feb 4 2019, 10:31 PM
Krinkle moved this task from Blocked to Doing on the Performance-Team board.Feb 19 2019, 9:43 PM

I did manually install php7.2-tideways-xhprof on mwdebug1001 and I now see the following error:

 Fatal error: Uncaught Error: Call to undefined function register_postsend_function() in /srv/mediawiki/wmf-config/profiler.php:359 
Stack trace: #0 /srv/mediawiki/wmf-config/profiler.php(46): wmfSetupTideways(Array) 
#1 /srv/mediawiki/wmf-config/PhpAutoPrepend.php(45): wmfSetupProfiler(Array) #2 {main} thrown in /srv/mediawiki/wmf-config/profiler.php on line 359

which isn't surprising given register_postsend_function is HHVM-specific AIUI.

Change 491518 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] profile: use register_shutdown_function

https://gerrit.wikimedia.org/r/491518

Change 491533 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::php: install tideways-xhprof, remove tideways

https://gerrit.wikimedia.org/r/491533

Krinkle renamed this task from Set up request profiling for PHP 7 to Set up request debug profiling for PHP 7.Feb 19 2019, 9:51 PM
Krinkle closed this task as Resolved.EditedFeb 22 2019, 9:32 PM

This now works as expected via the WikimediaDebug extension. Enabling the "Profile" and "PHP 7" options creates an XHGui record. For example, https://performance.wikimedia.org/xhgui/run/view?id=5c70688f3f3dfa6d433bbdc6.

There are minor differences between how HHVM/XHProf work and how php72-tideways work. For example, HHVM inserted fake stack frames to represent file scope.

This resulted in extra entries between main() and actual statements, such as run_init:/srv/mediawiki/php--/load.php and run_init:/srv/mediawiki/wmf-config/CommonSettings.php. These no longer exist in Tideways, which means we lose some amount of detail and useful grouping of statements.

On the other hand, it encourages us to wrap files in functions and call those, which is probably a good idea anyway.