Maniphest T206152

Set up request debug profiling for PHP 7
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Oct 3 2018, 6:12 PM

Description

For request profiling in debug requests we currently use HHVM's built-in XHProf extension in a number of ways:

Capture one-off profiles and send to XHGui at https://performance.wikimedia.org/.
Capture one-off profiles and send to stdout, via MediaWiki's Profiler and ProfilerOutput interfaces.

The logic for XHGui is in wmf-config/profiler.php. (Basically, just xhprof_enable/disable.)

The logic for MediaWiki is in core, and currently supports php-xhprof (PHP 5.x), php-tideways (PHP 5.x and PHP 7.x) and HHVM's XHProf.

Links:

https://pecl.php.net/package/xhprof / https://github.com/phacility/xhprof
Originally facebook/xhprof, now phacility/xhprof. Not much activity, but maybe it's just stable. (issue)
Supports PHP 5.x only, but there's an experimental branch for PHP 7 support.
HHVM also includes a port of XHProf, which is what we currently use in production.

Before we can switch from HHVM to PHP 7, we need to decide which extension to use for PHP debug profiling.

Options:

phacility/xhprof:experimental (see https://github.com/phacility/xhprof/tree/experimental).
tideways/tideways_xhprof (see https://github.com/tideways/php-xhprof-extension).
... something else?

Details

	Subject	Repo	Branch	Lines +/-
	profile::mediawiki::php: install php-tideways, php-mongodb	operations/puppet	production	+9 -0

Customize query in gerrit

Revisions and Commits

rOMWC Wikimedia - MediaWiki Config
	rOMWC3545a906d6b1 Excimer and Tideways support

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T176370 Migrate to PHP 7 in WMF production
		Resolved		Krinkle	T206152 Set up request debug profiling for PHP 7

Event Timeline

Krinkle triaged this task as Medium priority.Oct 3 2018, 6:12 PM

Krinkle created this task.

Krinkle mentioned this in T176916: Set up sampling profiler for PHP 7 (alternative to HHVM Xenon).Oct 3 2018, 6:18 PM

• Imarlier moved this task from Inbox, needs triage to To-do: Goals prioritized current Quarter on the Performance-Team board.Oct 11 2018, 8:27 PM

php-tideways is packaged in debian stretch, and although we will need to rebuild it (to add php 7.2 support), it should be pretty straightforward. AIUI, it should allow us to continue using the same tooling as before.

Krinkle claimed this task.Dec 5 2018, 4:43 PM

Please install tideways, but it should only be enabled in php.ini on the debug servers, since it will cause a performance degradation even without being used. Also, please install php-mongodb, the PHP driver for MongoDB, since this is recommended for XHGui saving on PHP 7. I am working on the mediawiki-config patch which will use these extensions.

In T206152#4804916, @tstarling wrote:

Please install tideways, but it should only be enabled in php.ini on the debug servers, since it will cause a performance degradation even without being used. Also, please install php-mongodb, the PHP driver for MongoDB, since this is recommended for XHGui saving on PHP 7. I am working on the mediawiki-config patch which will use these extensions.

For now I'll just install/enable them on the debug servers, which will then merit their own separate role.

Mentioned in SAL (#wikimedia-operations) [2018-12-07T17:15:50Z] <_joe_> uploading php-tideways (rebuilt with php 7.2 support) to stretch-wikimedia thirdparty/php72 T206152

Change 478594 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::php: install php-tideways, php-mongodb

https://gerrit.wikimedia.org/r/478594

gerritbot added a project: Patch-For-Review.Dec 10 2018, 7:22 AM

I did some benchmarks , using the same setup I used for T206341, with tideways enabled and disabled. I could not notice any clear trend besides a slight variance due probably to external factors.

See the graphs collected here https://people.wikimedia.org/~oblivian/T206341/tideways/images/ for reference. I am in favour of enabling it everywhere, unless there is some non-performance-related concern.

Mentioned in SAL (#wikimedia-operations) [2018-12-10T14:50:29Z] <_joe_> uploading php-mongodb 1.5.3 to stretch-wikimedia thirdparty/php72 T206152

Change 478594 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::php: install php-tideways, php-mongodb

https://gerrit.wikimedia.org/r/478594

Krinkle moved this task from To-do: Goals prioritized current Quarter to Doing (old) on the Performance-Team board.Jan 14 2019, 8:51 PM

Assuming the above means tideways is now installed and available with PHP7 on mwdebug servers, next step is to make it work with our X-Wikimedia-Debug "Profiling" option so that the profiles are captured and send to XHGui.

Upon initially trying this to day, nothing happened (no errors, either). Probably because of an HHVM-conditional somewhere, will look again tomorrow.

Speaking of Mongo: T213996: New MongoDB version is not DFSG-compatible, dropped by Debian.

Krinkle moved this task from Doing (old) to Blocked (old) on the Performance-Team board.Jan 23 2019, 12:28 AM

With 3545a906d6b1 deployed (by @tstarling). I'll try this again tomorrow.

Tried both via forceprofile=1 (GET parameter) and via the profile (XWD attribute), and neither seems to work yet on PHP 7.

They do still work on HHVM.

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:28 PM

Krinkle moved this task from Doing (old) to Blocked (old) on the Performance-Team board.Feb 4 2019, 10:31 PM

Krinkle moved this task from Blocked (old) to Doing (old) on the Performance-Team board.Feb 19 2019, 9:43 PM

In T176916#4964977, @Joe wrote:
I did manually install php7.2-tideways-xhprof on mwdebug1001 and I now see the following error:
 Fatal error: Uncaught Error: Call to undefined function register_postsend_function() in /srv/mediawiki/wmf-config/profiler.php:359 
Stack trace: #0 /srv/mediawiki/wmf-config/profiler.php(46): wmfSetupTideways(Array) 
#1 /srv/mediawiki/wmf-config/PhpAutoPrepend.php(45): wmfSetupProfiler(Array) #2 {main} thrown in /srv/mediawiki/wmf-config/profiler.php on line 359
which isn't surprising given register_postsend_function is HHVM-specific AIUI.

In T176916#4965053, @gerritbot wrote:

Change 491518 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] profile: use register_shutdown_function

https://gerrit.wikimedia.org/r/491518

In T176916#4965274, @gerritbot wrote:

Change 491533 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::php: install tideways-xhprof, remove tideways

https://gerrit.wikimedia.org/r/491533

Krinkle renamed this task from Set up request profiling for PHP 7 to Set up request debug profiling for PHP 7.Feb 19 2019, 9:51 PM

This now works as expected via the WikimediaDebug extension. Enabling the "Profile" and "PHP 7" options creates an XHGui record. For example, https://performance.wikimedia.org/xhgui/run/view?id=5c70688f3f3dfa6d433bbdc6.

There are minor differences between how HHVM/XHProf work and how php72-tideways work. For example, HHVM inserted fake stack frames to represent file scope.

This resulted in extra entries between main() and actual statements, such as run_init:/srv/mediawiki/php--/load.php and run_init:/srv/mediawiki/wmf-config/CommonSettings.php. These no longer exist in Tideways, which means we lose some amount of detail and useful grouping of statements.

On the other hand, it encourages us to wrap files in functions and call those, which is probably a good idea anyway.

Krinkle mentioned this in T176370: Migrate to PHP 7 in WMF production.Apr 10 2019, 4:01 PM

Krinkle mentioned this in Blog Post: WikimediaDebug v2 is here!.Dec 14 2019, 2:34 AM

Set up request debug profiling for PHP 7Closed, ResolvedPublicActions