Page MenuHomePhabricator

Wall-clock Excimer profiling in production
Closed, ResolvedPublic

Description

The flame graphs on https://performance.wikimedia.org/php-profiling/ are based on Excimer CPU-time profiles. It'd be useful to have wall-clock profiling too.

For one-shot profiling, we run two Excimer instances (one wall-clock, the other CPU) in parallel. Can we do the same for production profiling? It looks pretty easy to do.

The wall-clock profiler would publish samples to a different channel, and the end result would be an additional set of log files and flame graphs, existing alongside the current CPU-based ones.

Event Timeline

Change 597654 had a related patch set uploaded (by Ori.livneh; owner: Ori.livneh):
[operations/mediawiki-config@master] wall-clock excimer profiling for production

https://gerrit.wikimedia.org/r/597654

Krinkle changed the task status from Open to Stalled.Jul 18 2020, 2:51 AM
Krinkle triaged this task as Medium priority.

@dpifke @aaron This might be worth considering now, also in context of memcached-on-host and its potential change in latency cost which the current CPU time might not (completely) capture.

Change 664591 had a related patch set uploaded (by Dave Pifke; owner: Dave Pifke):
[operations/puppet@production] arclamp: add excimer-real pipeline

https://gerrit.wikimedia.org/r/664591

Change 664591 merged by Dzahn:
[operations/puppet@production] arclamp: add excimer-real pipeline

https://gerrit.wikimedia.org/r/664591

Change 597654 merged by jenkins-bot:
[operations/mediawiki-config@master] profiler: wall-clock excimer instance

https://gerrit.wikimedia.org/r/597654

Mentioned in SAL (#wikimedia-operations) [2021-02-19T00:12:47Z] <dpifke@deploy1001> Synchronized wmf-config/profiler.php: Deploying excimer-wall profiler pipeline T253160 (duration: 01m 02s)

Mentioned in SAL (#wikimedia-operations) [2021-02-19T00:13:59Z] <dpifke@deploy1001> Synchronized wmf-config/PhpAutoPrepend.php: Deploying excimer-wall profiler pipeline T253160 (duration: 01m 03s)

Confimed data has started appearing on webperf1002.