Page MenuHomePhabricator

Capturing and publish CPU Flame Graphs for Parsoid
Closed, ResolvedPublic

Description

It would be interesting to generalise our performance metrics to outside just the MediaWiki application stack.

I don't know to what degree the Performance Team has resources to work on this directly, but we may find some times to help the Parsing Team get it set up for Parsoid. From there we would generalise it to then apply to other Node.js services.

For the MediaWiki runtime:

  • The Xenon extension in HHVM captures stack traces at regular intervals for code on-CPU.
  • wikimedia/arc-lamp: PHP script subscribes to the data and sends it to a Redis queue (currently on fluorine; Production actually has a copy of it in wmf-config/StartProfiler)
  • xenon-log deamon on flourine subscribes to the queue and produces log files.
  • An hourly/daily script produces interactive SVG Flame Graphs at https://performance.wikimedia.org/xenon/

It should be feasible to produce stack traces in a compatible (possibly with a converter in between) and set up another pipeline like this.

Related links:

Event Timeline

Krinkle created this task.Jun 23 2015, 3:45 AM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 23 2015, 3:45 AM
Krinkle set Security to None.Jun 23 2015, 3:45 AM
Krinkle added subscribers: ori, GWicke.
Krinkle updated the task description. (Show Details)Jun 23 2015, 3:49 AM

In my experience the v8 profiler delivers generally more accurate and actionable node profiling information than perf. If you look closely at the images linked above, you will notice that most of the time seems to be spent in v8 internals, with very limited accounting for what the associated JS code is, despite source maps.

Simple flame graphs also tend to be less useful with fairly asynchronous applications like Parsoid, simply because most of the time there is not much on the stack. Graph-based profile viewers like kcachegrind offer more flexibility when studying complex profiles. See https://github.com/gwicke/nodegrind for a screenshot of such a graph view.

Arlolra triaged this task as Low priority.Jul 7 2015, 2:32 AM
Arlolra added a subscriber: Arlolra.
ssastry moved this task from Needs Triage to Performance on the Parsoid board.Dec 18 2017, 10:04 PM
Imarlier moved this task from Inbox to Radar on the Performance-Team board.Jan 8 2018, 9:16 PM
Imarlier edited projects, added Performance-Team (Radar); removed Performance-Team.

@Pchelolo, @mobrovac thoughts on this? Are any other node.js services doing this?

@ssastry no, none of the other node services do that but it indeed might be interesting. I'll have a look if there's some existing solutions for node.js

We need to retarget this phab task for Parsoid/PHP.

Krinkle added a comment.EditedJan 7 2020, 4:49 PM

Given that Parsoid/PHP runs on the MediaWiki production stack, profiling comes for free and is mostly automated.

The auto-prepend configuration and wmf-config loads and sets up sampling profiling. The Arc Lamp then takes it from there and organises it by entrypoint, where rest.php got picked up automatically as well.

Daily flame graphs are published at https://performance.wikimedia.org/arclamp/svgs/daily/.

I've gone ahead and also added a shortcut link on the profiling homepage for easy access:

rPFDRf40cb6e290c6: php-profiling: Make flame graph imaged linked, add "rest" endpoint
https://performance.wikimedia.org/php-profiling/

Krinkle closed this task as Resolved.Jan 7 2020, 4:54 PM
Krinkle claimed this task.