Uniform performance insight for different services (tracking)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Sep 30 2016, 8:46 PM

Description

Inspired by GOTO 2016 • What I Wish I Had Known Before Scaling Uber to 1000 Services (Matt Ranney).

Call graphs / Flame graphs

Around 21m00s he mentions that while tooling differs between programming languages and frameworks, it was quite useful to have a uniform performance insight by converting the output of those (different) tools to flame graphs, which give a more familiar feeling.

We currently do this only for the MediaWiki run-time (with Xenon for HHVM).

I'd be interesting to get similar statistics going for other run-times and services.

Some services that we should consider adding a sampling profiler, and aggregating logs into daily flame graphs:

Varnish front-end and back-end. (Wikimedia VCL; operations/puppet)
RESTBase (Node.js)
Parsoid (Node.js)
EventLogging (Python)
Statsv (Python; analytics/statsv)
~~RCStream (Python; mediawiki/services/rcstream)~~
EventStreams (Node.js)
MediaWiki front-end JavaScript (maybe capture via headless Chrome as part of asset-check.py in operations/puppet)

It'd be great to be able easily dig into any of these services. Both for the teams that maintain these services, as well as for e.g. Performance-Team as part of routine inspection.

Distributed tracing

Aside from flame graphs for individual services (from sampling profilers and/or one-off full debug profiling), we should also look into ways to have a high-level overviews.

These high-level overviews could also include information about backend services (e.g. from MediaWiki to MySQL, from RESTBase to Cassandra, etc.).

Related Objects
Search...

Status	Assigned	Task
Resolved	Krinkle	T147101 Uniform performance insight for different services (tracking)
Resolved	Krinkle	T103461 Capturing and publish CPU Flame Graphs for Parsoid
Resolved	Krinkle	T185234 Document common/shared infra for service instrumentation
Declined	• Imarlier	T185235 Add profiling for webperf/navtiming

Event Timeline

Krinkle created this task.Sep 30 2016, 8:46 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 30 2016, 8:46 PM

phuedx awarded a token.Oct 1 2016, 8:31 AM

phuedx added a subscriber: phuedx.

ori triaged this task as Medium priority.Oct 3 2016, 6:48 PM

ori moved this task from Inbox to Backlog: Maintenance on the Performance-Team board.

Two main tasks stemming this:

Gathering aggregated stack traces from different services in different programming languages. These will likely require different solutions depending on the runtime. We're not aiming for a single solution here. Merely a universal output format as supported by Flame Graph to be published periodically on performance.wikimedia.org.

Distributed tracing across different services. Especially to accurately measure latency when services call each other, and to be able to see how and in which service that time is being spent. E.g. through waterfall graphs that show how one service calls another.

Depending on whether this second point includes in-process tracing, it will depend on point 1.

I'd like to prioritise point 1 for the time being. Although while we work on that, we can start thinking about the requirements for point 2. For example how we'll associate the different service requests (e.g. passing around a header? and evaluating which -if any- utility libraries we might need to deploy inside these services in different languages.)

A few relevant links as shared in the triage meeting earlier today:

Peter added a subscriber: Peter.Oct 4 2016, 8:03 AM

Possibly relevant to 2: just came across this yesterday http://opentracing.io

Krinkle claimed this task.Dec 7 2016, 6:29 PM

Krinkle removed Krinkle as the assignee of this task.Mar 8 2017, 1:43 AM

Krinkle updated the task description. (Show Details)Sep 12 2017, 3:49 PM

Krinkle mentioned this in T175710: Add profiling for Varnish and VCL.Sep 12 2017, 4:38 PM

Krinkle created subtask T175710: Add profiling for Varnish and VCL.

Krinkle added a subtask: T103461: Capturing and publish CPU Flame Graphs for Parsoid.Dec 22 2017, 6:16 PM

Krinkle moved this task from Backlog: Maintenance to Backlog: Future Goals on the Performance-Team board.Jan 18 2018, 10:38 AM

elukey added a subscriber: elukey.Jan 18 2018, 10:39 AM

Krinkle renamed this task from Uniform performance insight for different services to Uniform performance insight for different services (tracking).Jan 18 2018, 5:49 PM

Krinkle moved this task from Backlog: Future Goals to Backlog: Maintenance on the Performance-Team board.

Krinkle added a project: Epic.

Krinkle closed subtask T185235: Add profiling for webperf/navtiming as Declined.Feb 19 2018, 10:11 PM

Krinkle mentioned this in T193078: Investigate distributed tracing systems .Apr 30 2018, 8:05 PM

• Imarlier merged a task: T193078: Investigate distributed tracing systems .Apr 30 2018, 8:58 PM

• Imarlier added a subscriber: • Imarlier.

Krinkle updated the task description. (Show Details)May 1 2018, 2:14 AM

Krinkle added a subtask: T193050: Include request id (if present) in a comment in DB queries.

JJMC89 merged a task: T193078: Investigate distributed tracing systems .Jul 1 2018, 4:56 PM

Krinkle mentioned this in T201409: Harmonise the identification of requests across our stack.Aug 8 2018, 3:31 AM

Krinkle edited projects, added Performance-Team (Radar); removed Performance-Team.Jun 6 2019, 9:29 AM

Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.