longer writeup to-do, but, tldr: it's perhaps undesirable to (at least initially) force lots of things to depend on the OpenTelemetry libraries just for context propagation, when we could instead have a list of headers to copy from incoming requests (if present) to outgoing sub-requests
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T340551 distributed tracing epic | |||
| Open | None | T320549 distributed tracing v0 [minimum viable] | |||
| Open | None | T320556 Micro-specification for how service owners should propagate tracing headers | |||
| Resolved | pmiazga | T320559 Trace header propagation for MediaWiki | |||
| Resolved | pmiazga | T344926 Propagate x-request-id header from MultiHttpClient in MediaWiki (e.g. SessionStore) | |||
| Resolved | pmiazga | T345295 MultiHttpClient should handle headers from __construct | |||
| Resolved | Jdforrester-WMF | T365053 WikiLambda OrchestratorRequest does not propagate request tracing headers | |||
| Resolved | EBernhardson | T371129 Extension:CirrusSearch not propagating tracing headers | |||
| Open | None | T320561 Trace header propagation for service-template-node and all service-runner services | |||
| Open | None | T344148 Trace header propagation for Proton service | |||
| Resolved | pmiazga | T346209 Send tracing headers from Shellbox client |
Event Timeline
Comment Actions
Would also like some guidance for how to pass on this info in emitted data events. We already produce the request_id, but perhaps we can make a more standardized tracing schema fragment for this.
Comment Actions
I agree with this, it should be a reusable fragment for events, probably a versioned one (it's possible we add more fields in the future).
In Mediawiki core we now have Wikimedia/Http/TelemetryHeadersInterface, which although doesn't provide a data model, allows for pluggable implementations that extract the necessary headers, and currently only has one implementation: https://gerrit.wikimedia.org/g/mediawiki/core/+/master/includes/http/Telemetry.php
It seems like a reasonable idea to add to that interface a way to extract the same telemetry context as a fragment for events.
Comment Actions
I have a very quick first draft at https://wikitech.wikimedia.org/wiki/Distributed_tracing/Propagating_tracing_context
Comment Actions
There is a quick way already - Telemetry::getRequestHeaders().
You can find it used across MediaWiki, for example:
- https://github.com/wikimedia/mediawiki/blob/5c6b75659f59ca6baa00218e2ac5c1276fc03468/includes/http/MWHttpRequest.php#L201
- https://github.com/wikimedia/mediawiki/blob/master/includes/libs/http/MultiHttpClient.php#L685
The idea is - wherever you send a request, call Telemetry::getInstance()->getRequestHeaders() and inject those into the request you're sending.
Currently, this will only fully propagate or generate the X-Request-Id and only propagate 'tracestate', 'traceparent' . To create new spans we would need a library. @mszabo is working on a POC of lightweight lib that would allow us to create new spans.
Comment Actions
Yep -- my link above is RFC, not code -- although it does reference Telemetry::getInstance()->getRequestHeaders() :)