Page MenuHomePhabricator

mwdebug dashboard on logstash is full of "Failed to connect to exporter" messages (tracing channel) since 7 January
Closed, ResolvedPublic

Description

The mwdebug dashboard currently has lots of these messages – distribution over the last 30 days (23565 messages in total during that time):

image.png (1,438×245 px, 21 KB)

This makes the dashboard less useful to detect deployment problems:
image.png (1,920×640 px, 115 KB)

Details

Related Changes in Gerrit:

Event Timeline

(No idea what the right tags for this would be, Wikimedia-Logstash and Observability-Tracing are just guesses really :/)

colewhite added subscribers: fgiunchedi, CDanis, colewhite.

Removing logstash tag as this appears to be coming from the opentelemetry library built into MW.

Change #1115436 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/mediawiki-config@master] Trace only on k8s. Not (yet?) available on bare metal

https://gerrit.wikimedia.org/r/1115436

Change #1115436 merged by jenkins-bot:

[operations/mediawiki-config@master] Trace only on k8s.

https://gerrit.wikimedia.org/r/1115436

Mentioned in SAL (#wikimedia-operations) [2025-01-30T16:09:45Z] <cdanis@deploy2002> cdanis: Backport for [[gerrit:1115436|Trace only on k8s. (T321211 T340552 T385037)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-01-30T16:17:57Z] <cdanis@deploy2002> Finished scap sync-world: Backport for [[gerrit:1115436|Trace only on k8s. (T321211 T340552 T385037)]] (duration: 11m 55s)

Sorry for the logspam, fixed now.