Page MenuHomePhabricator

Reduce cardinality of `path` component of Test Kitchen UI (MPIC) Prometheus metrics
Closed, ResolvedPublic3 Estimated Story PointsBUG REPORT

Description

The Prometheus logging approach presently used in Test Kitchen UI logs the path component logs for all URLs, logged-in states, and success/error dispositions. This leads to some noise and can in principle lead to slower query execution for this service in Prometheus or the central aggregation point Thanos.

An approach to reduce this noise could do something like the following:

  1. When logging to Prometheus for URLs that don't actually exist, log the path with some synthetic path (e.g., /invalid_path_for_request). [1]
  2. When logging to Prometheus for static asset URLs that do exist, log the path with some synthetic path (e.g., /static_asset). In practice, this relates to JavaScript and CSS 200 responses.

[1] In practice, this relates to things that are being answered with both 400 series 200 series responses. Also, the UA for a user who has been signed out may make a request that ultimately triggers 404s for legitimate paths (e.g., the /user path), and so there's a little wrinkle to work out here - it's useful to know that the user was accessing in a non-logged in context when the path is a legitimate one (imagine the case of a CAS-SSO misconfiguration affecting this application or even the case of some route accidentally being put into the requires-authentication routing case when really it's supposed to be an anonymous route), but it's also noise.

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Add route-specific metricsrepos/data-engineering/test-kitchen!328phuedxwork/phuedx/T407341main
Customize query in GitLab

Event Timeline

Thanks @Aklapper for the tag. I closed my laptop and realized, "oh no, I've forgotten the tag!". But you were on it already!

dr0ptp4kt renamed this task from Reduce cardinality of `path` component of xLab UI (MPIC)) Prometheus metrics to Reduce cardinality of `path` component of xLab UI (MPIC) Prometheus metrics.Oct 15 2025, 11:31 AM
dr0ptp4kt updated the task description. (Show Details)
JVanderhoop-WMF moved this task from Incoming to READY TO GROOM on the Test Kitchen board.
JVanderhoop-WMF set the point value for this task to 3.
JVanderhoop-WMF subscribed.

Allow list - which paths we legitimately care about.

Sfaci renamed this task from Reduce cardinality of `path` component of xLab UI (MPIC) Prometheus metrics to Reduce cardinality of `path` component of Test Kitchen UI (MPIC) Prometheus metrics.Mar 16 2026, 9:33 AM
Sfaci updated the task description. (Show Details)
JVanderhoop-WMF lowered the priority of this task from High to Medium.May 14 2026, 3:12 PM

Change #1294984 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.3.7 release to staging

https://gerrit.wikimedia.org/r/1294984

Change #1294985 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.3.7 release to production

https://gerrit.wikimedia.org/r/1294985

Change #1294984 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.3.7 release to staging

https://gerrit.wikimedia.org/r/1294984

Change #1294985 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.3.7 release to production

https://gerrit.wikimedia.org/r/1294985