Page MenuHomePhabricator

Label value spam in ncredir_requests_total metric
Closed, ResolvedPublic

Description

I'm looking at Thanos/Prometheus resource usage and noticed there's significant spam for ncredir_requests_total in the vhost label, looks like this is user-controlled (?)

ncredir_requests_total{cluster="misc",instance="ncredir1001:3904",job="ncredir",method="GET",prog="ncredir.mtail",scheme="http",site="eqiad",status="301",vhost="73da8.sesel.top"}
ncredir_requests_total{cluster="misc",instance="ncredir1001:3904",job="ncredir",method="GET",prog="ncredir.mtail",scheme="http",site="eqiad",status="301",vhost="7a3jz.t6g.top"}	13
ncredir_requests_total{cluster="misc",instance="ncredir1001:3904",job="ncredir",method="GET",prog="ncredir.mtail",scheme="http",site="eqiad",status="301",vhost="7b99c312.telot.top"}

Can we either ditch the label altogether (is it used anywhere?) or sanitize it somehow?

Event Timeline

Change 978619 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] ncredir: stop exporting 'vhost' label via mtail

https://gerrit.wikimedia.org/r/978619

Change 978619 merged by Filippo Giunchedi:

[operations/puppet@production] ncredir: stop exporting 'vhost' label via mtail

https://gerrit.wikimedia.org/r/978619

fgiunchedi claimed this task.

This is done, job="ncredir" metrics are now two orders of magnitude less

2023-11-29-173649_1669x1557_scrot.png (1×1 px, 108 KB)