Evaluate running a statsd-exporter in the mw-script namespace
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	akosiaris
	Oct 8 2024, 1:08 PM

Description

We are currently running the following statsd-exporters

mw-api-ext                      statsd-exporter-prometheus                3/3       3            3           112d
mw-api-int                      statsd-exporter-prometheus                3/3       3            3           112d
mw-debug                        statsd-exporter-prometheus                2/2       2            2           119d
mw-jobrunner                    statsd-exporter-prometheus                3/3       3            3           112d
mw-misc                         statsd-exporter-prometheus                1/1       1            1           97d
mw-parsoid                      statsd-exporter-prometheus                3/3       3            3           116d
mw-web                          statsd-exporter-prometheus                3/3       3            3           112d
mw-wikifunctions                statsd-exporter-prometheus                1/1       1            1           97d

Conspicuously, what's missing from there is mw-script (and mw-videoscaler). I assume we don't want to differ here and we should be running one for mw-script as well. And if for whatever reason, we end up with a different namespace from the currently implemented as systemd timers recurring scripts, run in in that namespace too.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
mw-script: Die with a clearer error when $RELEASE_NAME is "prometheus"	operations/deployment-charts	master	+3 -0
deployment_server: Skip prometheus in mwscript-cleanup	operations/puppet	production	+3 -0
mw-script: Add prometheus-statsd-exporter	operations/deployment-charts	master	+25 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	dancy	T392693 Drop support for Debian Buster
Resolved	Clement_Goubert	T341560 Migrate mwmaint server functionality to mw-on-k8s
Resolved	RLazarus	T341553 Allow running one-off scripts manually
Resolved	RLazarus	T376714 Evaluate running a statsd-exporter in the mw-script namespace

Event Timeline

akosiaris created this task.Oct 8 2024, 1:08 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptOct 8 2024, 1:08 PM

• mszabo subscribed.Oct 8 2024, 1:15 PM

Change #1078666 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mw-script: Add prometheus-statsd-exporter

https://gerrit.wikimedia.org/r/1078666

gerritbot added a project: Patch-For-Review.Oct 8 2024, 1:20 PM

Lucas_Werkmeister_WMDE unsubscribed.Oct 8 2024, 1:31 PM

Thanks!

And if for whatever reason, we end up with a different namespace from the currently implemented as systemd timers recurring scripts

This is the plan; the namespace for CronJobs will be mw-cron, so it sounds like we should have an exporter there too when it's ready.

Change #1078666 merged by jenkins-bot:

[operations/deployment-charts@master] mw-script: Add prometheus-statsd-exporter

https://gerrit.wikimedia.org/r/1078666

statsd-exporter deployment merged and deployed in both eqiad and codfw. It is addressable via the standard mechanisms that the other deployment share. General docs are at https://wikitech.wikimedia.org/wiki/Kubernetes/Metrics and https://wikitech.wikimedia.org/wiki/Prometheus/statsd_k8s, more specific to this implementation T365265: Create a per-release deployment of statsd-exporter for mw-on-k8s (and we need to update docs for this one once we resolve that task). I 'll resolve this for now, in the interest of not letting it linger. Feel free to reopen!

Change #1080321 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] deployment_server: Skip prometheus in mwscript-cleanup

https://gerrit.wikimedia.org/r/1080321

Change #1080321 merged by RLazarus:

[operations/puppet@production] deployment_server: Skip prometheus in mwscript-cleanup

https://gerrit.wikimedia.org/r/1080321

Change #1080331 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mw-script: Die with a clearer error when $RELEASE_NAME is "prometheus"

https://gerrit.wikimedia.org/r/1080331

Change #1080331 merged by jenkins-bot: