toolforge prometheus servers OOMing
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	Nov 1 2023, 11:03 AM

Description

The Toolforge Prometheus server has been crashing for the last day or so.

	Title	Reference	Author	Source Branch	Dest Branch
	Revert "wmcs-k8s-metrics: rollback tools"	repos/cloud/toolforge/toolforge-deploy!183	taavi	taavi/metrics	main
	wmcs-k8s-metrics: rollback tools	repos/cloud/toolforge/toolforge-deploy!125	taavi	taavi/revert-metrics	main

Status	Assigned	Task
Open	None	T362869 [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30)
Open	None	T362868 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29
Open	None	T362867 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28
Open	None	T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
Open	None	T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26
Open	None	T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25
Resolved	aborrero	T307651 Upgrade Toolforge Kubernetes to version 1.24
Open	None	T360699 Toolsbeta: migrate to Debian Bullseye or later
Open	None	T311897 Toolforge: migrate to Debian Bullseye or later
Open	None	T311908 Migrate Toolforge Kubernetes hosts to Debian Bullseye or later
Resolved	• Bstorm	T262550 Toolforge returns HTTP 502 error
Open	taavi	T262562 [infra] Fix the mis-named k8s service in tools and toolsbeta projects
Resolved	taavi	T355883 Create a pool of NFS-less Toolforge Kubernetes workers
Resolved	taavi	T284656 Toolforge k8s: Migrate workers to Containerd and Bookworm
Resolved	taavi	T349795 Upgrade cadvisor
Resolved	taavi	T350227 toolforge prometheus servers OOMing

taavi triaged this task as High priority.Nov 1 2023, 11:03 AM

taavi created this task.

The instances are using g3.cores8.ram36.disk20, so I'm a bit surprised they're running out of RAM.

wmcs-k8s-metrics: rollback tools

wmcs-k8s-metrics: rollback tools

Mentioned in SAL (#wikimedia-cloud) [2023-11-02T13:13:31Z] <taavi> wiping data directory from tools-prometheus-7 so we have least one working server T350227

taavi closed this task as Resolved.Nov 3 2023, 2:48 PM

Revert "wmcs-k8s-metrics: rollback tools"

Revert "wmcs-k8s-metrics: rollback tools"