[wmcs-cookbooks] Downtime alerts from cloudcumins
Open, MediumPublic
Actions

Assigned To

Authored By

	fnegri
	Sep 27 2023, 1:31 PM

Description

Currently, the functions in wmcs_libs/alerts.py ssh to alert1001.wikimedia.org to downtime alerts. This works only if you run the cookbook from your laptop and you have global root privileges.

We have several cookbooks that we want to run from cloudcumins (e.g. wmcs.ceph.roll_reboot_osds or wmcs.openstack.roll_reboot_cloudgws) that require downtiming some alerts on wmcs-managed physical hosts. At the moment from cloudcumins you can ssh into those hosts (with the cloud_cumin_master key) but you cannot silence alerts related to those hosts.

Some thoughts:

we can probably ignore Icinga alerts, as we want to move away from them anyway
we need a way to silence Prometheus alerts only for wmcs-managed hosts
- is there a way to give limited access to the Prometheus API/CLI or do we need a separate Prometheus instance?

Details

Subject	Repo	Branch	Lines +/-
openstack: cloudvirt: safe_reboot: Downtime during reboot	cloud/wmcs-cookbooks	main	+23 -0
alertmanager: fix timezone bug	operations/software/spicerack	master	+15 -3
cloud_management: add cloudcumins to am api rw	operations/puppet	production	+2 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	fnegri	T337848 WMCS-roots wiki replica access
		Restricted Task
		Restricted Task
		Restricted Task
Resolved	fnegri	T343330 WMCS cookbooks: provide shared hosts for people without global root privileges
Open	None	T344412 cloudcumin: support reimage and other operations
Stalled	None	T347979 Remove wmcs-admin access from production cumin hosts
Open	taavi	T347490 [wmcs-cookbooks] Downtime alerts from cloudcumins

Event Timeline

fnegri created this task.Sep 27 2023, 1:31 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 27 2023, 1:31 PM

fnegri added parent tasks: T343330: WMCS cookbooks: provide shared hosts for people without global root privileges, T344412: cloudcumin: support reimage and other operations.Sep 27 2023, 1:32 PM

fnegri moved this task from Backlog to wmcs-cookbooks on the Cloud-VPS board.

fnegri triaged this task as Medium priority.Sep 27 2023, 1:51 PM

fnegri added a parent task: Restricted Task.

is there a way to give limited access to the Prometheus API/CLI or do we need a separate Prometheus instance?

AFAIK Alertmanager doesn't have any authz/n and we're managing access via firewall right now. But I hope to be corrected and that they added new features that could be leveraged for this.

fnegri added a subscriber: taavi.Sep 27 2023, 3:29 PM

fnegri mentioned this in T325067: cloudcumin: decide sudoers rules for users without global root.Oct 2 2023, 12:57 PM

fnegri mentioned this in T347979: Remove wmcs-admin access from production cumin hosts.Oct 3 2023, 11:32 AM

fnegri added a parent task: T347979: Remove wmcs-admin access from production cumin hosts.

lmata edited projects, added SRE Observability (FY2023/2024-Q2); removed SRE Observability.Oct 11 2023, 2:17 PM

The easiest at the moment is to add cloudcumin hosts to profile::alertmanager::api::rw

Change 965468 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] cloud_management: add am profile for silences

https://gerrit.wikimedia.org/r/965468

gerritbot added a project: Patch-For-Review.Oct 12 2023, 8:44 AM

Change 965468 merged by David Caro:

[operations/puppet@production] cloud_management: add cloudcumins to am api rw

https://gerrit.wikimedia.org/r/965468

Maintenance_bot removed a project: Patch-For-Review.Oct 12 2023, 10:30 AM

fnegri mentioned this in rCCKB88e4ba9757a3: set_maintenance: do not downtime host.Nov 14 2023, 5:34 PM

fgiunchedi removed a project: SRE Observability (FY2023/2024-Q2).Jan 15 2024, 11:13 AM

fgiunchedi added a project: Observability-Alerting.

One problem is that spicerack uses this to construct the silence start/end dates:

spicerack/alertmanager.py

start = datetime.utcnow().astimezone(tz=timezone.utc)

However, this seems to do double timezone-conversion, datetime.utcnow() gets the current UTC time but as timezone-unaware, and astimezone then assumes that's local time and applies a duplicate offset:

# It is currently 13:42 EET, so 11:42 UTC
>>> datetime.utcnow().astimezone()
datetime.datetime(2024, 1, 16, 11, 42, 36, 960520, tzinfo=datetime.timezone(datetime.timedelta(seconds=7200), 'EET'))
>>> datetime.utcnow().astimezone(tz=timezone.utc)
datetime.datetime(2024, 1, 16, 9, 43, 27, 417246, tzinfo=datetime.timezone.utc)

That needs to be fixed so we can use the same code on cloudcumin (which just can't SSH to the AM box) and local laptops (which don't tend to be in UTC).

Change 991016 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] openstack: cloudvirt: safe_reboot: Downtime during reboot

https://gerrit.wikimedia.org/r/991016

gerritbot added a project: Patch-For-Review.Jan 16 2024, 1:57 PM

Change 991017 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/software/spicerack@master] alertmanager: fix timezone bug

https://gerrit.wikimedia.org/r/991017

Change 991017 merged by jenkins-bot:

[operations/software/spicerack@master] alertmanager: fix timezone bug

https://gerrit.wikimedia.org/r/991017

fnegri edited projects, added cloud-services-team (FY2023/2024-Q3-Q4); removed cloud-services-team (FY2023/2024-Q1-Q2).Feb 1 2024, 11:14 AM

fnegri added a project: Goal.Feb 1 2024, 11:22 AM