| Blake | |
| Jun 26 2026, 10:17 AM |
| F90557315: image.png | |
| Jun 26 2026, 10:17 AM |
We expect this alert to fire while nodes are being reimaged, but currently it's creating tasks (https://phabricator.wikimedia.org/T430218) when they aren't required. This should be silenced as part of the reimaging cookbook.
(Note to self: it would be good to understand first why the period where the sessions were flapping was so long, before deciding on a silence period.)
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| bgp: Remove port from instance and remote_instance names. | Blake | operations/alerts | master | +16 -10 |
The downtime is set in the cookbook using https://doc.wikimedia.org/spicerack/v3.2.1/api/spicerack.alertmanager.html. That is a bit opinionated to what labels metrics need to carry in order to be included in the silence. You could verify in code or by looking at a currently set downtime (during reimaging) how the matchers look exactly.
Change #1307787 had a related patch set uploaded (by Blake; author: Blake):
[operations/alerts@master] bgp: Remove :0 from instance and remote_instance names.
Icinga downtime and Alertmanager silence (ID=6fd1f992-9594-4d65-8840-33a5f806e6a3) set by blake@cumin1003 for 0:30:00 on 1 host(s) and their services with reason: Verifying matchers for silence
wikikube-worker1160.eqiad.wmnet
It looks like downtimes are correctly created, and the matcher in question will catch instances which happen to have port numbers at the end. I think nothing needs to change here, and we should just be sure to use sre.k8s.renumber-node to reimage nodes which require a vlan move.
I'll submit the CR with just the adjustments to the summary and description templating, to clean it up a bit.
Cross-posting from T431443: In the case of a VLAN move that changes the intended BGP peer, we should try to ensure that the silence extends all the way through the homer commit that deconfigures the old sessions, or else we'll still get spurious alerts.
Change #1307787 merged by jenkins-bot:
[operations/alerts@master] bgp: Remove port from instance and remote_instance names.
I don't think there's anything left to do here - there's now a flag to pass the renumber-node cookbook (--run_homer_inline) (e.g. sudo cookbook sre.k8s.renumber-node -t T421711 wikikube-worker1067.eqiad.wmnet --os=trixie --run_homer_inline) to help ensure that we don't miss the step.
@Blake we still see this alert incoming: https://phabricator.wikimedia.org/T432990
Is there extra silencing needed here?
There's a tradeoff here, I think. The silence is currently time-limited, and we do not have any guarantees that the cookbook run will proceed within any given timeframe (it waits for operator input). If the second homer run hasn't been completed by the time the silence expires, we're going to get an alert. I don't think we want to silence for an arbitrarily long time, but we could consider increasing the duration if this is a common occurrence.