Improve the InterfaceSpeedError alert
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ayounsi
	Dec 13 2023, 2:25 PM

Project Tags

Referenced Files

None

Subscribers

Description

From: T351862: InterfaceSpeedError, to make it more actionable by DCops:

The dashboard link points to an unrelated dashboard
description: on alert1001:9100 has the wrong speed: 1.2e+07. Should have a better notation, like Speed: 100Mb/s
The task title could be a bit more explicit (maybe mention the host)
The runbook link should point to https://wikitech.wikimedia.org/wiki/Monitoring/check_eth#InterfaceSpeedError (to be more specific) and have steps to follow when it triggers
Most of the information is repeated twice and it's unclear what's the actual bit of important information as opposed to all the metadata.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Improve InterfaceSpeedError messages	Hnowlan	operations/alerts	master	+8 -8

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		tappof	T395441 Port all Icinga checks to Prometheus/Alertmanager: preparation
Open		tappof	T321808 Port all Icinga checks to Prometheus/Alertmanager
Resolved	Goal	colewhite	T332764 Port base host checks from Icinga to Alertmanager
Resolved		cmooney	T333007 validate what we need from the check_eth check
Resolved		SLyngshede-WMF	T347312 Expose ethtool metrics to Prometheus
Resolved		hnowlan	T353323 Improve the InterfaceSpeedError alert

Event Timeline

ayounsi created this task.Dec 13 2023, 2:25 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 13 2023, 2:25 PM

ayounsi added a parent task: T347312: Expose ethtool metrics to Prometheus.Dec 13 2023, 2:25 PM

Volans updated the task description. (Show Details)Dec 13 2023, 2:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1053.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1053.eqiad.wmnet with OS bookworm completed:

cloudvirt1053 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202406210916_aborrero_181938_cloudvirt1053.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

sorry for the reimage messages, I copy-pasted the wrong phab ticket id.

• aborrero mentioned this in T368129: cloludvirt1035: InterfaceSpeedError: brq7425e328-56.Jun 21 2024, 9:47 AM

andrea.denisse triaged this task as Medium priority.Jan 10 2025, 9:40 PM

Change #1307432 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/alerts@master] Improve InterfaceSpeedError messages

https://gerrit.wikimedia.org/r/1307432

gerritbot added a project: Patch-For-Review.Jul 3 2026, 12:40 PM

Filed a review to address some of this - unfortunately the title is a bigger problem within phalerts to solve, but one that's worth reexamining (T351389)

Change #1307432 merged by jenkins-bot: