InterfaceSpeedError - mw2286
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	May 14 2024, 2:01 PM

Description

Common information

dashboard: https://grafana.wikimedia.org/d/000000562
description: on mw2286:9100 has the wrong speed: 1.2e+07.
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_eth
summary: on mw2286:9100 has the wrong speed: 1.2e+07.

alertname: InterfaceSpeedError
cluster: api_appserver
instance: mw2286:9100
interface: eno1
job: node
operstate: up
prometheus: ops
severity: task
site: codfw
source: prometheus
team: dcops

Firing alerts

dashboard: https://grafana.wikimedia.org/d/000000562
description: on mw2286:9100 has the wrong speed: 1.2e+07.
runbook: https://wikitech.wikimedia.org/wiki/Monitoring/check_eth
summary: on mw2286:9100 has the wrong speed: 1.2e+07.
alertname: InterfaceSpeedError
cluster: api_appserver
instance: mw2286:9100
interface: eno1
job: node
operstate: up
prometheus: ops
severity: task
site: codfw
source: prometheus
team: dcops
Source

Event Timeline

phaultfinder created this task.May 14 2024, 2:01 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2024, 2:01 PM

Maintenance_bot added a project: SRE.May 14 2024, 2:29 PM

Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.May 14 2024, 2:45 PM

the cable or the 1G SFP might need to be replaced. can we downtime the server for a small window to test the cabling?

Dzahn renamed this task from InterfaceSpeedError to InterfaceSpeedError - mw2286.May 14 2024, 4:37 PM

Dzahn added a project: serviceops.

Mentioned in SAL (#wikimedia-operations) [2024-05-14T16:39:17Z] <mutante> depooled mw2286.codfw.wmnet because of interface error / needed cable replacement T364863

RhinosF1 subscribed.May 14 2024, 4:40 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-14T16:40:46Z] <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on mw2286.codfw.wmnet with reason: T364863

Mentioned in SAL (#wikimedia-operations) [2024-05-14T16:41:00Z] <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mw2286.codfw.wmnet with reason: T364863

@Jhancock.wm cc: @RLazarus I depooled the server and set a downtime of 24 hours.

I reseated the sfp and the cable. Looks fixed and remained steady for an hour. Should be good to add back.

Jhancock.wm claimed this task.May 15 2024, 2:27 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-15T14:38:26Z] <claime> Removing downtime on mw2286.codfw.wmnet - T364863

Mentioned in SAL (#wikimedia-operations) [2024-05-15T14:39:37Z] <claime> Repooling mw2286.codfw.wmnet - T364863

Host downtime removed and repooled, thanks @Dzahn @Jhancock.wm

Dzahn awarded a token.May 15 2024, 3:23 PM

InterfaceSpeedError - mw2286Closed, ResolvedPublicActions

Description

Common information

Firing alerts

Event Timeline

InterfaceSpeedError - mw2286
Closed, ResolvedPublic
Actions