Page MenuHomePhabricator
Create Task
Maniphest T364863

InterfaceSpeedError - mw2286
Closed, ResolvedPublic
Actions

Description

Common information

  • alertname: InterfaceSpeedError
  • cluster: api_appserver
  • instance: mw2286:9100
  • interface: eno1
  • job: node
  • operstate: up
  • prometheus: ops
  • severity: task
  • site: codfw
  • source: prometheus
  • team: dcops

Firing alerts

Event Timeline

phaultfinder created this task.Tue, May 14, 2:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, May 14, 2:01 PM
Maintenance_bot added a project: SRE.Tue, May 14, 2:29 PM
Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Tue, May 14, 2:45 PM
Jhancock.wm subscribed.Tue, May 14, 2:49 PM

the cable or the 1G SFP might need to be replaced. can we downtime the server for a small window to test the cabling?

Dzahn renamed this task from InterfaceSpeedError to InterfaceSpeedError - mw2286.Tue, May 14, 4:37 PM
Dzahn added a project: serviceops.
Stashbot added a comment.Tue, May 14, 4:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-14T16:39:17Z] <mutante> depooled mw2286.codfw.wmnet because of interface error / needed cable replacement T364863

RhinosF1 subscribed.Tue, May 14, 4:40 PM
Stashbot added a comment.Tue, May 14, 4:40 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-14T16:40:46Z] <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on mw2286.codfw.wmnet with reason: T364863

Stashbot added a comment.Tue, May 14, 4:41 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-14T16:41:00Z] <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mw2286.codfw.wmnet with reason: T364863

Dzahn added subscribers: RLazarus, Dzahn.Tue, May 14, 4:43 PM

@Jhancock.wm cc: @RLazarus I depooled the server and set a downtime of 24 hours.

Jhancock.wm added a comment.Wed, May 15, 2:27 PM

I reseated the sfp and the cable. Looks fixed and remained steady for an hour. Should be good to add back.

Jhancock.wm claimed this task.Wed, May 15, 2:27 PM
Stashbot added a comment.Wed, May 15, 2:38 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-15T14:38:26Z] <claime> Removing downtime on mw2286.codfw.wmnet - T364863

Stashbot added a comment.Wed, May 15, 2:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-15T14:39:37Z] <claime> Repooling mw2286.codfw.wmnet - T364863

Clement_Goubert closed this task as Resolved.Wed, May 15, 2:42 PM
Clement_Goubert subscribed.

Host downtime removed and repooled, thanks @Dzahn @Jhancock.wm

Dzahn awarded a token.Wed, May 15, 3:23 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL