Page MenuHomePhabricator
Create Task
Maniphest T247881

Three ports on asw2-d-eqiad are not working as expected
Closed, ResolvedPublic
Actions

Description

During the debugging of T247561 the hosts kafka-jumbo1006 and stat1005 were moved to different switch ports that didn't really work.

Timeline:

  • stat1005 on ge-1/0/4 and kafka-jumbo1006 on ge-1/0/5 show up in icinga at the same time as DOWN
  • kafka-jumbo1006 is moved to ge-1/0/9 and regain connectivity
  • stat1005 is moved to ge-1/0/6 but still shows no connectivity
  • stat1005 is moved to ge-1/0/43 and regain connectivity

So at least 3 ports on asw2-d-eqiad are not working as expected: ge-1/0/4, ge-1/0/5 and ge-1/0/6

Arzhel suggested to test those ports with a laptop or similar to see if they are really not working at all or not.

Details

SubjectRepoBranchLines +/-
move dumps around on the snapshots in prep for network upgrade workoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedayounsiT247881 Three ports on asw2-d-eqiad are not working as expected
 Resolved• CmjohnsonT256112 eqiad row D switch fabric recabling
Unknown Object (Task)

Event Timeline

elukey created this task.Mar 17 2020, 5:47 PM
Restricted Application added a project: SRE. · View Herald TranscriptMar 17 2020, 5:47 PM
elukey updated the task description. (Show Details)Mar 17 2020, 5:49 PM
wiki_willy assigned this task to • Cmjohnson.Mar 26 2020, 5:36 PM
• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Mar 30 2020, 8:13 PM
• Cmjohnson added a comment.May 11 2020, 6:14 PM

I just attempted us use ge-1/0/6 and it did not work

ayounsi subscribed.May 15 2020, 11:10 AM

If they're dead:

  • Either we need them (eg. short on ports), and in that case we need to replace the switch. Which is a heavy operations.
  • Or we mark the ports as dead (with a mention of that task), disable them and call it a day.
faidon subscribed.May 15 2020, 12:16 PM

If three ports are permanently failed, I'm not sure how we could ever trust that switch again. Perhaps it's better to do a painful but planned replacement rather than have it fail at some inconvenient time and having to rush a replacement then?

ayounsi changed the task status from Open to Stalled.May 19 2020, 7:29 AM
ayounsi removed • Cmjohnson as the assignee of this task.
ayounsi triaged this task as Low priority.
ayounsi added a subscriber: • Cmjohnson.

Sounds good! This will have to wait for a time we for example do T196487. Outside of COVID times as it's impactful and not urgent.

• Cmjohnson moved this task from Hardware Failure / Troubleshoot to Blocked on the ops-eqiad board.May 21 2020, 9:46 PM
ayounsi mentioned this in T256112: eqiad row D switch fabric recabling.Jun 23 2020, 10:38 AM
ayounsi added a subtask: T256112: eqiad row D switch fabric recabling.
Marostegui subscribed.Aug 26 2020, 2:10 PM
gerritbot added a comment.Aug 30 2020, 4:09 PM

Change 623177 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] move dumps around on the snapshots in prep network upgrade work

https://gerrit.wikimedia.org/r/623177

gerritbot added a project: Patch-For-Review.Aug 30 2020, 4:09 PM
gerritbot added a comment.Aug 31 2020, 5:14 AM

Change 623177 merged by ArielGlenn:
[operations/puppet@production] move dumps around on the snapshots in prep for network upgrade work

https://gerrit.wikimedia.org/r/623177

Maintenance_bot removed a project: Patch-For-Review.Aug 31 2020, 6:10 AM
ayounsi closed subtask T256112: eqiad row D switch fabric recabling as Resolved.Sep 24 2020, 3:01 PM
Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 9:00 PM
• Cmjohnson added a comment.Aug 25 2021, 6:37 PM

@ayounsi I did some investigating on this today and there have been servers plugged into 2 of 3 (ge-1/0/5 and 1/0/6) ports now for quite some time and no issues. Maybe we had issues that were not related to the switch. Do you want to keep this open or close and re-visit if necessary.

ayounsi closed this task as Resolved.Aug 30 2021, 12:48 PM
ayounsi claimed this task.

Noted, thanks! Yeah fine to close for now, and re-open if any issues.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL