Page MenuHomePhabricator
Create Task
Maniphest T330754

hw troubleshooting: Link hard down (probably cable) for cloudcephosd2002-dev.codfw.wmnet
Closed, ResolvedPublicRequest
Actions

Description

  • - Provide FQDN of system.

cloudcephosd2002-dev.codfw.wmnet

  • - If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.

You can take it down anytime (it's still powered on, but feel free to turn it off).

  • - Put system into a failed state in Netbox.
  • - Provide urgency of request, along with justification (redundancy, dependencies, etc)

It's not a very urgent request, as only affects our testing setup. This blocks though in a medium span any changes we want to make to production (ex. 316544)

The link shows hard down on both sides (host and switch):

root@cloudcephosd2002-dev:~# ip a show dev eno2
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc mq state DOWN group default qlen 1000
    link/ether 2c:ea:7f:3f:ed:d4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.4.2/24 scope global eno2
       valid_lft forever preferred_lft forever
  • - Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

Related Objects
Search...

Event Timeline

dcaro created this task.Feb 28 2023, 2:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 28 2023, 2:04 PM
dcaro added a parent task: T329507: [ceph] Test crush tree with rack level HA on codfw.Feb 28 2023, 2:08 PM
Papaul added a subscriber: Jhancock.wm.Feb 28 2023, 6:23 PM

@Jhancock.wm can you please comment what troubleshooting you did today for this link?

Thanks

Jhancock.wm added a comment.Feb 28 2023, 6:58 PM

I found the port down with no link indicator lights. I reseated both ends but it did not help. After that I tested with a new patch and it also did not show activity. Patch was normalized to original cable.

Papaul added a comment.Feb 28 2023, 11:47 PM

@Jhancock.wm what you can try to do if you are on site before me is plug a patch cable to ge-1/0/1 and the other end to sretest2001 and see if you have link if not we will try to use another port on the switch.

Thanks

dcaro added a parent task: T329778: [ceph] Investigate if there's a way to degrade instead of failing when jumbo frames are being dropped in the network.Mar 1 2023, 12:44 PM
Papaul closed this task as Resolved.Mar 2 2023, 3:34 PM

@dcaro it looks like we are having a bad interface on asw-b1-codfw(ge-1/0/1) so i switch the server connection to ge-1/0/12. You should be all good now. Let me know if you have any questions.

Interface       Admin Link Description
ge-1/0/12       up    up   cloudcephosd2002-dev

papaul@asw-b-codfw# run show interfaces ge-1/0/1 descriptions
Interface       Admin Link Description
ge-1/0/1        down  down DISABLED
dcaro added a comment.Mar 2 2023, 3:49 PM

Thanks a lot! The cluster is already working on restoring itself :)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL