Page MenuHomePhabricator
Create Task
Maniphest T379719

wikikube-ctrl2002: Switch network cable from port 2 to port 1 on the 10G NIC
Closed, ResolvedPublic
Actions

Description

In T379629: wikikube-ctrl1001.eqiad.wmnet fails PXE boot we figured out that PXE boot does not work reliably when we use port 2 on the BCM57412.

We should switch from port 2 to port 1 because of this and to adhere to our general default.

After switching the port a cold reboot might be required so that the card does detect link. Also the BIOS settings need changing:

  • Ensure PXE is disabled on all embedded NIC ports
  • Ensure PXE is enabled on all 10G NIC ports
  • Ensure Port 1 is first in the boot order list (after Harddrive C:)

As the server is in service, we need to coordinate a short maintenance window.

Related Objects
Search...

Event Timeline

JMeybohm created this task.Wed, Nov 13, 10:30 AM
Restricted Application added a project: DC-Ops. · View Herald TranscriptWed, Nov 13, 10:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Maintenance_bot added a project: SRE.Wed, Nov 13, 11:29 AM
Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Wed, Nov 13, 3:18 PM
Jhancock.wm subscribed.Thu, Nov 14, 3:22 PM

I could do this today. or we can wait until next week. assuming no one wants to do a maintenance on a friday.

ops-monitoring-bot added a comment.Thu, Nov 14, 3:28 PM

depool host wikikube-ctrl2002.codfw.wmnet by jayme@cumin2002 with reason: None

ops-monitoring-bot added a comment.Thu, Nov 14, 3:28 PM

Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin2002 depool for host wikikube-ctrl2002.codfw.wmnet completed:

  • wikikube-ctrl2002.codfw.wmnet (PASS)
    • Host wikikube-ctrl2002.codfw.wmnet depooled from wikikube-codfw
Stashbot added a comment.Thu, Nov 14, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-14T15:29:13Z] <jayme@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2002.codfw.wmnet with reason: T379719

Stashbot added a comment.Thu, Nov 14, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-14T15:29:29Z] <jayme@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2002.codfw.wmnet with reason: T379719

ops-monitoring-bot added a comment.Thu, Nov 14, 3:45 PM

pool host wikikube-ctrl2002.codfw.wmnet by jayme@cumin2002 with reason: None

ops-monitoring-bot added a comment.Thu, Nov 14, 3:45 PM

Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin2002 pool for host wikikube-ctrl2002.codfw.wmnet completed:

  • wikikube-ctrl2002.codfw.wmnet (PASS)
    • Host wikikube-ctrl2002.codfw.wmnet pooled in wikikube-codfw
JMeybohm closed this task as Resolved.EditedThu, Nov 14, 3:47 PM
JMeybohm claimed this task.

@Jhancock.wm swapped the cable into port 1, I've changed BIOS settings and the server is back up. Thanks!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits