emailed our rep about the firmware load erorrs:

had been slightly unseated over time even though velcroed in place as it was fully online when i checked today, but pushed in cables.

@wiki_willy : FYI as this was an outstanding task item from May DC Ops meetings. All supermicro models (including GPU model) are now in netbox. Additionally, All dell models have been consolidated via mass edits. Example: If there was 'R440 - Config A 202101' and 'R440 - Config A 202203' they have all been consolidated into ''R440 - Config A'. Same holds true for all models (R450 - Config A, R750xs - Config I, etc...). I also audited all the existing items on generic entries (example: Dell PowerEdge R750) and found their entries only have custom builds, or found them to need to be moved from generic to Config specific entries (about 30 of them). This is now complete, just CCing you in via phab notifications so you are aware of completion and ready to move forward on all new supermicro orders into netbox.

PowerEdge R730 - 1 custom config
PowerEdge R650 - 2 custom config
PowerEdge R660 - 4 custom config
PowerEdge R630 - predates standard configs
PowerEdge R750 - had 8 devices that should have been 'PowerEdge R750 - ConfigI' so moved them, 1 custom device
PowerEdge R750xs - had 30 devices half split to configI of same model and the other half to ConfigH of same model.

Network: https://netbox.wikimedia.org/extras/reports/results/5914311/ : Errors but none in caching sites.

Cables: No errors: https://netbox.wikimedia.org/extras/reports/results/5914371/
Rack: No errors: https://netbox.wikimedia.org/extras/reports/results/5914357/
Librenms: No errors: https://netbox.wikimedia.org/extras/reports/results/5914376/
Mgmt Console: No errors: https://netbox.wikimedia.org/extras/reports/results/5914361/

Accounting Report: https://netbox.wikimedia.org/extras/reports/results/5914366/

Looks good to me on this end, thank you!

@Jclark-ctr or @VRiley-WMF: Would one of you be able to take care of this on your next on-site visit? We have light on the drmrs side, so you should get light and a link connection as soon as this is patched in.

Added the mr1 for magru, and now all items on the error report are for Accounting to add to the main ledger area (they handle that) and everything below the threshhold is now done.

The msws don't have any serial number in the elevation document (wasn't populated with other serials( or on the racking directions (where some serials were listed). I'm not sure what they are for the two msws, so unless @Jhancock.wm or @Papaul wrote those serials down elsewhere in their notes, I'll have to put in a remote hands to find out the serials.

Please note as of 2024-05-30 all DC-Ops tasks related to ops-magru are completed and the only thing keeping this task from resolution is A/B testing via tasks T362421, T359054, & T363722.

WMF quote request for Dell USA - NIC upgrades to existing wikikube-ctrl fleet in eqiad - T366204

Dell Team,

We have three hosts in our Ashburn facility we would like to install Broadcom 57414 Dual Port 10/25GbE NICs into these three older hosts.

R440 service tags: 9WXL4Z2, DFRB8B3, DHH98B3

Ideally we'd like to install (1) Broadcom 57414 Dual Port 10/25GbE NIC per host. However, if the 57414 isn't available for these, we can go with the Broadcom 57416 or 57412 (depending on what you can sell to install into these older R440s).

Would you please check for one of the above Broadcom NICs to quote (3) for installation into these hosts?

• Site: eqiad (Ashburn)
• Ref: T366204

Thanks in advance,

Dell Team,

Actually, we need a second quote for this exact same thing for our servers in our other site codfw.

So updating this request for 2 quotes:
Quote 1

• eqiad service tags: 9WXL4Z2, DFRB8B3, DHH98B3
• (3) Broadcom 57414 Dual Port 10/25GbE NIC per
• Site: eqiad (Ashburn, VA)
• Ref: T366204

Quote 2

• codfw service tags: 1JYB613, 1JXB613, GZ298B3
• (3) Broadcom 57414 Dual Port 10/25GbE NIC per
• Site: codfw (Carolton, TX)
• Ref: T366205

Thanks in advance!

There are a few things to note:

• wikikube-ctrl1001 purchased in 2019-09-09 via T228214. - 5 years service on 2024-09-09.
• wikikube-ctrl100[23] purcahsed in 2021-02-10 via T271155. - 5 years service on 2025-02-10
  • One host is near end of its typical service life of 5 years, but we've extended that out past 5 now on many hosts so this is no longer a hard blocker to upgrade.
• All are currently located in 1G racks, so each would have to be migrated to a 10G capable rack once the NIC is installed for re-imaging.
• Quote reqeuested, once returned can review with yourself and Willy for budgeting and capacity planning.

SSDs confirmed onsite by shipping, so I can go onsite whenever we schedule to take and install the SSD upgrades.

@wiki_willy: We should be aware of this as it will affect 50+ servers in both codfw and eqiad, and we may see an increase in power usage across these fleets.

My SSH/HTTPS Tunnel proxy addon is broken and I've yet to get it working properly again. Are you able to pull up via proxy https://wikikubectrl1001.mgmt.eqiad.wmnet and update the idrac firmware to https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=rttph&oscode=wst14&productcode=poweredge-r440 please?