Thanks for testing this out @Papaul and @Marostegui. So if it's a setting issue that needed to be changed to fix this, is there a way we can run a script to see if any of the other servers still need to be changed to "A/B Grid redundant"?

To close the loop, we received a credit back for $269.17

Thanks for the heads up @fgiunchedi! If this helps performance and optimization, a half kW increase per cabinet isn't bad at all. Thanks for working on this!

Thanks for the heads up @ayounsi, I went ahead and submitted an email to them requesting for the credits.

Thanks @tappof! Looks like they're showing up now

Hi @tappof - that's correct, it's 3-phase. I'm not sure why it's not showing up on the ServerTech site, but here's the quote from one of the orders that we placed: T382370

Hi @tappof - sorry about that, it was a typo. It was missing a "2" and should be C2L36TE-DQME2M99. Here's a list of all the PDUs below in their rack locations below:

Adding ops-eqiad and tagging @VRiley-WMF and @Jclark-ctr

@RobH - can you request quote for the disk? Since we're in the middle of figuring out whether we should extend out the server life cycle for things, it might be good to just buy the disk for this one.

Hi @klausman - to follow up, do you need 25g on this one because it's a MI350X instead of a MI300X? Just trying to see what changed, and if we need to order the same optics for the GPU server being set to be delivered in Q3/Q4.

Hi @VRiley-WMF - it looks like the PDUs in E9 thru E14 aren't showing up in Grafana graphs below. After you configure the PDUs in E13 and E14, can you check to see if something else needs to be done to get the PDU power numbers to show up here:

Assigning to Rob, and replied back to the Digital Realty email that he'll be the point person.

Definitely, I've added this server to the list that Dell's Platform Engineering team is looking into

Adding ops-eqiad tag and assigning to Valerie.

Hey @Jhancock.wm - following up on the troubleshooting call you had with Dell Tech Support the other day on the db2209 host; is there anything there that you learned, which can be applied to this host? Thanks, WIlly

Thanks for looping me in. Before I approve though, just wanted to provide space for any comments or concerns that anyone from SRE might have.

Chatted with @RobH, who will follow up on this one. Thanks, Willy

Hey @cmooney - thanks for getting this on the radar. Totally makes sense to me, so I think we can just do this upgrade during the same time as the drmrs refresh. The catch here though is that we're still trying to figure out when to do the overall drmrs refresh...if we're going to proceed in Q2, push to later in the FY, push to next FY, etc. So whenever we schedule it, we should definitely upgrade the links to 100g.

@MoritzMuehlenhoff - sure, no problem. Repurposing it works for us as well. As long as we have all the updates in Phabricator, I think that's all we really need for tracking. Procedural wise (to John's point above), treating it as a rename/reimage would work for us too.

In regards to buying a new CPU - we don't have any more budget available for FY25-26, but I'm ok with going over budget if this is the best route forward. We'll have four additional cp servers (as part of the initiative for AI scraping and DDoS) scheduled in Q2 as well, if that helps with things. Just let me know you're preference on how to best proceed. Thanks, Willy

Thanks @elukey, I went ahead and sent it over to Ken from Supermicro, so that he can try to push this along a bit quicker.

Hey @elukey - do you have the Supermicro case number for this one? Thanks, Willy

@Jclark-ctr & @VRiley-WMF - can you provide a status on this one?

Thanks @Clement_Goubert! @VRiley-WMF & @Jclark-ctr - this is the hardware being repurposed that I mentioned about during our Dc-Ops meeting yesterday. Thanks, Willy

Thanks @Jclark-ctr. Hi @isarantopoulos - since we're looking to refresh this soon, do you still need us to purchase a replacement drive? Thanks, Willy

Thanks @RLazarus, we'll get it added to agenda for Infra Foundations meeting on Monday, March 30.

Adding the ops-eqiad tag and removing ops-eqdfw. @Jclark-ctr will take a look at it a bit later today.

Hi @ssingh - I was thinking along the lines of seeing if we would be able to calculate SERT ourselves instead of installing the tool. When I dig around a bit, it looks like SERT takes the weighted geometric mean of 65% for CPU, 30% for Memory, and 5% for Storage workloads. Since we only have 25 servers, I was thinking an estimate might be good enough; and we could just let them know that's how we came up with the metric since we can't install SERT on our hosts. Another alternative, we could also consolidate all our questions and concerns together, and Rob could email it out to customercare@ to see if the support team can provide any other guidance.

Hey @ssingh - I guess technically, the way it's worded refers to just "new "servers. However, it'd also be a little weird if they're asking for just "new" because I don't think we provided this info last year either. Since our footprint at caching sites are super small, do you think providing an estimate would be possible? When we provide them the info, we could explain the reasoning behind it as well.

I was reading the notification for DRMRS a bit more closely, and it looks like March 31 is the due date for Digital Realty to report the data to the EU, but the due date for us to provide the info to Digital Realty is this Friday, March 13. Updating the subject line to reflect the date.

Hi @BTullis - sure, that sounds like a good test plan. One thing to keep in mind though is the data center switchover (from codfw to eqiad) will be happening on March 24 and 25. We'll likely see an overall power increase of 15-20kW at eqiad during that time, so hopefully that's enough time to observe any potentially changes or findings by then. Here's what I typically use for monitoring power by racks (at the bottom of the page):

Yeah, it could be because we're using new PDU models for cabinets E9-E14. Let's see what the Observability team recommends.

Sounds good, thanks @brouberol !

Sure @BTullis - no problem at all. Can you do me a favor though and let us know which hostnames on the EOL document, referencing this ticket number? That way, we'll leave it off of our radar. Thanks, Willy

Sounds good @RobH, that plan works for me as well. Do you know if Jin has access to any of these parts by any chance? If he is able to get a hold of them, he could just add the cost onto our invoice.

Hey @RobH - did Jin say what kind of initial troubleshooting he did? Like did he do a power drain, reseat certain parts, etc? I think we can go ahead and purchase parts to see if it'll help fix this, though it'll be helpful knowing what was attempted so far. Thanks, Willy

Swap out R430 spare drives with newer drives (1 for 1 swap), along with memory

@Jclark-ctr - can you help out @Marostegui with getting a RMA for the DIMM?

Hi @VRiley-WMF - the access to create RMA cases should be resolved now per Juniper, so hopefully it unblocks you on this one. Thanks, Willy

Hi @brouberol - thanks for opening this task. Is this one ready to be handed over to DC-Ops? Thanks, Willy

Thanks @jasmine_ !

Awesome, thanks so much @jasmine_ !

Hi @Clement_Goubert & @jasmine_ - to follow up on this one, I think we're still waiting on this task to be passed over to Dc-Ops. Can you split this into two different tasks (one for ops-eqiad and one for ops-codfw), for us to unrack the servers? Much appreciated in advance. Thanks, Willy

Hi @jasmine_ - just checking if you had an ETA on wrapping up wikikube-ctrl1001 for decommissioning? We're hoping to have this Phabricator task passed over to Dc-Ops, to help free up some rack space in eqiad. Much appreciated in advance. Thanks, Willy

Hi @brouberol & @BTullis - I don't think we've seen the Phabricator task for Data Center ops to decommission these servers from the racks. Can you submit that over to us via the Decom workflow below so we can unrack these to free up some rackspace:

Thanks @RobH. Our account team has changed quite a bit, but you can follow up with Hossam and Dawn after creating the support ticket

++ @RobH - can you work with John on getting a 25g Broadcom NIC for this one?

Adding @BTullis and @Stevemunene for feedback on an appropriate window for an-worker1128

Awesome, thank you!

Thanks @MoritzMuehlenhoff!

Hi @MoritzMuehlenhoff - are you able to help confirm the racking details and update site.pp on this one? Thanks, Willy

Hi @MoritzMuehlenhoff - are you able to confirm the racking details and update the site.pp info on this one? Thanks, Willy

Hi @Jclark-ctr - can you provide info on where the controllers from T393941 are, so that you and @VRiley-WMF can work with Matthew on the controller swap? Thanks, Willy

Resolving task, we will be installing two new Fundraising cabinets as a solution instead.

Hi @Jhancock.wm - since @Papaul is out on sabbatical, can you take a look at this one? It's related to debugging some of the Supermicro issues.. Thanks, Willy

Re-opening. @Jhancock.wm - per @Marostegui's previous comment:

Hi @elukey - can you or @Volans send me an email summarizing everything you need from Dell? I'll add the Technical Account Rep to the email thread to loop you in with him.

If we don't find the issue we'd probably need to contact Dell to verify if we need to do something extra or not. @wiki_willy Hi! This is the task about IDRAC 10 that we were discussing the other day, we'd probably need to get in touch with DELL to figure out what we have to do :(

Hey @Volans - I think we've come up with a couple solutions since this task was created. One is providing a monthly Netbox dump to the Accounting team, so that they can see which hosts have been set to "offline" since the previous month. And the second one is creating an ongoing EOL Server list, to track down SRE teams that haven't decommissioned their hardware after the hardware refresh. I think we can resolve this task, but maybe we can brainstorm some other ways of improving the EOL Server list on the side.