Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

They make it very difficult to cancel:

• called into 800-480-8365, they do nothing but open a ticket on our behalf
• the account retension team called to try to convince me to stay
  • "I am just an employee and don't make decisions, I am canceling this because I was told to do so."
  • Not quite true but it is the short answer ;D
• 1-3 business days for the cancelallation and billing team to call me back, so I should get a call from them later this week.

Dell has confirmed case update and will dispatch a new mainboard and cpu bracket. Once they do, they'll email/update with tracking and then dispatch will reach back out to schedule the third unisys site visit.

I'll put a more detailed timeline and update tomorrow but as it stands now:

Mainboard swap will occur on Wednesday, April 8th @ 10:00Singapore time which is Tuesday, Tuesday April 7th 18:00 Pacific.

Please note 'S480845X3505676' is NOT a valid serial under Supermicro support, but S480845X4915849 is.

This host was purchased 2024-08-07, so it is still under warranty. If Papaul doesn't know how to use the SUM (I've never used it) then the support ticket is the way to go.

I've gotten back the following links to the whitepapers for our currently used SSDs and NVMe offerings:

Who is best to address and fix this bug?

description: Rule: Port with no description on access switch Faults: #1: ge-0/0/47 - ge-0/0/47

Fixed on Friday, synced up in meeting today and no morre errors. Cathal closing the ticket on the Lumen portal.

Account cancellation cannot be accomplished in the portal, I'll have to call in later today.

Summary:

• EdgeUno says they see no errors only our flap
• Arzhel replied back stating that we are still seeing errors, stressed that we've already swapped optics and fibers on our end and re-requested they do the same as I did in the original request.
• They replied back about 45 minutes or so ago stating 'We will check our side as requested and let you know.'

Please note the ticket was opened but their portal doesn't seem to email myself, Arzhel, or Cathal even though I listed all three of us on the ticket. The only way to see ticket updates is to login to the actual ticket view: https://edgeuno.cloud/tickets.php/view/484278000266995093

The optic was swapped, but the errors resumed.

Errors returned, Arzhel redrained the link, update sent to ticket:

Comentário gerado em Smart Hands: Good afternoon,

We carried out the replacement of the fiber optic patch cable. A 10‑meter patch cable available in Rack B03 was used.
Attached are the evidences of the activity performed.

Inventory of materials available in Rack B03:

07 units of fiber optic patch cables – 2 meters
02 units of MPO patch cables – 1 meter
02 units QFX‑SFP‑10GE‑LR
01 unit JNP‑SFP‑25G‑LR
01 unit QFX‑SFP‑1GE‑T

Support,

The link came back up after your cleaning and re-seating the optic and patch cable, but the errors have resumed after the circuit came back online.

Next step, we would like to have you swap patch cable 70152 with a spare patch cable in our rack (should be on top of the servers) for this link at your earliest possible convenience. Please let us know the cable ID of this new patch, if it doesn't have one, please apply ID 260301.

The link is currently depooled (will show a link light but it is not serving traffic). When you replace the fiber patch cable, it should resume its link light.

Please also check in our racks and report back an inventory of how many spare fiber optic patch cables and lengths we have, along with the spare optics. These should be in our racks on top of the servers.

This work can take place at any time. Thank you in advance!

This also looks like its no longer throwing errors, but I've done nothing:

Remote Hands Directions:
I can write up the directions for them to pull the patch and clean it, and also reseat the optic in the port. However, all the patch IDs so far in magru have been incorrect when photos are matched to netbox, so we'll likely need to tell them the port and optic serial, and suggest the patch ID and note it may differ. They'll take photos at our request to match the patch cable, so someone needs to be around and answering the tickets in case they require another confirmation.

Remote hands cleaned the patch cable and reseated the optic along with photos to show the work.

The distro swap did not fix this host, it will require a mainboard swap via a procurement task (linked in)

Sent a gentle followup to the Dell team today.

Confirm with @cmooney via IRC that 70152 is indeed xe-0//0 in these photos and updated the remote hands for Wednesday.

I want to ensure I'm reading the photos correctly, but the update from remote hands is the fiber ID 70091 wasn't found, and it appears to me that the fiber ID for the patch in xe-0/1/0 is 70152.

Ok, bios update done and its booting to the debian loader so handing back to @MoritzMuehlenhoff

Updated the idrac, then the backplane firmware, and as it was rebooting to update the BIOS firmware the SEL updated with:

They had an issue where they couldn't locate the fiber listed and instead skipped the work entirely! I need to review the photos and find out what the patch is actually labeled and confirm they need to remove it.

Tech is onsite and performing the hw power distro board swap on cp5022

Tech is running late, their dispatcher called me to let me know. They were set to be onsite at 7AM, but it will now be closer to 10:30AM / 19:30 Pacific

They'll bill us less though so already improvement thank you! I've gone ahead and updated the checklist so it is a bit more detailed on what has happened and next steps.

Please note there are (3) racking tasks for the (3) orders of cloudcephosd hosts in eqiad that have just been placed. Assumptions have been made about hostname, simply picking the next in sequence. Please review each of these racking tasks and update the racking info for network and rack placement in addition to the boilderplate below:

Please note there are (3) racking tasks for the (3) orders of cloudcephosd hosts in eqiad that have just been placed. Assumptions have been made about hostname, simply picking the next in sequence. Please review each of these racking tasks and update the racking info for network and rack placement in addition to the boilderplate below:

Please note there are (3) racking tasks for the (3) orders of cloudcephosd hosts in eqiad that have just been placed. Assumptions have been made about hostname, simply picking the next in sequence. Please review each of these racking tasks and update the racking info for network and rack placement in addition to the boilderplate below:

With the migration of status.wikimedia.org, are we good to kill this? It has tripled in cost over the last year amd is overdue for extension in coupa.

After spending about 30 minutes on the Dell site I'm not locating the usual whitepapers the Dell Team sent us back when we selected the SSDs years and years ago, so I've asked for them directly:

Please note the maint window for this offline host is 2026-03-13 @ 07:00 AM Singapore / which is 5PM Thursday evening for me. I'll be online to remotely supervise the swap and attempt to login to the idrac when done.

Hi Rob,

This is to confirm that we received the files, and this have been shared with the relevant team.

Thank you.

Regards,

ROSCHELLE SHIELLA LOTO
Customer Care Associate - Special Projects

After firmware updates and resetting the SEL and rebooting the issue now seems to have cleared up.

I had some ISP issues with upload speeds to magru, so Papaul helped me out and flashed the firmware for idrac, bios, and backplane. The error persists, so I'm now doing a TSR report collection and download before resetting the logs and seeing if it returns.

Ok, that is annoying, these auto created tasks cannot have things appended into the task descirption or phaultfinder removes it...

To:CustomerCare@digitalrealty.com
Wikimedia metrics for Energy Efficiency Directive (EED) - AMS17 and MRS02
Customer Care,

We (Wikimedia) received a notice that we were required to fill out these templates for our customer accounts/deployments in AMS17 and MRS02. I've done so according to directions and am now submitting to CustomerCare@digitalrealty.com.

Please advise if any further action is required at this time.

Thank you in advance,

Thanks everyone for the feedback, I'll fill out the templates and submit them over!

Dell will require all the firmware and such be the latest versions before they call it a failure, so I'll steal this and update the firmware on this host and pull the logs and see if it clears it up or not.

In T413409#11695960, @ayounsi wrote:

Rob, could you investigate those as well. Same as T415743: Inbound errors on interface cr2-magru:xe-0/1/0 (Transit: EdgeUno (E1-SER-7853-IP) {#70091}). Please sync up with us to drain the link ahead of time.

Not sure who wants to take point on this, but since I chatted briefly with Arzhel in IRC I'll default to him and ya'll can reassign as needed!

Remote hands ticket CS1254900 filed:

Apologies this was neglected. Since we need to likely give 24 hours notice for smart hands to avoid expedite fees, I suggest we schedule this for Monday, 2026-03-16.

In T418411#11696664, @ayounsi wrote:

I added rough network numbers.

Sent an email to investigate the return/repair of the two optics

They swapped the optic GT3AAG00314 out of the switch for optic GT3AAG00316 and now the link shows up:

The order is placed and I'm currently scheduling the Unisys/Dell engineer to go onsite sometime between Friday-Wednesday of this/next week. Host is hard down, so no traffic intervention required.

Support,

You have swapped the optic on the router side, and the MPO patch cable. The link is still down, so we'd like you to swap the optic on the switch side. The switch is located in B3:U37::asw1-b3-magru port et-0/0/50. Please remove the optic serial GT3AAG00314 in asw1-b3-magru port et-0/0/50 and swap it with another 100G optic spare from our rack.

Please place the optic serial GT3AAG00314 in an envelope marked T119524-switch and set aside in our racks until we determine what caused the link failure.

They've now replace the patch cable but we're still seeing down:

Support,

Thank you, we can see the old module QSFP-100GBASE-SR4 SN GT3AAG00321 was removed and replaced with QSFP-100GBASE-SR4 module GT3AAG00315. However, the link is still showing down for us.

Acknowledged that all optics are actually QSFP-100GBASE-SR4, my initial listing of 40G was incorrect.

Thank you for the photos, as they show the new optic did not resolve the issue and the link is still offline (red) not online (green).
Please re-seat both sides of the patch cable and check for link light. If the link light doesn't go green, please source a new fiber patch from our rack spares, note its label (and report it back to us) and then replace patch cable ID 70130 with a new patch.

Ok, they swapped the optic in cr2-magru but still shows down:

CS1253254 filed, listed myself, Arzhel, Cathal, and Papaul on the CC list.

I'll work on this now.

Set to failed.