Page MenuHomePhabricator

RobH (Rob Halsell)
Senior Data Center EngineerAdministrator

Today

  • No visible events.

Tomorrow

  • No visible events.

Saturday

  • No visible events.

User Details

User Since
Nov 24 2014, 1:43 PM (620 w, 3 d)
Roles
Administrator
Availability
Available
IRC Nick
RobH
LDAP User
RobH
MediaWiki User
RobH [ Global Accounts ]

My GPG Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7 0245 D22A

I am an Senior Data Center Engineer on Wikimedia's Data Center SRE Team.

Please note that private message via phabricator is not my preferred contact means. Please feel free to contact me (robh) directly via irc/freenode, or email my @wikimedia.org email address.

Recent Activity

Today

RobH changed the status of Unknown Object (Task), a subtask of T440305: eqiad: WMCS switch refresh 2026 (C8 & D5), from Stalled to Open.
Thu, Oct 15, 4:38 PM · DC-Ops, netops, ops-eqiad, Infrastructure-Foundations, SRE

Yesterday

RobH added a comment to T431374: cloudvirt1071 crash, again.

Recommended next steps by onsite:

Wed, Oct 14, 2:45 PM · SRE, ops-eqiad, tools-infrastructure-team, DC-Ops, cloud-services-team, Cloud-VPS
RobH moved T431374: cloudvirt1071 crash, again from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Wed, Oct 14, 2:40 PM · SRE, ops-eqiad, tools-infrastructure-team, DC-Ops, cloud-services-team, Cloud-VPS

Tue, Oct 13

RobH added a comment to T436499: ulsfo: OOB migration from copper to fiber.

They missed some settings on their end, IPv4 is up but IPv6 still down. Followup sent to email thread.

Tue, Oct 13, 10:39 PM · SRE, DC-Ops, ops-ulsfo
RobH closed T436668: Inbound errors on interface asw1-23-ulsfo:mgmt0 (Core: msw2-ulsfo:18) as Resolved.
Tue, Oct 13, 10:38 PM · SRE, DC-Ops, ops-ulsfo
RobH closed T436669: Inbound errors on interface asw1-22-ulsfo:mgmt0 (Core: msw1-ulsfo:18) as Resolved.
Tue, Oct 13, 10:38 PM · SRE, DC-Ops, ops-ulsfo
RobH added a comment to T440999: Netbox: modify move-server script to allow for 1G->10G upgrade during process.

Edit addition.

Tue, Oct 13, 5:29 PM · netops, Infrastructure-Foundations, SRE
RobH added a comment to T440999: Netbox: modify move-server script to allow for 1G->10G upgrade during process.

Cathal and I chatted about this in IRC so I'm summarizing my points here for task visibility.

Tue, Oct 13, 5:22 PM · netops, Infrastructure-Foundations, SRE
RobH added a comment to T436499: ulsfo: OOB migration from copper to fiber.

I've sent a followup to the migration thread with the vendor, cc'd in all of NetOps, to determine whats up.

Tue, Oct 13, 5:13 PM · SRE, DC-Ops, ops-ulsfo
RobH assigned T441013: Q3:rack/setup/install civi1003 to Jgreen.

Please double-check and fill out the racking details section of this racking task.

Tue, Oct 13, 5:05 PM · SRE, fundraising-tech-ops, ops-eqiad, DC-Ops
RobH added a parent task for T441013: Q3:rack/setup/install civi1003: Unknown Object (Task).
Tue, Oct 13, 5:04 PM · SRE, fundraising-tech-ops, ops-eqiad, DC-Ops
RobH created T441013: Q3:rack/setup/install civi1003.
Tue, Oct 13, 5:04 PM · SRE, fundraising-tech-ops, ops-eqiad, DC-Ops
RobH added a comment to T436499: ulsfo: OOB migration from copper to fiber.

Rob Halsell,

Your case # 01207593: Remote hands for IP Transit migration from copper to fiber has been updated with the following:

"Created by: mmariscalmata Fibers have been swapped. TX/RX has been properly aligned."
Tue, Oct 13, 4:54 PM · SRE, DC-Ops, ops-ulsfo

Wed, Oct 7

RobH changed the status of Unknown Object (Task), a subtask of T440305: eqiad: WMCS switch refresh 2026 (C8 & D5), from Open to Stalled.
Wed, Oct 7, 7:00 PM · DC-Ops, netops, ops-eqiad, Infrastructure-Foundations, SRE
RobH updated the task description for T436499: ulsfo: OOB migration from copper to fiber.
Wed, Oct 7, 5:39 PM · SRE, DC-Ops, ops-ulsfo

Tue, Oct 6

RobH updated the task description for T436499: ulsfo: OOB migration from copper to fiber.
Tue, Oct 6, 8:47 PM · SRE, DC-Ops, ops-ulsfo
RobH updated subscribers of T436499: ulsfo: OOB migration from copper to fiber.

Ok, confirmed the parts have arrived.

Tue, Oct 6, 8:13 PM · SRE, DC-Ops, ops-ulsfo
RobH moved T436499: ulsfo: OOB migration from copper to fiber from Procurement to Hardware Failure / Repair on the ops-ulsfo board.
Tue, Oct 6, 7:25 PM · SRE, DC-Ops, ops-ulsfo
RobH updated the task description for T440349: esams router and switch maint.
Tue, Oct 6, 3:34 PM · Infrastructure-Foundations, DC-Ops, SRE, netops, ops-esams
RobH updated the task description for T440349: esams router and switch maint.
Tue, Oct 6, 3:33 PM · Infrastructure-Foundations, DC-Ops, SRE, netops, ops-esams
RobH updated the task description for T440349: esams router and switch maint.
Tue, Oct 6, 3:28 PM · Infrastructure-Foundations, DC-Ops, SRE, netops, ops-esams
RobH updated subscribers of T440349: esams router and switch maint.
Tue, Oct 6, 3:28 PM · Infrastructure-Foundations, DC-Ops, SRE, netops, ops-esams
RobH created T440349: esams router and switch maint.
Tue, Oct 6, 3:23 PM · Infrastructure-Foundations, DC-Ops, SRE, netops, ops-esams

Fri, Oct 2

RobH assigned T440030: Q3:rack/setup/install lvs102[1-4] to ssingh.

I've made some assumptions regarding hostnames and networking. Please review and also provide racking deails.

Fri, Oct 2, 7:28 PM · SRE, Traffic, ops-eqiad, DC-Ops
RobH created T440030: Q3:rack/setup/install lvs102[1-4].
Fri, Oct 2, 7:27 PM · SRE, Traffic, ops-eqiad, DC-Ops

Wed, Sep 30

RobH renamed T439754: Q2:rack/setup/install (3) dumps100[123] from Q2:rack/setup/install (3) new cloud dump hosts to Q2:rack/setup/install (3) dumps100[123].
Wed, Sep 30, 5:41 PM · cloud-services-team (Hardware), SRE, ops-eqiad, DC-Ops
RobH added a parent task for T439754: Q2:rack/setup/install (3) dumps100[123]: Unknown Object (Task).
Wed, Sep 30, 5:40 PM · cloud-services-team (Hardware), SRE, ops-eqiad, DC-Ops
RobH assigned T439754: Q2:rack/setup/install (3) dumps100[123] to Andrew.

In the racking details for these 3 new hosts you listed hostnames dumps100[123], but we already have existing 'clouddumps100[12]' so shouldn't these be clouddumps100[345]?

Wed, Sep 30, 5:40 PM · cloud-services-team (Hardware), SRE, ops-eqiad, DC-Ops
RobH created T439754: Q2:rack/setup/install (3) dumps100[123].
Wed, Sep 30, 5:38 PM · cloud-services-team (Hardware), SRE, ops-eqiad, DC-Ops

Tue, Sep 29

RobH placed T438253: Q2:rack/setup/install ml-serve1017 up for grabs.
Tue, Sep 29, 8:45 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, SRE, ops-eqiad, Machine-Learning-Team, DC-Ops

Mon, Sep 28

RobH added a subtask for T438952: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/e23/s4: Unknown Object (Task).
Mon, Sep 28, 3:17 PM · SRE, ops-eqiad, DC-Ops
RobH added a comment to T438952: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/e23/s4.

@RobH - can you request quote for the disk? Since we're in the middle of figuring out whether we should extend out the server life cycle for things, it might be good to just buy the disk for this one.

Oh, OK. I didn't realise that the warranty had expired. We could survive without it, but a replacement would be nice.

Mon, Sep 28, 3:17 PM · SRE, ops-eqiad, DC-Ops
RobH added a parent task for T439428: Q3:rack/setup/install 8 new wikikube hosts: Unknown Object (Task).
Mon, Sep 28, 2:51 PM · ServiceOps-Upgrades-Hardware, ServiceOps, SRE, ops-codfw, DC-Ops
RobH created T439430: 8 wikikube in codfw imlementation tracking.
Mon, Sep 28, 2:51 PM · ServiceOps-Upgrades-Hardware, ServiceOps, SRE
RobH created T439429: 26 wikikube in eqiad imlementation tracking.
Mon, Sep 28, 2:50 PM · ServiceOps-Upgrades-Hardware, ServiceOps, SRE
RobH renamed T439427: Q3:rack/setup/install 26 wikikube hosts from Q#:rack/setup/install 26 wikikube hosts to Q3:rack/setup/install 26 wikikube hosts.
Mon, Sep 28, 2:50 PM · ServiceOps-Upgrades-Hardware, ServiceOps, SRE, ops-eqiad, DC-Ops
RobH assigned T439428: Q3:rack/setup/install 8 new wikikube hosts to Clement_Goubert.

Please review and update this rack task. This task requires update for all hostnames, racking location, and checklist entries for hostnames.

Mon, Sep 28, 2:49 PM · ServiceOps-Upgrades-Hardware, ServiceOps, SRE, ops-codfw, DC-Ops
RobH created T439428: Q3:rack/setup/install 8 new wikikube hosts.
Mon, Sep 28, 2:49 PM · ServiceOps-Upgrades-Hardware, ServiceOps, SRE, ops-codfw, DC-Ops
RobH added a parent task for T439427: Q3:rack/setup/install 26 wikikube hosts: Unknown Object (Task).
Mon, Sep 28, 2:47 PM · ServiceOps-Upgrades-Hardware, ServiceOps, SRE, ops-eqiad, DC-Ops
RobH assigned T439427: Q3:rack/setup/install 26 wikikube hosts to Clement_Goubert.

Please review and update this rack task. This task requires update for all hostnames, racking location, and checklist entries for hostnames.

Mon, Sep 28, 2:46 PM · ServiceOps-Upgrades-Hardware, ServiceOps, SRE, ops-eqiad, DC-Ops
RobH created T439427: Q3:rack/setup/install 26 wikikube hosts.
Mon, Sep 28, 2:45 PM · ServiceOps-Upgrades-Hardware, ServiceOps, SRE, ops-eqiad, DC-Ops

Fri, Sep 25

RobH added a parent task for T439293: Q3:rack/setup/install ms-fe102[5-8]: Unknown Object (Task).
Fri, Sep 25, 10:45 PM · SRE-swift-storage, Data-Persistence, SRE, ops-eqiad, DC-Ops
RobH assigned T439293: Q3:rack/setup/install ms-fe102[5-8] to MatthewVernon.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

Fri, Sep 25, 10:45 PM · SRE-swift-storage, Data-Persistence, SRE, ops-eqiad, DC-Ops
RobH created T439293: Q3:rack/setup/install ms-fe102[5-8].
Fri, Sep 25, 10:44 PM · SRE-swift-storage, Data-Persistence, SRE, ops-eqiad, DC-Ops
RobH assigned T439292: Q3:rack/setup/install ms-fe202[5-8] to MatthewVernon.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

Fri, Sep 25, 10:43 PM · SRE, SRE-swift-storage, Data-Persistence, ops-codfw, DC-Ops
RobH added a parent task for T439292: Q3:rack/setup/install ms-fe202[5-8]: Unknown Object (Task).
Fri, Sep 25, 10:42 PM · SRE, SRE-swift-storage, Data-Persistence, ops-codfw, DC-Ops
RobH created T439292: Q3:rack/setup/install ms-fe202[5-8].
Fri, Sep 25, 10:42 PM · SRE, SRE-swift-storage, Data-Persistence, ops-codfw, DC-Ops

Mon, Sep 21

RobH added a parent task for T436285: Q1:rack/setup/install cirrussearch11[26-30]: Unknown Object (Task).
Mon, Sep 21, 5:07 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, SRE, ops-eqiad, DC-Ops

Thu, Sep 17

RobH renamed T438368: Q1:rack/setup/install ms-be110[1-4] from Q1:rack/setup/install ms-be110[1-6] to Q1:rack/setup/install ms-be110[1-4].
Thu, Sep 17, 3:23 PM · SRE-swift-storage, ops-eqiad, SRE, Data-Persistence, DC-Ops
RobH renamed T438370: Q1:rack/setup/install ms-be2[099-104] from Q1:rack/setup/install ms-be2[099-102] to Q1:rack/setup/install ms-be2[099-104].
Thu, Sep 17, 3:23 PM · SRE-swift-storage, SRE, Data-Persistence, ops-codfw, DC-Ops
RobH removed a parent task for T438368: Q1:rack/setup/install ms-be110[1-4]: Unknown Object (Task).
Thu, Sep 17, 3:22 PM · SRE-swift-storage, ops-eqiad, SRE, Data-Persistence, DC-Ops
RobH removed a parent task for T438370: Q1:rack/setup/install ms-be2[099-104]: Unknown Object (Task).
Thu, Sep 17, 3:22 PM · SRE-swift-storage, SRE, Data-Persistence, ops-codfw, DC-Ops
RobH added a parent task for T438370: Q1:rack/setup/install ms-be2[099-104]: Unknown Object (Task).
Thu, Sep 17, 3:21 PM · SRE-swift-storage, SRE, Data-Persistence, ops-codfw, DC-Ops
RobH added a parent task for T438368: Q1:rack/setup/install ms-be110[1-4]: Unknown Object (Task).
Thu, Sep 17, 3:21 PM · SRE-swift-storage, ops-eqiad, SRE, Data-Persistence, DC-Ops
RobH added a parent task for T438370: Q1:rack/setup/install ms-be2[099-104]: Unknown Object (Task).
Thu, Sep 17, 3:20 PM · SRE-swift-storage, SRE, Data-Persistence, ops-codfw, DC-Ops
RobH assigned T438370: Q1:rack/setup/install ms-be2[099-104] to MatthewVernon.

I've moved the ordering tasks along as they were going to expire if not placed to order, and I've now taken some assumptions on hostnames to create the racking task. Please review the above information in the task description and correct as needed.

Thu, Sep 17, 3:20 PM · SRE-swift-storage, SRE, Data-Persistence, ops-codfw, DC-Ops
RobH created T438370: Q1:rack/setup/install ms-be2[099-104].
Thu, Sep 17, 3:19 PM · SRE-swift-storage, SRE, Data-Persistence, ops-codfw, DC-Ops
RobH added a parent task for T438368: Q1:rack/setup/install ms-be110[1-4]: Unknown Object (Task).
Thu, Sep 17, 3:17 PM · SRE-swift-storage, ops-eqiad, SRE, Data-Persistence, DC-Ops
RobH assigned T438368: Q1:rack/setup/install ms-be110[1-4] to MatthewVernon.

I've moved the ordering tasks along as they were going to expire if not placed to order, and I've now taken some assumptions on hostnames to create the racking task. Please review the above information in the task description and correct as needed.

Thu, Sep 17, 3:16 PM · SRE-swift-storage, ops-eqiad, SRE, Data-Persistence, DC-Ops
RobH created T438368: Q1:rack/setup/install ms-be110[1-4].
Thu, Sep 17, 3:15 PM · SRE-swift-storage, ops-eqiad, SRE, Data-Persistence, DC-Ops

Wed, Sep 16

RobH added a parent task for T438253: Q2:rack/setup/install ml-serve1017: Unknown Object (Task).
Wed, Sep 16, 9:51 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, SRE, ops-eqiad, Machine-Learning-Team, DC-Ops
RobH assigned T438253: Q2:rack/setup/install ml-serve1017 to isarantopoulos.

Please double check the racking details for this, as I made assumptions based on previous gpu hosts.

Wed, Sep 16, 9:51 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, SRE, ops-eqiad, Machine-Learning-Team, DC-Ops
RobH updated the task description for T438253: Q2:rack/setup/install ml-serve1017.
Wed, Sep 16, 9:50 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, SRE, ops-eqiad, Machine-Learning-Team, DC-Ops
RobH renamed T438253: Q2:rack/setup/install ml-serve1017 from Q#:rack/setup/install X to Q2:rack/setup/install ml-serve1017.
Wed, Sep 16, 9:50 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, SRE, ops-eqiad, Machine-Learning-Team, DC-Ops
RobH created T438253: Q2:rack/setup/install ml-serve1017.
Wed, Sep 16, 9:49 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, SRE, ops-eqiad, Machine-Learning-Team, DC-Ops
RobH added a project to T438230: Cookbook issues with cumin1003: Infrastructure-Foundations.
Wed, Sep 16, 7:01 PM · SRE, Infrastructure-Foundations, ops-eqiad, DC-Ops
RobH added a parent task for T438213: Q1:rack/setup/install 10 new ceph nodes: Unknown Object (Task).
Wed, Sep 16, 5:24 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30), SRE, ops-eqiad, DC-Ops
RobH assigned T438213: Q1:rack/setup/install 10 new ceph nodes to BTullis.

Please update this task description, checklists, and title with the new hostnames, as well as the racking details section.

Wed, Sep 16, 5:24 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30), SRE, ops-eqiad, DC-Ops
RobH created T438213: Q1:rack/setup/install 10 new ceph nodes.
Wed, Sep 16, 5:22 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30), SRE, ops-eqiad, DC-Ops

Sep 15 2026

RobH updated the task description for T438052: Jin onsite work for 2026-09-16 @ 09:00 UTC.
Sep 15 2026, 7:26 PM · Infrastructure-Foundations, netops, Traffic, SRE, ops-eqsin
RobH updated the task description for T438052: Jin onsite work for 2026-09-16 @ 09:00 UTC.
Sep 15 2026, 5:54 PM · Infrastructure-Foundations, netops, Traffic, SRE, ops-eqsin
RobH updated the task description for T438052: Jin onsite work for 2026-09-16 @ 09:00 UTC.
Sep 15 2026, 5:53 PM · Infrastructure-Foundations, netops, Traffic, SRE, ops-eqsin
RobH moved T438052: Jin onsite work for 2026-09-16 @ 09:00 UTC from Hardware Failure / Repair to Racking Tasks on the ops-eqsin board.
Sep 15 2026, 5:51 PM · Infrastructure-Foundations, netops, Traffic, SRE, ops-eqsin
RobH moved T438052: Jin onsite work for 2026-09-16 @ 09:00 UTC from Backlog to Hardware Failure / Repair on the ops-eqsin board.
Sep 15 2026, 3:59 PM · Infrastructure-Foundations, netops, Traffic, SRE, ops-eqsin
RobH updated the task description for T438052: Jin onsite work for 2026-09-16 @ 09:00 UTC.
Sep 15 2026, 3:53 PM · Infrastructure-Foundations, netops, Traffic, SRE, ops-eqsin
RobH created T438052: Jin onsite work for 2026-09-16 @ 09:00 UTC.
Sep 15 2026, 3:47 PM · Infrastructure-Foundations, netops, Traffic, SRE, ops-eqsin
RobH moved T432317: Fix cables with placeholders names and planned status from Hardware Failure / Repair to Racking Tasks on the ops-eqsin board.
Sep 15 2026, 3:38 PM · SRE, DC-Ops, ops-eqsin, ops-eqdfw, ops-codfw, ops-magru, ops-eqiad, ops-esams, ops-drmrs, ops-ulsfo
RobH moved T432317: Fix cables with placeholders names and planned status from Backlog to Hardware Failure / Repair on the ops-eqsin board.
Sep 15 2026, 3:37 PM · SRE, DC-Ops, ops-eqsin, ops-eqdfw, ops-codfw, ops-magru, ops-eqiad, ops-esams, ops-drmrs, ops-ulsfo
RobH closed T433602: Unresponsive management for ganeti5007.mgmt:22 as Declined.
Sep 15 2026, 3:37 PM · SRE, ops-eqsin
RobH closed T416499: Unresponsive management for cp5022.mgmt:22 as Declined.
Sep 15 2026, 3:37 PM · SRE, ops-eqsin
RobH closed T423724: EQSIN:Switch refresh diagram and wiring, a subtask of T418439: EQSIN:New switch setup/configuration, as Resolved.
Sep 15 2026, 3:36 PM · Infrastructure-Foundations, SRE, netops, ops-eqsin, DC-Ops
RobH closed T423724: EQSIN:Switch refresh diagram and wiring as Resolved.
Sep 15 2026, 3:36 PM · Infrastructure-Foundations, SRE, netops, ops-eqsin, DC-Ops
RobH closed T436182: Inbound errors on interface cr2-eqsin:et-0/0/2 (Core: asw1-604-eqsin:ethernet-1/55) as Declined.
Sep 15 2026, 3:36 PM · SRE, ops-eqsin
RobH closed T437800: Inbound errors on interface cr2-eqsin:xe-0/1/1 (Transit: Tata (1028127)) as Declined.
Sep 15 2026, 3:36 PM · SRE, ops-eqsin

Sep 14 2026

RobH updated the task description for T437915: sessionstore1005.eqiad.wmnet is down (again).
Sep 14 2026, 8:47 PM · SRE, DC-Ops, ops-eqiad, Cassandra

Sep 10 2026

RobH added a comment to T418012: eqiad row A/B switch refresh prep.
  • All switches except the lsw and ssw in A8 are racked, connected to SCS

Can we confirm what ports the lsw and ssw in rack A1 are connected to? Port 17 - labelled asw-a1-eqiad - does not seem to have anything connected? Plus no port on the scs is labelled ssw1-a1-eqiad, and that switch in Netbox shows nothing connected to its console port.

The remaining switches, apart from those in A8, are reachable from management and configured.

Sep 10 2026, 4:39 PM · Infrastructure-Foundations, netops, DC-Ops, SRE, ops-eqiad
RobH added a comment to T418012: eqiad row A/B switch refresh prep.

IRC Sync update:

Sep 10 2026, 4:14 PM · Infrastructure-Foundations, netops, DC-Ops, SRE, ops-eqiad

Sep 8 2026

RobH moved T436499: ulsfo: OOB migration from copper to fiber from Backlog to Procurement on the ops-ulsfo board.

Turns out the reply may not have included all the right parties so I did another reply today and included the additional parties on the DR side.

Sep 8 2026, 8:15 PM · SRE, DC-Ops, ops-ulsfo
RobH reassigned T431651: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet from RobH to ssingh.

@ssingh: The new memory is dectected, same speed as the old just I can compare the serial output to the photos, throwing the photos here for reference in future. New memory installed, 2 spare dimms left to use, and the old memory set aside in case it wasn't the issue. You can resolve this task once you are aware, thanks!

Sep 8 2026, 7:06 PM · Traffic, ops-drmrs, DC-Ops
RobH updated the task description for T431651: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet.
Sep 8 2026, 7:05 PM · Traffic, ops-drmrs, DC-Ops
RobH added a comment to T431651: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet.

This had remote hands update that it was done, double checking the work now.

Sep 8 2026, 6:59 PM · Traffic, ops-drmrs, DC-Ops

Sep 3 2026

RobH added a comment to T436966: Need raid fw files on cumin to upgrade cloudcephosd1045-1047.

Pushed SAS-RAID_Firmware_2WJX5_WN64_52.38.0-6897_A13_01.EXE into the R660 storage directory on cumin1003. should be able to run and see it there and on all other cumin hosts after the nightly rsync.

Sep 3 2026, 6:17 PM · tools-infrastructure-team, Cloud-VPS
RobH assigned T436964: Q1:rack/setup/install dse-k8s-worker10[29-38] to BTullis.
Sep 3 2026, 5:41 PM · Essential-Work, Data-Platform-SRE (2026-08-28 - 2026-09-18), SRE, ops-eqiad, DC-Ops
RobH added a parent task for T436964: Q1:rack/setup/install dse-k8s-worker10[29-38]: Unknown Object (Task).
Sep 3 2026, 5:41 PM · Essential-Work, Data-Platform-SRE (2026-08-28 - 2026-09-18), SRE, ops-eqiad, DC-Ops
RobH created T436964: Q1:rack/setup/install dse-k8s-worker10[29-38].
Sep 3 2026, 5:40 PM · Essential-Work, Data-Platform-SRE (2026-08-28 - 2026-09-18), SRE, ops-eqiad, DC-Ops

Sep 2 2026

RobH added a comment to T431651: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet.

Dear customer,

We acknowledge receipt of your case and will proceed to complete your requested task within the timeframe you specified. We will update you upon completion or reach out if we have any questions.

Best Regards,

Sep 2 2026, 2:17 PM · Traffic, ops-drmrs, DC-Ops
RobH assigned T436675: magru: faulty link between cr1-magru and asw1-b3-magru to ayounsi.

remote hands update:

Sep 2 2026, 12:48 AM · ops-magru

Sep 1 2026

RobH updated the task description for T431651: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet.
Sep 1 2026, 4:42 PM · Traffic, ops-drmrs, DC-Ops
RobH added a comment to T431651: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet.

There is a 48 hour minimum window to have remote hands work on things without an urgency fee applied. I picked Friday with a note that the work can actually take place at any time. Typically they'll read that and just get to it sometime in the next day or so.

Sep 1 2026, 4:40 PM · Traffic, ops-drmrs, DC-Ops
RobH added a comment to T431651: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet.

The ram landed yesterday, filing remote hands shortly to work on this host.

Sep 1 2026, 4:29 PM · Traffic, ops-drmrs, DC-Ops