Page MenuHomePhabricator

RobH (Rob Halsell)
Senior Data Center EngineerAdministrator

Projects (18)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Nov 24 2014, 1:43 PM (435 w, 1 d)
Roles
Administrator
Availability
Available
IRC Nick
RobH
LDAP User
RobH
MediaWiki User
RobH [ Global Accounts ]

My GPG Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7 0245 D22A

I am an Senior Data Center Engineer on Wikimedia's Data Center SRE Team.

Please note that private message via phabricator is not my preferred contact means. Please feel free to contact me (robh) directly via irc/freenode, or email my @wikimedia.org email address.

Recent Activity

Yesterday

RobH added a comment to T333135: Offboard nfraison.

@jbond, this task isn't editable by most users (so i cannot remove the invalid project), please remove the procurement project.

Mon, Mar 27, 12:00 PM · SRE, Infrastructure-Foundations, Infrastructure Security
RobH added a comment to T333135: Offboard nfraison.

Not sure why procurement was added (so it showed up in my notifications) as this user isn't in the acl*procurement review, they are in the acl*sre-team so I took the liberty of removing them since I was checking the two groups that give access to the procurement space.

Mon, Mar 27, 12:00 PM · SRE, Infrastructure-Foundations, Infrastructure Security
RobH removed a member for acl*sre-team: nfraison.
Mon, Mar 27, 11:58 AM

Tue, Mar 21

RobH reopened T325153: cookbooks: sre.hosts.reboot-single update to support disabled puppet as "Open".

Please note we've run into this issue again today:

Tue, Mar 21, 2:38 PM · Infrastructure-Foundations, SRE-tools
RobH edited P45901 firmware update cookbook.
Tue, Mar 21, 1:19 PM
RobH created P45901 firmware update cookbook.
Tue, Mar 21, 1:15 PM

Mon, Mar 20

RobH edited Description on grizzly-sprint.
Mon, Mar 20, 1:33 PM
RobH edited Description on grizzly-sprint.
Mon, Mar 20, 1:32 PM
RobH edited Description on grizzly-sprint.
Mon, Mar 20, 1:31 PM
RobH changed the edit policy for grizzly-sprint.
Mon, Mar 20, 1:30 PM
RobH edited Description on grizzly-sprint.
Mon, Mar 20, 1:27 PM

Tue, Mar 14

RobH added a comment to T331860: Urgent: Two failed disks in ms-be2040.

Since this is out of warranty, the pending purchase of 5 disks was raised to 7 on T331988 to accommodate this repair.

Tue, Mar 14, 12:14 PM · ops-codfw, SRE-swift-storage, SRE, DC-Ops

Mon, Mar 13

RobH changed the edit policy for grizzly-sprint.
Mon, Mar 13, 5:23 PM
RobH created grizzly-sprint.
Mon, Mar 13, 5:22 PM
RobH added a comment to T331886: Q4/Q1:knams racking elevations & planning.

Note that we will only have one mx480 (current cr3-esams), the other router will be the current cr3-knams (mx204).

Future refresh will replaced that mx480 with a device similar to a mx204. So it might be better to keep the mx204 at the top, as well as space for the mx480 replacement.

Ideally we should also have 1 patch panel per rack.

Mon, Mar 13, 5:07 PM · netops, Traffic, SRE, ops-knams, DC-Ops, Infrastructure-Foundations
RobH added a subtask for T329219: Main Tracking Task for ESAMS Migration to KNAMS: T331886: Q4/Q1:knams racking elevations & planning.
Mon, Mar 13, 5:04 PM · SRE, ops-knams, DC-Ops
RobH added a parent task for T331886: Q4/Q1:knams racking elevations & planning: T329219: Main Tracking Task for ESAMS Migration to KNAMS.
Mon, Mar 13, 5:04 PM · netops, Traffic, SRE, ops-knams, DC-Ops, Infrastructure-Foundations
RobH moved T331886: Q4/Q1:knams racking elevations & planning from Backlog to Racking Tasks on the ops-knams board.
Mon, Mar 13, 5:04 PM · netops, Traffic, SRE, ops-knams, DC-Ops, Infrastructure-Foundations
RobH updated the task description for T331886: Q4/Q1:knams racking elevations & planning.
Mon, Mar 13, 4:36 PM · netops, Traffic, SRE, ops-knams, DC-Ops, Infrastructure-Foundations
RobH created T331886: Q4/Q1:knams racking elevations & planning.
Mon, Mar 13, 3:44 PM · netops, Traffic, SRE, ops-knams, DC-Ops, Infrastructure-Foundations

Thu, Mar 9

RobH added a comment to T328957: eqsin & ulsfo: new R450s drawing far more power than R440s (power over contracted caps in both sites).

We chatted about this during the last knams sync up call, as our racks there have a higher cap due to this.

Thu, Mar 9, 3:56 PM · SRE, ops-eqsin, ops-ulsfo, DC-Ops

Wed, Mar 8

RobH renamed ops-knams from ops-esams to ops-knams.
Wed, Mar 8, 8:09 PM
RobH closed T323092: decommission bast5001.wikimedia.org as Resolved.

This is in 'decom' state with all the other now decommissioned eqsin hardware. I'm resolving this task as everything for decom is now tracking disposal via T330313

Wed, Mar 8, 4:14 PM · decommission-hardware
RobH changed the status of Unknown Object (Task), a subtask of T329219: Main Tracking Task for ESAMS Migration to KNAMS, from In Progress to Open.
Wed, Mar 8, 1:41 PM · SRE, ops-knams, DC-Ops
RobH added a comment to T331519: Should we have two versions of the Juniper QFX5120-48Y in Netbox?.

+1 to fixing them as separate entries as its easy enough to add more models to netbox. Particularly since Homer can report them as such as well!

Wed, Mar 8, 1:25 PM · Infrastructure-Foundations, netbox

Tue, Mar 7

RobH added a subtask for T329219: Main Tracking Task for ESAMS Migration to KNAMS: Unknown Object (Task).
Tue, Mar 7, 5:21 PM · SRE, ops-knams, DC-Ops
RobH changed the status of Unknown Object (Task), a subtask of T329219: Main Tracking Task for ESAMS Migration to KNAMS, from Open to In Progress.
Tue, Mar 7, 1:31 PM · SRE, ops-knams, DC-Ops

Mon, Mar 6

RobH moved T331358: Audit future knams power usage from Backlog to Procurement on the ops-knams board.
Mon, Mar 6, 9:46 PM · DC-Ops, SRE, ops-knams
RobH added projects to T331358: Audit future knams power usage: SRE, DC-Ops.
Mon, Mar 6, 9:46 PM · DC-Ops, SRE, ops-knams
RobH triaged T331358: Audit future knams power usage as Medium priority.
Mon, Mar 6, 9:46 PM · DC-Ops, SRE, ops-knams

Thu, Mar 2

RobH reassigned T330984: mr1-ulsfo dead? from RobH to ayounsi.

They pulled power, plugged it back in, and now its online and responsive.

Thu, Mar 2, 12:49 PM · SRE, ops-ulsfo
RobH moved T330984: mr1-ulsfo dead? from Backlog to Hardware Failure / Repair on the ops-ulsfo board.

Case Order #00839073

Thu, Mar 2, 12:21 PM · SRE, ops-ulsfo
RobH added a comment to T330984: mr1-ulsfo dead?.

The scs is also only accessible via mr1-ulsfo, so I'll put in a remote hands for them to power cycle it manually.

Thu, Mar 2, 12:15 PM · SRE, ops-ulsfo

Tue, Feb 28

RobH closed T299443: Q3: rack/setup/install dumpsdata100[67] as Resolved.

both hosts online and ready for your use!

Tue, Feb 28, 5:01 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops
RobH closed T299443: Q3: rack/setup/install dumpsdata100[67], a subtask of T308027: private repo deployment - perccli implementation, as Resolved.
Tue, Feb 28, 5:01 PM · Infrastructure-Foundations, SRE, DC-Ops
RobH updated the task description for T299443: Q3: rack/setup/install dumpsdata100[67].
Tue, Feb 28, 5:01 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops
RobH updated subscribers of T299443: Q3: rack/setup/install dumpsdata100[67].
Tue, Feb 28, 4:35 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops
RobH added a comment to T299443: Q3: rack/setup/install dumpsdata100[67].

Whatever array you create first on the new 15th gen raid controller gets the higher ID, which is the opposite of 14th generation raid controller.

Tue, Feb 28, 4:34 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops
RobH added a comment to T299443: Q3: rack/setup/install dumpsdata100[67].
  • dumpdata1007 has ssds raid1 virtual drive id 238, hdds raid10 virtual drive id 239, installs fine.
Tue, Feb 28, 2:55 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops

Mon, Feb 27

RobH added a comment to T299443: Q3: rack/setup/install dumpsdata100[67].
Virtual Disk 238: RAID10, 43.661TB, Ready, Initialization 42%
Mon, Feb 27, 7:07 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops
RobH added a comment to T299443: Q3: rack/setup/install dumpsdata100[67].
Virtual Disk 238: RAID10, 43.661TB, Ready, Initialization 6%                  
Virtual Disk 239: RAID1, 446.625GB, Ready
Mon, Feb 27, 3:43 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops
RobH added a comment to T299443: Q3: rack/setup/install dumpsdata100[67].

LVM data still exists on disks from a previous failed install attempt and the dd method didn't seem to remove, suspended instllation on dumpsdata1006 and set it to fully initialize all the disks, so I'll set a timer and come back to this in about 2 hours.

Mon, Feb 27, 3:05 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops
RobH changed the status of T299443: Q3: rack/setup/install dumpsdata100[67] from Open to In Progress.

If I have an overwhelming number of notifications in a short period (seems I did around January 18th) I may miss direct pings where the task isn't assigned to me. Accidental of course!

Mon, Feb 27, 2:17 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops
RobH changed the status of T299443: Q3: rack/setup/install dumpsdata100[67], a subtask of T308027: private repo deployment - perccli implementation, from Open to In Progress.
Mon, Feb 27, 2:17 PM · Infrastructure-Foundations, SRE, DC-Ops

Feb 23 2023

RobH removed projects from T327812: eqsin hosts are not rebooting when running sre.hosts.reimage cookbook: ops-eqsin, DC-Ops.

This seems to have stalled with an inconclusive assumption that the NIC firmware update solved it, but we have no confirmation of that (see @ssingh's last comment above).

Feb 23 2023, 12:41 PM · SRE, Traffic

Feb 16 2023

RobH moved T329865: Q3:rack/setup/install cloudlb200[23]-dev from Backlog to Racking Tasks on the ops-codfw board.
Feb 16 2023, 5:55 PM · SRE, cloud-services-team (Hardware), ops-codfw, DC-Ops
RobH added a parent task for T329865: Q3:rack/setup/install cloudlb200[23]-dev: Unknown Object (Task).
Feb 16 2023, 5:55 PM · SRE, cloud-services-team (Hardware), ops-codfw, DC-Ops
RobH created T329865: Q3:rack/setup/install cloudlb200[23]-dev.
Feb 16 2023, 5:54 PM · SRE, cloud-services-team (Hardware), ops-codfw, DC-Ops
RobH added a parent task for T329863: Q3:rack/setup/install cloudvirtlocal10[01-03]: Unknown Object (Task).
Feb 16 2023, 5:47 PM · ops-eqiad, SRE, cloud-services-team (Hardware), DC-Ops
RobH created T329863: Q3:rack/setup/install cloudvirtlocal10[01-03].
Feb 16 2023, 5:46 PM · ops-eqiad, SRE, cloud-services-team (Hardware), DC-Ops

Feb 8 2023

RobH updated the task description for T326362: Q3:rack/setup/install mw2420-mw2451.
Feb 8 2023, 8:21 PM · ops-codfw, SRE, serviceops, DC-Ops
RobH updated the task description for T326362: Q3:rack/setup/install mw2420-mw2451.
Feb 8 2023, 8:11 PM · ops-codfw, SRE, serviceops, DC-Ops
RobH updated the task description for T326362: Q3:rack/setup/install mw2420-mw2451.
Feb 8 2023, 7:59 PM · ops-codfw, SRE, serviceops, DC-Ops
RobH updated the task description for T326362: Q3:rack/setup/install mw2420-mw2451.
Feb 8 2023, 7:53 PM · ops-codfw, SRE, serviceops, DC-Ops
RobH added a subtask for T329219: Main Tracking Task for ESAMS Migration to KNAMS: Unknown Object (Task).
Feb 8 2023, 6:32 PM · SRE, ops-knams, DC-Ops
RobH added subtasks for T329219: Main Tracking Task for ESAMS Migration to KNAMS: Unknown Object (Task), Unknown Object (Task), Unknown Object (Task), Unknown Object (Task).
Feb 8 2023, 6:31 PM · SRE, ops-knams, DC-Ops

Feb 7 2023

RobH closed T324555: cr2-drmrs:xe-0/1/1 stuck optic as Resolved.

remote hands successfully removed the optic this AM and placed it in our racks, we'll just have it thrown away next remote hands work we need. router appears fine.

Feb 7 2023, 5:00 PM · SRE, Infrastructure-Foundations, netops, ops-drmrs

Feb 6 2023

RobH added a comment to T328957: eqsin & ulsfo: new R450s drawing far more power than R440s (power over contracted caps in both sites).

Ok, pulling some peak numbers off the idrac https interface which includes weekly graphing and peaks.

Feb 6 2023, 11:12 PM · SRE, ops-eqsin, ops-ulsfo, DC-Ops
RobH moved T328957: eqsin & ulsfo: new R450s drawing far more power than R440s (power over contracted caps in both sites) from Backlog to Hardware Failure / Repair on the ops-ulsfo board.
Feb 6 2023, 8:31 PM · SRE, ops-eqsin, ops-ulsfo, DC-Ops
RobH moved T328957: eqsin & ulsfo: new R450s drawing far more power than R440s (power over contracted caps in both sites) from Backlog to Hardware Failure / Repair on the ops-eqsin board.
Feb 6 2023, 8:31 PM · SRE, ops-eqsin, ops-ulsfo, DC-Ops
RobH updated subscribers of T328957: eqsin & ulsfo: new R450s drawing far more power than R440s (power over contracted caps in both sites).
Feb 6 2023, 8:31 PM · SRE, ops-eqsin, ops-ulsfo, DC-Ops
RobH updated the task description for T328957: eqsin & ulsfo: new R450s drawing far more power than R440s (power over contracted caps in both sites).
Feb 6 2023, 8:07 PM · SRE, ops-eqsin, ops-ulsfo, DC-Ops
RobH triaged T328957: eqsin & ulsfo: new R450s drawing far more power than R440s (power over contracted caps in both sites) as High priority.
Feb 6 2023, 8:04 PM · SRE, ops-eqsin, ops-ulsfo, DC-Ops

Jan 31 2023

RobH updated the task description for T324555: cr2-drmrs:xe-0/1/1 stuck optic.
Jan 31 2023, 7:36 PM · SRE, netops, Infrastructure-Foundations, ops-drmrs
RobH raised the priority of T324555: cr2-drmrs:xe-0/1/1 stuck optic from Low to Medium.
Jan 31 2023, 7:33 PM · SRE, netops, Infrastructure-Foundations, ops-drmrs
RobH added a comment to T324555: cr2-drmrs:xe-0/1/1 stuck optic.

Support,

We have three items for remote hands to accomplish for us on this request:

  1. Please pickup DEL0117661, unpackage it into our racks, and photograph both the package contents and the packing slip. Ensure packing slip matches contents.
  1. Inventory the spare optics in our racks (placed on top of the servers) and give us a full count of the number/type of spare optics, patch cables, and power cables.
  1. Troubleshoot removal of a broken optic in our router cr2-drmrs:xe-0/1/1, located in B13:U45. The optic was reported by remote hands to be stuck, missing its lever to eject the optic.

I've attached photos of the same type of optic in use. Can support attempt to remove this optic by using a small flat screwdriver or probe to manually work the ejection mechanism? The attached photos and video show this operation.

If removal isn't possible, we'll have to put in a support case with Juniper to swap the entire router, causing disruptive maintainance, so we'd prefer to avoid that if possible.

Also can the attempt to remove the optic take place when our Network Admin is around in the event of unexpected downtime. Please have this work take place on Tuesday, February 7th @ 9AM local time to ensure we have network coverage.

Scheduling: 1 & 2 can be done at anytime but please restrict the optic troubleshooting to Tuesday, February 7th starting at 9AM CET.

Jan 31 2023, 6:34 PM · SRE, netops, Infrastructure-Foundations, ops-drmrs

Jan 27 2023

RobH changed the status of T324555: cr2-drmrs:xe-0/1/1 stuck optic from Open to In Progress.

Neglected to do this earlier this week, I have the photos so I'll work on this today.

Jan 27 2023, 2:27 PM · SRE, netops, Infrastructure-Foundations, ops-drmrs

Jan 25 2023

RobH closed T327877: Disable NETBIOS on some IPMI as Resolved.
Jan 25 2023, 4:41 PM · Data-Persistence, cloud-services-team (Hardware), SRE, ops-codfw, ops-eqiad
RobH added a comment to T327877: Disable NETBIOS on some IPMI.

Arzhel linked to some docs and commented netbios is called wins in HP ilom, and I had noticed the wins enablement under IPv4 so disabled and the issue went away for db1140. Going to disable on the rest of them.

Jan 25 2023, 3:57 PM · Data-Persistence, cloud-services-team (Hardware), SRE, ops-codfw, ops-eqiad
RobH added a comment to T327877: Disable NETBIOS on some IPMI.

I've disabled the multicast discovery on the ilom interface for db1140 (10.65.1.134) as a test to see if it stops the netbios port broadcasts from the ilom interfaces.

Jan 25 2023, 3:09 PM · Data-Persistence, cloud-services-team (Hardware), SRE, ops-codfw, ops-eqiad
RobH added projects to T327877: Disable NETBIOS on some IPMI: cloud-services-team (Hardware), Data-Persistence.
Jan 25 2023, 2:24 PM · Data-Persistence, cloud-services-team (Hardware), SRE, ops-codfw, ops-eqiad
RobH moved T327877: Disable NETBIOS on some IPMI from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Jan 25 2023, 2:24 PM · Data-Persistence, cloud-services-team (Hardware), SRE, ops-codfw, ops-eqiad
RobH moved T327877: Disable NETBIOS on some IPMI from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Jan 25 2023, 2:24 PM · Data-Persistence, cloud-services-team (Hardware), SRE, ops-codfw, ops-eqiad
RobH added a comment to T327877: Disable NETBIOS on some IPMI.

Please note all 5 of these host are old HP ProLiants. I'm not sure where this setting is on these hosts, but I'm assuming in the bios and each of these will require downtime/reboot to disable.

Jan 25 2023, 2:24 PM · Data-Persistence, cloud-services-team (Hardware), SRE, ops-codfw, ops-eqiad

Jan 24 2023

RobH updated the task description for T325824: decommission atlas-ulsfo.
Jan 24 2023, 6:07 PM · SRE, ops-ulsfo, decommission-hardware
RobH closed T325824: decommission atlas-ulsfo as Resolved.

added details to the recycled tab in the datacenter asset tags tracking google sheet and delted from netbox.

Jan 24 2023, 6:07 PM · SRE, ops-ulsfo, decommission-hardware
RobH closed T327765: Procurement Tickets Access Request for pfischer as Resolved.

@pfischer: I've added you to #acl*procurement-review and you can now access tasks in the S4 procurement space:

Jan 24 2023, 1:58 PM · SRE, SRE-Access-Requests
RobH added a member for acl*procurement-review: pfischer.
Jan 24 2023, 1:56 PM

Jan 23 2023

RobH closed T307026: decommission atlas-esams as Declined.

device resurrected itself, decom task declined as its now reporting into ripe portal

Jan 23 2023, 4:15 PM · DC-Ops, SRE, ops-knams, Infrastructure-Foundations, decommission-hardware
RobH closed T307026: decommission atlas-esams, a subtask of T303242: ripe-atlas-esams down, as Declined.
Jan 23 2023, 4:15 PM · SRE, DC-Ops, ops-knams

Jan 20 2023

lmata awarded T319434: service implementation tracking: arclamp1001.eqiad.wmnet a Love token.
Jan 20 2023, 2:13 PM · SRE Observability (FY2022/2023-Q3), Patch-For-Review, User-fgiunchedi, observability, SRE
lmata awarded T313849: Q1:rack/setup/install logstash103[67] a Love token.
Jan 20 2023, 1:54 PM · SRE Observability, observability, ops-eqiad, SRE, DC-Ops

Jan 18 2023

herron awarded T313849: Q1:rack/setup/install logstash103[67] a Love token.
Jan 18 2023, 8:12 PM · SRE Observability, observability, ops-eqiad, SRE, DC-Ops
RobH moved T327295: Q3:rack/setup/install an-worker11[49-56] from Backlog to Racking Tasks on the ops-eqiad board.
Jan 18 2023, 3:35 PM · Shared-Data-Infrastructure, Data-Engineering-Planning, SRE, ops-eqiad, DC-Ops
RobH added a parent task for T327295: Q3:rack/setup/install an-worker11[49-56]: Unknown Object (Task).
Jan 18 2023, 3:35 PM · Shared-Data-Infrastructure, Data-Engineering-Planning, SRE, ops-eqiad, DC-Ops
RobH created T327295: Q3:rack/setup/install an-worker11[49-56].
Jan 18 2023, 3:35 PM · Shared-Data-Infrastructure, Data-Engineering-Planning, SRE, ops-eqiad, DC-Ops

Jan 13 2023

RobH added a parent task for T326965: Q4:rack/setup/install backup2010, backup2011: Unknown Object (Task).
Jan 13 2023, 4:14 PM · SRE, Data-Persistence, bacula, Data-Persistence-Backup, ops-codfw, DC-Ops
RobH moved T326965: Q4:rack/setup/install backup2010, backup2011 from Backlog to Racking Tasks on the ops-codfw board.
Jan 13 2023, 4:14 PM · SRE, Data-Persistence, bacula, Data-Persistence-Backup, ops-codfw, DC-Ops
RobH created T326965: Q4:rack/setup/install backup2010, backup2011.
Jan 13 2023, 4:14 PM · SRE, Data-Persistence, bacula, Data-Persistence-Backup, ops-codfw, DC-Ops

Jan 12 2023

RobH removed a project from T326846: Q3:rack/setup/install ms-fe1013 - ms-fe1014, thanos-fe1004: serviceops.
Jan 12 2023, 8:07 PM · SRE, SRE-swift-storage, ops-eqiad, DC-Ops
RobH closed T326847: serviceops implementation tracking for ms-fe1013 - ms-fe1014, thanos-fe1004 as Invalid.

in valid this is actually data persistence i had it mislabeled

Jan 12 2023, 8:07 PM · serviceops, SRE-swift-storage
RobH closed T326847: serviceops implementation tracking for ms-fe1013 - ms-fe1014, thanos-fe1004, a subtask of T326846: Q3:rack/setup/install ms-fe1013 - ms-fe1014, thanos-fe1004, as Invalid.
Jan 12 2023, 8:07 PM · SRE, SRE-swift-storage, ops-eqiad, DC-Ops
RobH closed T326849: serviceops implementation tracking for ms-fe2013 - ms-fe2014, thanos-fe2004 as Invalid.

actually data persistence this was a mis categorization

Jan 12 2023, 8:06 PM · serviceops, SRE-swift-storage
RobH closed T326849: serviceops implementation tracking for ms-fe2013 - ms-fe2014, thanos-fe2004, a subtask of T326848: Q3:rack/setup/install ms-fe2013 - ms-fe2014, thanos-fe2004, as Invalid.
Jan 12 2023, 8:06 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops
RobH created T326849: serviceops implementation tracking for ms-fe2013 - ms-fe2014, thanos-fe2004.
Jan 12 2023, 6:47 PM · serviceops, SRE-swift-storage
RobH added a parent task for T326848: Q3:rack/setup/install ms-fe2013 - ms-fe2014, thanos-fe2004: Unknown Object (Task).
Jan 12 2023, 6:46 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops
RobH moved T326848: Q3:rack/setup/install ms-fe2013 - ms-fe2014, thanos-fe2004 from Backlog to Racking Tasks on the ops-codfw board.
Jan 12 2023, 6:45 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops
RobH created T326848: Q3:rack/setup/install ms-fe2013 - ms-fe2014, thanos-fe2004.
Jan 12 2023, 6:45 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops
RobH created T326847: serviceops implementation tracking for ms-fe1013 - ms-fe1014, thanos-fe1004.
Jan 12 2023, 6:42 PM · serviceops, SRE-swift-storage
RobH added a parent task for T326846: Q3:rack/setup/install ms-fe1013 - ms-fe1014, thanos-fe1004: Unknown Object (Task).
Jan 12 2023, 6:41 PM · SRE, SRE-swift-storage, ops-eqiad, DC-Ops