Page MenuHomePhabricator

RobH (Rob Halsell)
Operations EngineerAdministrator

Projects (19)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Nov 24 2014, 1:43 PM (360 w, 6 d)
Roles
Administrator
Availability
Available
IRC Nick
RobH
LDAP User
RobH
MediaWiki User
RobH [ Global Accounts ]

My GPG Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7 0245 D22A

I am an Operations Engineer on Wikimedia's Datacenter Operations Team.

I also am the primary triage engineer for the hardware-requests project, as well as the private S4 procurement space and procurement project.

All questions involving allocation of hardware can be initially addressed on https://wikitech.wikimedia.org/wiki/Operations_requests.

Please note that private message via phabricator is not my preferred contact means. Please feel free to contact me (robh) directly via irc/freenode, or email my @wikimedia.org email address.

Recent Activity

Fri, Oct 22

RobH assigned T294154: Q2:(Need By: TBD) rack/setup/install elastic20[61-72] to Papaul.
Fri, Oct 22, 9:15 PM · SRE, Discovery-Search, Elasticsearch, ops-codfw, DC-Ops
RobH added a parent task for T294154: Q2:(Need By: TBD) rack/setup/install elastic20[61-72]: Unknown Object (Task).
Fri, Oct 22, 9:14 PM · SRE, Discovery-Search, Elasticsearch, ops-codfw, DC-Ops
RobH renamed T294154: Q2:(Need By: TBD) rack/setup/install elastic20[61-72] from (Need By: TBD) rack/setup/install elastic20[61-72] to Q2:(Need By: TBD) rack/setup/install elastic20[61-72].
Fri, Oct 22, 9:14 PM · SRE, Discovery-Search, Elasticsearch, ops-codfw, DC-Ops
RobH created T294154: Q2:(Need By: TBD) rack/setup/install elastic20[61-72].
Fri, Oct 22, 9:13 PM · SRE, Discovery-Search, Elasticsearch, ops-codfw, DC-Ops
RobH assigned T294152: Q2:(Need By: 2021-12-17) rack/setup/install elastic108[4-8] to Jclark-ctr.
Fri, Oct 22, 8:49 PM · SRE, Discovery-Search, Elasticsearch, ops-eqiad, DC-Ops
RobH added a parent task for T294152: Q2:(Need By: 2021-12-17) rack/setup/install elastic108[4-8]: Unknown Object (Task).
Fri, Oct 22, 8:48 PM · SRE, Discovery-Search, Elasticsearch, ops-eqiad, DC-Ops
RobH created T294152: Q2:(Need By: 2021-12-17) rack/setup/install elastic108[4-8].
Fri, Oct 22, 8:47 PM · SRE, Discovery-Search, Elasticsearch, ops-eqiad, DC-Ops
RobH assigned T294139: Q2:(Need By: TBD) rack/setup/install ganeti202[78].codfw.wmnet to Papaul.
Fri, Oct 22, 5:12 PM · SRE, ops-codfw, DC-Ops
RobH created T294139: Q2:(Need By: TBD) rack/setup/install ganeti202[78].codfw.wmnet.
Fri, Oct 22, 5:12 PM · SRE, ops-codfw, DC-Ops
RobH renamed T293909: Q2:(Need By: TBD) rack/setup/install ganeti102[5-8] from (Need By: TBD) rack/setup/install ganeti102[56] to (Need By: TBD) rack/setup/install ganeti102[5-8].
Fri, Oct 22, 5:04 PM · SRE, Infrastructure-Foundations, ops-eqiad, DC-Ops
RobH added a parent task for T293909: Q2:(Need By: TBD) rack/setup/install ganeti102[5-8]: Unknown Object (Task).
Fri, Oct 22, 5:02 PM · SRE, Infrastructure-Foundations, ops-eqiad, DC-Ops
RobH assigned T294137: Q2:(Need By: TBD) rack/setup/install ms-fe1009-1012 to Jclark-ctr.
Fri, Oct 22, 4:34 PM · SRE, SRE-swift-storage, ops-eqiad, DC-Ops
RobH added a parent task for T294137: Q2:(Need By: TBD) rack/setup/install ms-fe1009-1012: Unknown Object (Task).
Fri, Oct 22, 4:34 PM · SRE, SRE-swift-storage, ops-eqiad, DC-Ops
RobH created T294137: Q2:(Need By: TBD) rack/setup/install ms-fe1009-1012.
Fri, Oct 22, 4:33 PM · SRE, SRE-swift-storage, ops-eqiad, DC-Ops
RobH updated the task description for T294136: Q2:(Need By: TBD) rack/setup/install ms-fe20[09-12].
Fri, Oct 22, 4:33 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops
RobH updated the task description for T294136: Q2:(Need By: TBD) rack/setup/install ms-fe20[09-12].
Fri, Oct 22, 4:33 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops
RobH assigned T294136: Q2:(Need By: TBD) rack/setup/install ms-fe20[09-12] to Papaul.
Fri, Oct 22, 4:26 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops
RobH renamed T294136: Q2:(Need By: TBD) rack/setup/install ms-fe20[09-12] from (Need By: TBD) rack/setup/install ms-fe20[09-12] to Q2:(Need By: TBD) rack/setup/install ms-fe20[09-12].
Fri, Oct 22, 4:25 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops
RobH moved T294136: Q2:(Need By: TBD) rack/setup/install ms-fe20[09-12] from Backlog to Racking Tasks on the ops-codfw board.
Fri, Oct 22, 4:25 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops
RobH added a parent task for T294136: Q2:(Need By: TBD) rack/setup/install ms-fe20[09-12]: Unknown Object (Task).
Fri, Oct 22, 4:25 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops
RobH created T294136: Q2:(Need By: TBD) rack/setup/install ms-fe20[09-12].
Fri, Oct 22, 4:25 PM · SRE, SRE-swift-storage, ops-codfw, DC-Ops

Wed, Oct 20

RobH changed the status of Unknown Object (Task), a subtask of T292630: Eqiad Expansion - LVS Connectivity Options, from Open to Stalled.
Wed, Oct 20, 11:11 PM · SRE, netops, Infrastructure-Foundations
RobH updated subscribers of T293938: (Need By: TBD) rack/setup/install an-test-coord1002.

This is a spare system we already have in netbox. It just needs to relocate from row D, as its being allocated into service as redundant to a server in D3. This can go in any 1G rack outside of row D (as rows A, B, and C all have analytics1 vlans available in them according to netbox.)

Wed, Oct 20, 9:19 PM · SRE, Analytics, ops-eqiad, DC-Ops
RobH updated the task description for T293938: (Need By: TBD) rack/setup/install an-test-coord1002.
Wed, Oct 20, 9:19 PM · SRE, Analytics, ops-eqiad, DC-Ops
RobH added a parent task for T293938: (Need By: TBD) rack/setup/install an-test-coord1002: Unknown Object (Task).
Wed, Oct 20, 9:16 PM · SRE, Analytics, ops-eqiad, DC-Ops
RobH closed Unknown Object (Task), a subtask of T287864: Deploy an-test-coord1002 to facilitate failover testing of analytics coordinator role, as Resolved.
Wed, Oct 20, 9:15 PM · Patch-For-Review, Data-Engineering, Data-Engineering-Kanban, Analytics-Kanban, Analytics-Clusters
RobH created T293938: (Need By: TBD) rack/setup/install an-test-coord1002.
Wed, Oct 20, 9:14 PM · SRE, Analytics, ops-eqiad, DC-Ops
RobH assigned T293934: Q2:(Need By: TBD) rack/setup/install cloudbackup100[34] to Jclark-ctr.
Wed, Oct 20, 8:58 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH added a parent task for T293934: Q2:(Need By: TBD) rack/setup/install cloudbackup100[34]: Unknown Object (Task).
Wed, Oct 20, 8:57 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH moved T293934: Q2:(Need By: TBD) rack/setup/install cloudbackup100[34] from Backlog to Racking Tasks on the ops-eqiad board.
Wed, Oct 20, 8:57 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH created T293934: Q2:(Need By: TBD) rack/setup/install cloudbackup100[34].
Wed, Oct 20, 8:57 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH renamed T293922: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet from (Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet to Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet.
Wed, Oct 20, 6:42 PM · SRE, Analytics, ops-eqiad, DC-Ops
RobH assigned T293922: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet to Jclark-ctr.
Wed, Oct 20, 6:39 PM · SRE, Analytics, ops-eqiad, DC-Ops
RobH added a parent task for T293922: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet: Unknown Object (Task).
Wed, Oct 20, 6:38 PM · SRE, Analytics, ops-eqiad, DC-Ops
RobH moved T293922: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet from Backlog to Racking Tasks on the ops-eqiad board.
Wed, Oct 20, 6:36 PM · SRE, Analytics, ops-eqiad, DC-Ops
RobH created T293922: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet.
Wed, Oct 20, 6:35 PM · SRE, Analytics, ops-eqiad, DC-Ops
RobH assigned T293909: Q2:(Need By: TBD) rack/setup/install ganeti102[5-8] to Jclark-ctr.
Wed, Oct 20, 4:19 PM · SRE, Infrastructure-Foundations, ops-eqiad, DC-Ops
RobH moved T293909: Q2:(Need By: TBD) rack/setup/install ganeti102[5-8] from Backlog to Racking Tasks on the ops-eqiad board.
Wed, Oct 20, 4:19 PM · SRE, Infrastructure-Foundations, ops-eqiad, DC-Ops
RobH created T293909: Q2:(Need By: TBD) rack/setup/install ganeti102[5-8].
Wed, Oct 20, 4:19 PM · SRE, Infrastructure-Foundations, ops-eqiad, DC-Ops

Tue, Oct 19

RobH added a comment to T293221: Agree how to document intra-DC patch panels in Netbox.

I'd like to remove using the google sheet and support any alternative that moves it wholly to netbox. I think with the circuits patch panel field entry combined with the comments field on the circuits in netbox could be enough to eliminate the gsheet. Currently I'm the only one maintaining it, and it doesn't really scale well to multiple maintainers since it has no real error checking. Having this info in netbox would get more eyes on it, as netbox is interacted with quite often in comparison (to the gsheet of xconnects). We wouldn't have to delete the old google sheet, just depreciate and no longer update it.

Tue, Oct 19, 5:38 PM · netbox, Infrastructure-Foundations

Mon, Oct 18

RobH updated the task description for T293709: Patch Telxius transport cross-connect to cr1-eqiad.
Mon, Oct 18, 10:06 PM · SRE, Infrastructure-Foundations, ops-eqiad, netops
RobH updated the task description for T293709: Patch Telxius transport cross-connect to cr1-eqiad.
Mon, Oct 18, 10:00 PM · SRE, Infrastructure-Foundations, ops-eqiad, netops
RobH added a parent task for T293709: Patch Telxius transport cross-connect to cr1-eqiad: Unknown Object (Task).
Mon, Oct 18, 10:00 PM · SRE, Infrastructure-Foundations, ops-eqiad, netops
RobH triaged T293709: Patch Telxius transport cross-connect to cr1-eqiad as Medium priority.
Mon, Oct 18, 10:00 PM · SRE, Infrastructure-Foundations, ops-eqiad, netops

Thu, Oct 14

RobH assigned T293391: Q2:(Need By: TBD) rack/setup/install cloudvirt1047.eqiad.wmnet to Jclark-ctr.
Thu, Oct 14, 4:19 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH added a parent task for T293391: Q2:(Need By: TBD) rack/setup/install cloudvirt1047.eqiad.wmnet: Unknown Object (Task).
Thu, Oct 14, 4:19 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH created T293391: Q2:(Need By: TBD) rack/setup/install cloudvirt1047.eqiad.wmnet.
Thu, Oct 14, 4:18 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops

Wed, Oct 13

RobH closed T290202: Q1:(Need By: TBD) rack/setup/install kubernetes10[18-21] as Resolved.
Wed, Oct 13, 8:30 PM · SRE, serviceops, ops-eqiad, DC-Ops
RobH added a comment to T293294: setup drmrs mgmt & private prefixs - question on switch status.

I chatted with @cmooney about this in IRC and we cannot see if there is a set pattern to which ranges are used for mgmt, private1, etc. Rather than guess incorrectly, I filed this task and assigned to Arzhel for his feedback when he is back at the keyboard.

Wed, Oct 13, 7:05 PM · Traffic, SRE, ops-drmrs, DNS, Infrastructure-Foundations, netops
RobH renamed T293294: setup drmrs mgmt & private prefixs - question on switch status from setup drmrs mgmt prefix/range to setup drmrs mgmt & private prefixs - question on switch status.
Wed, Oct 13, 7:05 PM · Traffic, SRE, ops-drmrs, DNS, Infrastructure-Foundations, netops
RobH added a parent task for T293294: setup drmrs mgmt & private prefixs - question on switch status: Unknown Object (Task).
Wed, Oct 13, 7:03 PM · Traffic, SRE, ops-drmrs, DNS, Infrastructure-Foundations, netops
RobH triaged T293294: setup drmrs mgmt & private prefixs - question on switch status as High priority.
Wed, Oct 13, 7:03 PM · Traffic, SRE, ops-drmrs, DNS, Infrastructure-Foundations, netops

Tue, Oct 12

RobH added a subtask for T292630: Eqiad Expansion - LVS Connectivity Options: Unknown Object (Task).
Tue, Oct 12, 6:49 PM · SRE, netops, Infrastructure-Foundations
RobH added a subtask for T292630: Eqiad Expansion - LVS Connectivity Options: Unknown Object (Task).
Tue, Oct 12, 6:27 PM · SRE, netops, Infrastructure-Foundations

Thu, Oct 7

RobH added a parent task for T292767: Q2:(Need By: TBD) rack/setup/install civi1002.frack.eqiad.wmnet: Unknown Object (Task).
Thu, Oct 7, 5:14 PM · SRE, fundraising-tech-ops, ops-eqiad, DC-Ops
RobH moved T292767: Q2:(Need By: TBD) rack/setup/install civi1002.frack.eqiad.wmnet from Backlog to Racking Tasks on the ops-eqiad board.
Thu, Oct 7, 5:13 PM · SRE, fundraising-tech-ops, ops-eqiad, DC-Ops
RobH created T292767: Q2:(Need By: TBD) rack/setup/install civi1002.frack.eqiad.wmnet.
Thu, Oct 7, 5:13 PM · SRE, fundraising-tech-ops, ops-eqiad, DC-Ops

Mon, Oct 4

RobH updated the task description for T290694: Q1:(Need By: TBD) rack/setup/install cp403[3-6].ulsfo.wmnet.
Mon, Oct 4, 3:06 PM · SRE, Traffic, ops-ulsfo, DC-Ops

Fri, Oct 1

RobH closed T289632: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet as Resolved.

These are now ready for use!

Fri, Oct 1, 7:48 PM · Analytics-Clusters, Data-Engineering, SRE, ops-eqiad, DC-Ops
RobH updated the task description for T289632: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet.
Fri, Oct 1, 7:47 PM · Analytics-Clusters, Data-Engineering, SRE, ops-eqiad, DC-Ops
RobH removed a project from T289632: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet: Patch-For-Review.
Fri, Oct 1, 6:10 PM · Analytics-Clusters, Data-Engineering, SRE, ops-eqiad, DC-Ops
RobH changed the status of T289632: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet from Open to In Progress.
Fri, Oct 1, 6:06 PM · Analytics-Clusters, Data-Engineering, SRE, ops-eqiad, DC-Ops
RobH updated the task description for T289632: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet.
Fri, Oct 1, 5:49 PM · Analytics-Clusters, Data-Engineering, SRE, ops-eqiad, DC-Ops
RobH added a comment to T289632: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet.

So I'm now reviewing the entire purchase history of this request.

Fri, Oct 1, 5:46 PM · Analytics-Clusters, Data-Engineering, SRE, ops-eqiad, DC-Ops

Thu, Sep 30

RobH added a comment to T290711: Clarify where to request adding a site to Google Search Console / Google News feed and who to handle.

According to https://wikitech.wikimedia.org/wiki/Google_Search_Console_access this is handled by SRE.

@RobH Is this you?

Thu, Sep 30, 11:02 PM · WMF-General-or-Unknown, Documentation
RobH moved T290694: Q1:(Need By: TBD) rack/setup/install cp403[3-6].ulsfo.wmnet from Backlog to Racking Tasks on the ops-ulsfo board.
Thu, Sep 30, 6:12 PM · SRE, Traffic, ops-ulsfo, DC-Ops

Wed, Sep 29

RobH moved T292095: Q2:(Need By: TBD) Rows E/F network racking task from Backlog to Racking Tasks on the ops-eqiad board.
Wed, Sep 29, 5:04 PM · SRE, Infrastructure-Foundations, netops, ops-eqiad, DC-Ops
RobH added a parent task for T292095: Q2:(Need By: TBD) Rows E/F network racking task: Unknown Object (Task).
Wed, Sep 29, 5:04 PM · SRE, Infrastructure-Foundations, netops, ops-eqiad, DC-Ops
RobH updated the task description for T292095: Q2:(Need By: TBD) Rows E/F network racking task.
Wed, Sep 29, 5:03 PM · SRE, Infrastructure-Foundations, netops, ops-eqiad, DC-Ops
RobH created T292095: Q2:(Need By: TBD) Rows E/F network racking task.
Wed, Sep 29, 5:03 PM · SRE, Infrastructure-Foundations, netops, ops-eqiad, DC-Ops

Sep 22 2021

RobH added a parent task for T291579: Q1:(Need By: TBD) rack/setup/install ml-train100[1-4]: Unknown Object (Task).
Sep 22 2021, 4:26 PM · SRE, Machine-Learning-Team, ops-eqiad, DC-Ops
RobH moved T291579: Q1:(Need By: TBD) rack/setup/install ml-train100[1-4] from Backlog to Racking Tasks on the ops-eqiad board.
Sep 22 2021, 4:25 PM · SRE, Machine-Learning-Team, ops-eqiad, DC-Ops
RobH created T291579: Q1:(Need By: TBD) rack/setup/install ml-train100[1-4].
Sep 22 2021, 4:25 PM · SRE, Machine-Learning-Team, ops-eqiad, DC-Ops

Sep 17 2021

RobH added a comment to T290318: hw troubleshooting: megaraid reset due to fatal error for labstore1005.eqiad.wmnet.
robh@labstore1005:~$ sudo journalctl -S "2021-09-15" | grep "Controller encountered a fatal error and was reset" | cut -d: -f 1 | sort | uniq -c
robh@labstore1005:~$ sudo journalctl -S "2021-09-16" | grep "Controller encountered a fatal error and was reset" | cut -d: -f 1 | sort | uniq -c
robh@labstore1005:~$ sudo journalctl -S "2021-09-17" | grep "Controller encountered a fatal error and was reset" | cut -d: -f 1 | sort | uniq -c
robh@labstore1005:~$
Sep 17 2021, 4:42 PM · SRE, ops-eqiad, DC-Ops

Sep 16 2021

RobH closed T289715: (Need By: TBD) rack/setup/install ganeti4004 as Resolved.
Sep 16 2021, 9:34 PM · SRE, ops-ulsfo, Traffic, DC-Ops
RobH closed T289715: (Need By: TBD) rack/setup/install ganeti4004, a subtask of T288579: decommission bast4002.wikimedia.org, as Resolved.
Sep 16 2021, 9:34 PM · DC-Ops, ops-ulsfo, SRE, Traffic, decommission-hardware
RobH removed a project from T289715: (Need By: TBD) rack/setup/install ganeti4004: Patch-For-Review.
Sep 16 2021, 8:52 PM · SRE, ops-ulsfo, Traffic, DC-Ops
RobH changed the status of T289715: (Need By: TBD) rack/setup/install ganeti4004 from Open to In Progress.
Sep 16 2021, 6:53 PM · SRE, ops-ulsfo, Traffic, DC-Ops
RobH changed the status of T289715: (Need By: TBD) rack/setup/install ganeti4004, a subtask of T288579: decommission bast4002.wikimedia.org, from Open to In Progress.
Sep 16 2021, 6:53 PM · DC-Ops, ops-ulsfo, SRE, Traffic, decommission-hardware
RobH updated the task description for T290318: hw troubleshooting: megaraid reset due to fatal error for labstore1005.eqiad.wmnet.
Sep 16 2021, 5:31 PM · SRE, ops-eqiad, DC-Ops
RobH changed the status of T290318: hw troubleshooting: megaraid reset due to fatal error for labstore1005.eqiad.wmnet from In Progress to Stalled.

So far so good:

[bstorm@labstore1005]:~ $ sudo journalctl -S "2021-09-15" | grep "Controller encountered a fatal error and was reset" | cut -d: -f 1 | sort | uniq -c
[bstorm@labstore1005]:~ $
Sep 16 2021, 5:29 PM · SRE, ops-eqiad, DC-Ops

Sep 15 2021

RobH reassigned T290318: hw troubleshooting: megaraid reset due to fatal error for labstore1005.eqiad.wmnet from RobH to Bstorm.

I incremented the idrac from 2.52 to 2.81, bypassing the known bad version 2.61. Seems 2.81 also has an https issue (different than 2.61) so John had to connect a crash cart and rollback the firmware via the lifecycle controller.

Sep 15 2021, 8:55 PM · SRE, ops-eqiad, DC-Ops
RobH added a comment to T290318: hw troubleshooting: megaraid reset due to fatal error for labstore1005.eqiad.wmnet.

tried to cut the difference and go half as old on firmware, didnt work (package verify failed) so started just incrementing one version at a time.

Sep 15 2021, 7:42 PM · SRE, ops-eqiad, DC-Ops
RobH added a comment to T290318: hw troubleshooting: megaraid reset due to fatal error for labstore1005.eqiad.wmnet.

attempted flash of bios and raid bios, failure to due unable to verify package - upload attempted twice from the same set of downloaded files
reattempted flash of bios and raid bios with new downloads and uploads - failure to due unable to verify package - upload attempted twice from the same set of newly downloaded files
attempted flash of idrac to resolve package verification issues, idrac https locked up and required racreset command via ssh
attempted flash of idrac from 2.30 to 2.81, failed due ot package verification. we're going to have to step this up manually by cutting the versions down one at a time.

Sep 15 2021, 7:12 PM · SRE, ops-eqiad, DC-Ops
RobH added a comment to T290318: hw troubleshooting: megaraid reset due to fatal error for labstore1005.eqiad.wmnet.

Attempting to flash the raid firmware results in failure to verify package contents on the firmware file. The firmware file is fine (re-downloaded and checked the checksum locally before upload twice, upload attempt 2 times per file download). It seems the idrac is just ancient, so it has to be updated first.

Sep 15 2021, 6:52 PM · SRE, ops-eqiad, DC-Ops
RobH changed the status of T290318: hw troubleshooting: megaraid reset due to fatal error for labstore1005.eqiad.wmnet from Open to In Progress.
Sep 15 2021, 6:36 PM · SRE, ops-eqiad, DC-Ops

Sep 14 2021

RobH moved T290987: Q1:(Need By: TBD) rack/setup/install an-presto10[06-15] from Backlog to Racking Tasks on the ops-eqiad board.
Sep 14 2021, 4:45 PM · Analytics-Radar, SRE, Data-Engineering, ops-eqiad, DC-Ops
RobH added a parent task for T290987: Q1:(Need By: TBD) rack/setup/install an-presto10[06-15]: Unknown Object (Task).
Sep 14 2021, 4:44 PM · Analytics-Radar, SRE, Data-Engineering, ops-eqiad, DC-Ops
RobH created T290987: Q1:(Need By: TBD) rack/setup/install an-presto10[06-15].
Sep 14 2021, 4:44 PM · Analytics-Radar, SRE, Data-Engineering, ops-eqiad, DC-Ops

Sep 13 2021

RobH updated the task description for T290899: Q1: eqiad: (32) PDUs for expansion.
Sep 13 2021, 6:27 PM · SRE, ops-eqiad, DC-Ops
RobH moved T290899: Q1: eqiad: (32) PDUs for expansion from Backlog to Racking Tasks on the ops-eqiad board.
Sep 13 2021, 5:30 PM · SRE, ops-eqiad, DC-Ops
RobH added a parent task for T290899: Q1: eqiad: (32) PDUs for expansion: Unknown Object (Task).
Sep 13 2021, 5:29 PM · SRE, ops-eqiad, DC-Ops
RobH created T290899: Q1: eqiad: (32) PDUs for expansion.
Sep 13 2021, 5:29 PM · SRE, ops-eqiad, DC-Ops
RobH renamed T290894: Q2: (Need By: TBD) rack/setup/install kubestage100[34].eqiad.wmnet from Q1: (Need By: TBD) rack/setup/install kubestage100[34].eqiad.wmnet to Q2: (Need By: TBD) rack/setup/install kubestage100[34].eqiad.wmnet.
Sep 13 2021, 4:41 PM · SRE, serviceops, ops-eqiad, DC-Ops
RobH added a parent task for T290894: Q2: (Need By: TBD) rack/setup/install kubestage100[34].eqiad.wmnet: Unknown Object (Task).
Sep 13 2021, 4:39 PM · SRE, serviceops, ops-eqiad, DC-Ops
RobH moved T290894: Q2: (Need By: TBD) rack/setup/install kubestage100[34].eqiad.wmnet from Backlog to Racking Tasks on the ops-eqiad board.
Sep 13 2021, 4:38 PM · SRE, serviceops, ops-eqiad, DC-Ops
RobH created T290894: Q2: (Need By: TBD) rack/setup/install kubestage100[34].eqiad.wmnet.
Sep 13 2021, 4:38 PM · SRE, serviceops, ops-eqiad, DC-Ops

Sep 9 2021

RobH added a comment to T290318: hw troubleshooting: megaraid reset due to fatal error for labstore1005.eqiad.wmnet.

Update from sub-task and IRC discussions:

Sep 9 2021, 10:53 PM · SRE, ops-eqiad, DC-Ops
RobH added a parent task for T290694: Q1:(Need By: TBD) rack/setup/install cp403[3-6].ulsfo.wmnet: Unknown Object (Task).
Sep 9 2021, 9:16 PM · SRE, Traffic, ops-ulsfo, DC-Ops
RobH created T290694: Q1:(Need By: TBD) rack/setup/install cp403[3-6].ulsfo.wmnet.
Sep 9 2021, 9:16 PM · SRE, Traffic, ops-ulsfo, DC-Ops