Page MenuHomePhabricator

RobH (Rob Halsell)
Operations EngineerAdministrator

Projects (20)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Nov 24 2014, 1:43 PM (332 w, 6 d)
Roles
Administrator
Availability
Available
IRC Nick
RobH
LDAP User
RobH
MediaWiki User
RobH [ Global Accounts ]

My GPG Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7 0245 D22A

I am an Operations Engineer on Wikimedia's Datacenter Operations Team.

I also am the primary triage engineer for the hardware-requests project, as well as the private S4 procurement space and procurement project.

All questions involving allocation of hardware can be initially addressed on https://wikitech.wikimedia.org/wiki/Operations_requests.

Please note that private message via phabricator is not my preferred contact means. Please feel free to contact me (robh) directly via irc/freenode, or email my @wikimedia.org email address.

Recent Activity

Thu, Apr 8

RobH added a comment to T275081: (Need By: TBD) rack/setup/install cloudvirt104[0-6].eqiad.wmnet.

I've emailed our Dell rep to determine where the NIC is for the seed server, cloudvirt1040. Once I have that info, I'll reassign this back to John for followup and installation of the NIC.

Thu, Apr 8, 10:21 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH updated the task description for T275081: (Need By: TBD) rack/setup/install cloudvirt104[0-6].eqiad.wmnet.
Thu, Apr 8, 10:21 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH updated the task description for T279721: Dc-Ops Commands for Cumin.
Thu, Apr 8, 9:54 PM · procurement, ops-codfw, ops-eqiad, DC-Ops, SRE
RobH updated the task description for T279721: Dc-Ops Commands for Cumin.
Thu, Apr 8, 9:54 PM · procurement, ops-codfw, ops-eqiad, DC-Ops, SRE

Wed, Apr 7

Dzahn awarded T274171: (Need By: TBD) rack/setup/install (35) mw2377 and upwards a Mountain of Wealth token.
Wed, Apr 7, 7:38 PM · SRE, ops-codfw, DC-Ops

Tue, Apr 6

RobH added a comment to T279244: CAS SSO for reedy.

I think racktables is replaced by netbox for Reedy's needs and he does have access to that. This ticket is down to "redirect racktables to netbox" or "add a banner telling users it's outdated" afaict.

Tue, Apr 6, 6:32 PM · CAS-SSO, SRE, LDAP-Access-Requests

Mon, Apr 5

RobH added a comment to T279244: CAS SSO for reedy.

To be clear, I don't do ANY of the overhead technical debt to maintain Racktables in a functional state, so I also have no strong viewpoints on if we should roll things back to enable Sam to access it. This is work that would likely fall to @MoritzMuehlenhoff, so I defer entirely to his judgement!

Mon, Apr 5, 4:48 PM · CAS-SSO, SRE, LDAP-Access-Requests

Fri, Apr 2

RobH updated the task description for T275511: (Need By: TBD) rack/setup/install moss-fe100[12].eqiad.wmnet.
Fri, Apr 2, 10:10 PM · SRE, ops-eqiad, DC-Ops
RobH removed a project from T275511: (Need By: TBD) rack/setup/install moss-fe100[12].eqiad.wmnet: Patch-For-Review.
Fri, Apr 2, 9:20 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T275511: (Need By: TBD) rack/setup/install moss-fe100[12].eqiad.wmnet.
Fri, Apr 2, 9:08 PM · SRE, ops-eqiad, DC-Ops

Thu, Apr 1

RobH updated the task description for T275081: (Need By: TBD) rack/setup/install cloudvirt104[0-6].eqiad.wmnet.
Thu, Apr 1, 9:35 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH added a comment to T275081: (Need By: TBD) rack/setup/install cloudvirt104[0-6].eqiad.wmnet.

I've manually added port descriptions and the primary interface to vlan-cloud-hosts1-eqiad.

Thu, Apr 1, 9:10 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH removed a project from T275081: (Need By: TBD) rack/setup/install cloudvirt104[0-6].eqiad.wmnet: Patch-For-Review.
Thu, Apr 1, 8:53 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH added a comment to T275081: (Need By: TBD) rack/setup/install cloudvirt104[0-6].eqiad.wmnet.

So the entire lot failed to hit the dhcp server, the network has NOT been setup on cloudsw1-d5-eqiad for any of these hosts.

Thu, Apr 1, 8:29 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH updated the task description for T275081: (Need By: TBD) rack/setup/install cloudvirt104[0-6].eqiad.wmnet.
Thu, Apr 1, 8:15 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH added a comment to T275081: (Need By: TBD) rack/setup/install cloudvirt104[0-6].eqiad.wmnet.

When I went to go install these, it seems that cloudvirt1040 is the one iwth the Intel nics, which means it was the seed server. The racking checklist was not followed, as cloudvirt1046 should have been the seed server. Hopefully the items were not received incorrectly in Coupa, as the packing slip service tags for the 6 purchased hosts need to have been inputted into coupa for the normal PO.

Thu, Apr 1, 7:27 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH added a comment to T276399: migrate services from bast1002 to bast1003.

Please note bast1003 is now staged in netbox and is ready for new roles and migration as needed!

Thu, Apr 1, 6:03 PM · Patch-For-Review, SRE
RobH removed a project from T276396: (Need By: TBD) rack/setup/install bast1003.wikimedia.org: Patch-For-Review.
Thu, Apr 1, 6:03 PM · SRE, ops-eqiad, DC-Ops
RobH closed T276396: (Need By: TBD) rack/setup/install bast1003.wikimedia.org as Resolved.
Thu, Apr 1, 6:03 PM · SRE, ops-eqiad, DC-Ops

Tue, Mar 30

RobH updated the task description for T278182: (Need By: TBD) rack/setup/install cp501[3-6].
Tue, Mar 30, 6:23 PM · SRE, ops-eqsin, DC-Ops
RobH added a parent task for T278182: (Need By: TBD) rack/setup/install cp501[3-6]: Unknown Object (Task).
Tue, Mar 30, 6:13 PM · SRE, ops-eqsin, DC-Ops
RobH reassigned T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet from RobH to Cmjohnson.

Would you mind if I leave the ticket open until the following are addressed in netbox?

  • Information on secondary interface. Which one is connected? I need this information for service implementation.
Tue, Mar 30, 2:38 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops

Fri, Mar 26

RobH closed T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet as Resolved.
Fri, Mar 26, 8:38 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH added a comment to T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet.

cloudgw100[12] both installed and set to staged. The netbox info for the secondary interface is missing, but is present on the switch, so an on-site will need to fix I suppose?

Fri, Mar 26, 8:38 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH added a comment to T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet.

@Cmjohnson there are outstanding changes in Homer for cloudsw1-c8
Also something is not right in Netbox, https://netbox.wikimedia.org/dcim/interfaces/20087/ is type virtual, and not connected to the device interface.

Fri, Mar 26, 7:48 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops

Tue, Mar 23

RobH updated subscribers of T278273: apply new hostname label for pki-root1001 (was auth1002).
Tue, Mar 23, 11:43 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T278273: apply new hostname label for pki-root1001 (was auth1002).
Tue, Mar 23, 11:35 PM · SRE, ops-eqiad, DC-Ops
RobH triaged T278273: apply new hostname label for pki-root1001 (was auth1002) as Low priority.
Tue, Mar 23, 11:34 PM · SRE, ops-eqiad, DC-Ops
RobH closed T276625: (Need By: TBD) rack/setup/install pki-root1001.eqiad.wmnet as Resolved.
Tue, Mar 23, 11:33 PM · SRE, ops-eqiad, DC-Ops
RobH added a comment to T276625: (Need By: TBD) rack/setup/install pki-root1001.eqiad.wmnet.

@jbond: be aware pki-root1001 is now staged for your use.

Tue, Mar 23, 11:32 PM · SRE, ops-eqiad, DC-Ops
RobH moved T278273: apply new hostname label for pki-root1001 (was auth1002) from Backlog to Lower Priority Items on the ops-eqiad board.
Tue, Mar 23, 11:31 PM · SRE, ops-eqiad, DC-Ops
RobH created T278273: apply new hostname label for pki-root1001 (was auth1002).
Tue, Mar 23, 11:31 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T276625: (Need By: TBD) rack/setup/install pki-root1001.eqiad.wmnet.
Tue, Mar 23, 11:30 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T276625: (Need By: TBD) rack/setup/install pki-root1001.eqiad.wmnet.
Tue, Mar 23, 8:10 PM · SRE, ops-eqiad, DC-Ops
RobH moved T278250: (Need By: TBD) install second SSD into payments100[5-8] from Triage to Blocked on the fundraising-tech-ops board.
Tue, Mar 23, 6:18 PM · SRE, ops-eqiad, fundraising-tech-ops, DC-Ops
RobH moved T278250: (Need By: TBD) install second SSD into payments100[5-8] from Backlog to Racking Tasks on the ops-eqiad board.
Tue, Mar 23, 6:18 PM · SRE, ops-eqiad, fundraising-tech-ops, DC-Ops
RobH added a parent task for T278250: (Need By: TBD) install second SSD into payments100[5-8]: Unknown Object (Task).
Tue, Mar 23, 6:18 PM · SRE, ops-eqiad, fundraising-tech-ops, DC-Ops
RobH created T278250: (Need By: TBD) install second SSD into payments100[5-8].
Tue, Mar 23, 6:18 PM · SRE, ops-eqiad, fundraising-tech-ops, DC-Ops
RobH reassigned T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet from RobH to Cmjohnson.

It appears this is plugged into the non cloud switch (after IRC sync). I chatted with Chris, who can fix this tomorrow and kick it back to me for imaging.

Tue, Mar 23, 12:06 AM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH moved T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet from Backlog to Racking / Decom on the cloud-services-team (Hardware) board.
Tue, Mar 23, 12:06 AM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH updated the task description for T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet.
Tue, Mar 23, 12:04 AM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops

Mon, Mar 22

RobH removed a project from T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet: Patch-For-Review.
Mon, Mar 22, 11:22 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH moved T278182: (Need By: TBD) rack/setup/install cp501[3-6] from Backlog to Racking Tasks on the ops-eqsin board.
Mon, Mar 22, 8:51 PM · SRE, ops-eqsin, DC-Ops
RobH added a parent task for T278182: (Need By: TBD) rack/setup/install cp501[3-6]: Unknown Object (Task).
Mon, Mar 22, 8:43 PM · SRE, ops-eqsin, DC-Ops
RobH created T278182: (Need By: TBD) rack/setup/install cp501[3-6].
Mon, Mar 22, 8:43 PM · SRE, ops-eqsin, DC-Ops

Fri, Mar 19

RobH claimed T276625: (Need By: TBD) rack/setup/install pki-root1001.eqiad.wmnet.
Fri, Mar 19, 8:49 PM · SRE, ops-eqiad, DC-Ops
RobH reassigned T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet from RobH to aborrero.
Fri, Mar 19, 2:47 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH added a comment to T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet.

Perhaps this wasn't conveyed at the time of order, and it may cause issues, but we don't support connecting mixed modes 1G and 10G within the same rack, as then it is taking up 3 ports on a 10G switch.

Fri, Mar 19, 2:47 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops

Thu, Mar 18

RobH reassigned T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet from RobH to aborrero.

The networking requirements for this request are wholly unclear at the time of this comment. The racking info currently has:

Thu, Mar 18, 8:31 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH added a comment to T272403: (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet.

@Cmjohnson
cloudgw1001 c8 u29 ports13/19 cableid #5322
cloudgw1002. d5 u21 ports 23/35 cableid #5320.

Thu, Mar 18, 8:27 PM · SRE, cloud-services-team (Hardware), ops-eqiad, DC-Ops
RobH closed T273566: (Need By: TBD) rack/setup/install db11[76-84] as Resolved.
Thu, Mar 18, 3:29 PM · Patch-For-Review, DBA, SRE, ops-eqiad, DC-Ops

Wed, Mar 17

RobH reassigned T274925: (Need By: TBD) rack/setup/install mc10[37-54].eqiad.wmnet from RobH to Jclark-ctr.

serviceops please be aware mc1037 and mc1038 are ready for your team to place into service. The rest are now assigned back to @Jclark-ctr for racking.

Wed, Mar 17, 11:05 PM · User-jijiki, SRE, serviceops, ops-eqiad, DC-Ops
RobH updated the task description for T274925: (Need By: TBD) rack/setup/install mc10[37-54].eqiad.wmnet.
Wed, Mar 17, 11:03 PM · User-jijiki, SRE, serviceops, ops-eqiad, DC-Ops
RobH updated the task description for T273566: (Need By: TBD) rack/setup/install db11[76-84].
Wed, Mar 17, 9:31 PM · Patch-For-Review, DBA, SRE, ops-eqiad, DC-Ops
RobH updated the task description for T273566: (Need By: TBD) rack/setup/install db11[76-84].
Wed, Mar 17, 8:17 PM · Patch-For-Review, DBA, SRE, ops-eqiad, DC-Ops
RobH added a comment to T273566: (Need By: TBD) rack/setup/install db11[76-84].

bios, raid, and idrac firmware updated on all hosts for this task.

Wed, Mar 17, 7:21 PM · Patch-For-Review, DBA, SRE, ops-eqiad, DC-Ops
RobH added a comment to T273566: (Need By: TBD) rack/setup/install db11[76-84].

So I just checked and confirmed with Chris that when the add server interface script was run for each of these, the skip ivp6 checkbox was checked, but they all seem to have IPv6 allocated in netbox (so do active db hosts!)

Wed, Mar 17, 7:21 PM · Patch-For-Review, DBA, SRE, ops-eqiad, DC-Ops

Mar 12 2021

RobH moved T277341: (Need By: TBD) install MPC7E-MRATE FPC into cr[12]-codfw from Backlog to Racking Tasks on the ops-codfw board.
Mar 12 2021, 8:26 PM · SRE, ops-codfw, DC-Ops
RobH added a parent task for T277341: (Need By: TBD) install MPC7E-MRATE FPC into cr[12]-codfw: Unknown Object (Task).
Mar 12 2021, 8:26 PM · SRE, ops-codfw, DC-Ops
RobH created T277341: (Need By: TBD) install MPC7E-MRATE FPC into cr[12]-codfw.
Mar 12 2021, 8:26 PM · SRE, ops-codfw, DC-Ops
RobH moved T277340: (Need By: TBD) rack/setup/install (2) new 10G switches from Backlog to Racking Tasks on the ops-eqiad board.
Mar 12 2021, 8:22 PM · SRE, ops-eqiad, DC-Ops
RobH moved T277339: (Need By: TBD) rack/setup/install new linecards into routers from Backlog to Racking Tasks on the ops-eqiad board.
Mar 12 2021, 8:22 PM · SRE, ops-eqiad, DC-Ops
RobH added a parent task for T277340: (Need By: TBD) rack/setup/install (2) new 10G switches: Unknown Object (Task).
Mar 12 2021, 8:16 PM · SRE, ops-eqiad, DC-Ops
RobH created T277340: (Need By: TBD) rack/setup/install (2) new 10G switches.
Mar 12 2021, 8:16 PM · SRE, ops-eqiad, DC-Ops
RobH added a parent task for T277339: (Need By: TBD) rack/setup/install new linecards into routers: Unknown Object (Task).
Mar 12 2021, 8:14 PM · SRE, ops-eqiad, DC-Ops
RobH created T277339: (Need By: TBD) rack/setup/install new linecards into routers.
Mar 12 2021, 8:14 PM · SRE, ops-eqiad, DC-Ops
RobH added a parent task for T277327: (Need By: 2021-04-30) rack/setup/install backup100[4-7]: Unknown Object (Task).
Mar 12 2021, 6:00 PM · SRE, ops-eqiad, DC-Ops
RobH moved T277327: (Need By: 2021-04-30) rack/setup/install backup100[4-7] from Backlog to Racking Tasks on the ops-eqiad board.
Mar 12 2021, 5:59 PM · SRE, ops-eqiad, DC-Ops
RobH created T277327: (Need By: 2021-04-30) rack/setup/install backup100[4-7].
Mar 12 2021, 5:59 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T277323: (Need By: 2021-04-30) rack/setup/install backup200[4-7].
Mar 12 2021, 5:58 PM · SRE, ops-codfw, DC-Ops
RobH added a parent task for T277323: (Need By: 2021-04-30) rack/setup/install backup200[4-7]: Unknown Object (Task).
Mar 12 2021, 5:51 PM · SRE, ops-codfw, DC-Ops
RobH renamed T277323: (Need By: 2021-04-30) rack/setup/install backup200[4-7] from (Need By: TBD) rack/setup/install backup200[4-7] to (Need By: 2021-04-30) rack/setup/install backup200[4-7].
Mar 12 2021, 5:49 PM · SRE, ops-codfw, DC-Ops
RobH assigned T277323: (Need By: 2021-04-30) rack/setup/install backup200[4-7] to Papaul.
Mar 12 2021, 5:49 PM · SRE, ops-codfw, DC-Ops
RobH created T277323: (Need By: 2021-04-30) rack/setup/install backup200[4-7].
Mar 12 2021, 5:48 PM · SRE, ops-codfw, DC-Ops

Mar 11 2021

RobH updated subscribers of T274184: (Need By: 2021-03-31) rack/setup/install backup1003.
Mar 11 2021, 9:40 PM · SRE, ops-eqiad, DC-Ops
RobH closed T274184: (Need By: 2021-03-31) rack/setup/install backup1003 as Resolved.

@jcrespo this is all setup, resolving task but pinging you in so you're aware this is ready for your use!

Mar 11 2021, 9:40 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T274184: (Need By: 2021-03-31) rack/setup/install backup1003.
Mar 11 2021, 9:39 PM · SRE, ops-eqiad, DC-Ops
RobH updated subscribers of T274184: (Need By: 2021-03-31) rack/setup/install backup1003.
Mar 11 2021, 9:39 PM · SRE, ops-eqiad, DC-Ops
RobH added a comment to T274184: (Need By: 2021-03-31) rack/setup/install backup1003.

Homer now fails with an error, pinged Arzhel about it as I'm not sure best way to proceed.

Mar 11 2021, 7:48 PM · SRE, ops-eqiad, DC-Ops
RobH added a comment to T274184: (Need By: 2021-03-31) rack/setup/install backup1003.

Arzhel caught this in IRC, it was setup for asw2-b1-eqiad:xe-2/0/33 when it should have been asw2-b2-eqiad:xe-2/0/33. I've removed the interface, reran the script, now rerunning dns (likely noop) and homer.

Mar 11 2021, 7:42 PM · SRE, ops-eqiad, DC-Ops
RobH claimed T274184: (Need By: 2021-03-31) rack/setup/install backup1003.
Mar 11 2021, 7:38 PM · SRE, ops-eqiad, DC-Ops
RobH reassigned T274184: (Need By: 2021-03-31) rack/setup/install backup1003 from RobH to Volans.

I've setup the hw raid and such to identically match backup2003 (the mirror of this in codfw which is already imaged successfully), and updated puppet repo to match.

Mar 11 2021, 7:33 PM · SRE, ops-eqiad, DC-Ops
RobH removed a project from T274184: (Need By: 2021-03-31) rack/setup/install backup1003: Patch-For-Review.
Mar 11 2021, 6:55 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T274184: (Need By: 2021-03-31) rack/setup/install backup1003.
Mar 11 2021, 6:51 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T274184: (Need By: 2021-03-31) rack/setup/install backup1003.
Mar 11 2021, 6:46 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T274184: (Need By: 2021-03-31) rack/setup/install backup1003.
Mar 11 2021, 6:03 PM · SRE, ops-eqiad, DC-Ops
RobH closed T274163: (Need By: TBD) rack/setup/install an-druid100[345] as Resolved.
Mar 11 2021, 4:00 PM · SRE, ops-eqiad, DC-Ops

Mar 10 2021

RobH closed T273778: (Need By: TBD) rack/setup/install kafka-logging100[123] as Resolved.
Mar 10 2021, 7:51 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T273778: (Need By: TBD) rack/setup/install kafka-logging100[123].
Mar 10 2021, 7:27 PM · SRE, ops-eqiad, DC-Ops
RobH removed a project from T273778: (Need By: TBD) rack/setup/install kafka-logging100[123]: Patch-For-Review.
Mar 10 2021, 5:01 PM · SRE, ops-eqiad, DC-Ops
RobH moved T274905: (Need By: TBD) rack/setup/install kafka-logging200[123].codfw.wmnet from Backlog to Racking Tasks on the ops-codfw board.
Mar 10 2021, 3:28 PM · SRE, ops-codfw, DC-Ops
RobH added a project to T274905: (Need By: TBD) rack/setup/install kafka-logging200[123].codfw.wmnet: ops-codfw.
Mar 10 2021, 3:27 PM · SRE, ops-codfw, DC-Ops
RobH closed T274206: (Need By: 2021-03-31) rack/setup/install ms-backup100[12] as Resolved.
Mar 10 2021, 12:17 AM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T274206: (Need By: 2021-03-31) rack/setup/install ms-backup100[12].
Mar 10 2021, 12:16 AM · SRE, ops-eqiad, DC-Ops

Mar 9 2021

RobH removed a project from T273778: (Need By: TBD) rack/setup/install kafka-logging100[123]: Patch-For-Review.
Mar 9 2021, 11:48 PM · SRE, ops-eqiad, DC-Ops
RobH removed a project from T274206: (Need By: 2021-03-31) rack/setup/install ms-backup100[12]: Patch-For-Review.
Mar 9 2021, 11:25 PM · SRE, ops-eqiad, DC-Ops
RobH updated the task description for T273778: (Need By: TBD) rack/setup/install kafka-logging100[123].
Mar 9 2021, 11:18 PM · SRE, ops-eqiad, DC-Ops
RobH added a comment to T273778: (Need By: TBD) rack/setup/install kafka-logging100[123].

I'm having partman issues since this needs to use a new recipe for hw raid 10 flat filesystem. flat.cfg didn't work, so went custom but still have grub issues...

Mar 9 2021, 9:22 PM · SRE, ops-eqiad, DC-Ops
RobH removed a project from T273778: (Need By: TBD) rack/setup/install kafka-logging100[123]: Patch-For-Review.
Mar 9 2021, 8:36 PM · SRE, ops-eqiad, DC-Ops
RobH reassigned T276922: cloudvirt1038: PCIe error from RobH to Cmjohnson.

Updated firmwares, but error persists.

Mar 9 2021, 8:15 PM · SRE, ops-eqiad, DC-Ops, cloud-services-team (Hardware)