dumpsdata1007 is now running 5.16.11, can you please retest?

I'm not familiar with perccli myself, if there's any uncertainties with the docs/setup let's clarify with Dell?

@RobH - Sorry to hijack this thread. Do you happen to know whether the PERC H750 will support JBOD mode?
In the past we've had to use single-drive RAID0 volumes to get the same effect. It works, but it's a bit of extra overhead, so JBOD support could bring a substantial benefit.
Thanks.

Raid testing.

I can poll the controller for basic info: root@dumpsdata1007:~# perccli64 /c0/dall show
and get BBU info: perccli64 /c0/bbu show all
perccli64 /c0/d0 show

I don't get how to poll for the virtual disk IDs in the references, it assumes you know them?

root@dumpsdata1007:~# perccli64 /c0/dall show
CLI Version = 007.0529.0000.0000 Sep 18, 2018
Operating system = Linux 5.16.0-0.bpo.3-amd64
Controller = 0
Status = Success
Description = Show Diskgroup Succeeded


TOPOLOGY :
========

-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR 
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID1 Optl  N  446.625 GB dflt N  N   dflt N      N  
 0 0   -   -        -   RAID1 Optl  N  446.625 GB dflt N  N   dflt N      N  
 0 0   0   64:12    0   DRIVE Onln  N  446.625 GB dflt N  N   dflt -      N  
 0 0   1   64:13    4   DRIVE Onln  N  446.625 GB dflt N  N   dflt -      N  
-----------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
TR=Transport Ready




root@dumpsdata1007:~# perccli64 /c0/v0 show
CLI Version = 007.0529.0000.0000 Sep 18, 2018
Operating system = Linux 5.16.0-0.bpo.3-amd64
Controller = 0
Status = Failure
Description = None

Detailed Status :
===============

----------------------------------
VD Status ErrCd ErrMsg            
----------------------------------
 0 Failed   255 Invalid Vd number 
----------------------------------



root@dumpsdata1007:~#

I'm unable to get the disk to go into missing to spin down, spin back up, and set to returned to test rebuilding an array. I can set it to offline, and thats about it.

Also unable to determine how to poll for virtual disk IDs, other htan dropping into raid bios, which won't work out for production. I need to keep hammering away at this, but could use some help if anyone is familiar at all with perccli64.

In T297913#7805533, @RobH wrote:

Also unable to determine how to poll for virtual disk IDs, other htan dropping into raid bios, which won't work out for production. I need to keep hammering away at this, but could use some help if anyone is familiar at all with perccli64.

Can't we reach out to Dell? I mean, they are the ones who discontinued our existing/perfectly fine working controller and they are the ones who want to sell us something...

@RobH I took a quick look at this yesterday, no luck. Since it is a new product i will recommend getting Dell help maybe this will save us time.

Emailed our Dell team with our issues, will update as they respond.

Dell Team,

We're currently attempting to get the new raid controllers for function for us so we can unblock and order a few dozen (or more) servers using it. However, I'm running into issues attempting to set a drive to missing to spin it down, remove it, and replace it in a simulated disk failure.

Before we push this controller into production, we have to be able to successfully do the following:

• set a disk to offline, missing, spin it down, remove it, add it back in, spin it back up, set online, add back into the array, rebuild the array
• create a raid[1|6|10] array via command line

I'm stuck on the setting it to missing onward for a 'failed disk' replacement test.

I've attached a text file of my output for info on the host and the working and failed commands. Can we get this escalated so I can have someone work with me to figure out what I'm doing wrong? I'm hoping with the info gathering commands in the text file, you'll be able to see what I'm doing wrong in command syntax and offer the correct command line examples.

Thanks in advance!

@MoritzMuehlenhoff.

Dell is stating they want us to upgrade to the newest revision of the utility for them to offer support. I'm pushing back, but they state we should be using:

Version 7.1623.0.0, A11, Release date:9/10/2021.
LINUX PERCCLI Utility For All Dell HBA/PERC Controllers: https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=j91yg
Release notes: https://dl.dell.com/FOLDER07575976M/1/PERCCLI_7.1623.00_A11_Release_Notes.txt

Can we get this put on dumpsdata1007 for testing? (I don't want to simply load it up without any kind of secondary security review.)

This was updated, same issue on dumpsdata1007 and sent info to our Dell team.

Dell suggestion some alternate arguments for the command line utility that didn't work, and then requested we open a case for them to escalate

Service Request 1090168698

Sent case # to our team to escalate.

Summary update from multiple out of band support emails and conversations with our Dell account team:

• confirmed that the missing/spin down doesn't work for them either, and chipset manufacturer is now involved to determine when and why the feature was disabled/removed
• dell is aware of our test workflow and our need to do the following:
  • use perccli to add/remove/swap failed disks in an array
  • use perccli to setup new raid arrays and edit existing in raid[1|6|10]
  • use perccli commands to monitor raid health

They'll attempt to get an update to us ASAP.

Current plan is to escalate all pending H750 orders by end of this month.

perccli64 /c0 show all also shows a physical disk list, we'll want to run to confirm it sees the disk gone when removed.

Todo:

• test on other distros we use
• get partman to work with this, as our existing recipes expect the flexbays to be the SDA virtual drive and the new controller always puts them at a higher ID # so they show up after the HDDs.
• get monitoring updated to leverage perccli64 to use for icinga monitor of raid health

That's promising progress! I have rebooted dumpsdata1007 back into Linux 5.10. This is the standard kernel we're running on Bullseye (we can ignore Buster/Stretch for testing the PERC 750 controller). Can you please re-test the RAID setup and failure scenario?

The "get monitoring updated to leverage perccli64 to use for icinga monitor of raid health" can be handled by SRE IF, simply create a separate Phab task for it once we've established that PERC 750 works for us.

I am seeing the procurement tasks being processed already, does that mean we have established that this controller will work 100% for us then?

In T297913#7890055, @Marostegui wrote:

I am seeing the procurement tasks being processed already, does that mean we have established that this controller will work 100% for us then?

I think Rob's tests with 5.16 and 5.10 were fine, so that means it's supported by the default Bullseye kernel. Unlikely by older OSes (which weren't even tested), but we decided not to care about them.

There are still various followups. At least:

• there's some work needed for Partman if I'm not mistaken (I lack the details, but Rob mentioned it on IRC)
• we need to figure out with Legal if we can add perccli64 to apt.wikimedia.org (or failing that we might need to consider an additional internal repo)
• perccli64 needs a deb package
• we need to adapt monitoring to also support perccli64 in the RAID checks (and some Puppet integration like the raid Puppet fact)

But given the long lead times of server purchases I'd be cautiously optimistic that all of this is done when the servers arrive.

Thanks @MoritzMuehlenhoff!

@jbond note to self, look at extending raid fact to support new controller

Change 809602 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the partman recipe for use with the new stat servers

https://gerrit.wikimedia.org/r/809602

Change 809602 merged by Btullis:

[operations/puppet@production] Update the partman recipe for use with the new stat servers

https://gerrit.wikimedia.org/r/809602

I have been doing some testing of an install of a server with an H750 card under ticket T307399 recently.

One thing that I have ascertained, with the help of @fgiunchedi, is that the swapping of /dev/sda and /dev/sdb can be reversed, so we don't need to make a lot of custom partman recipes for these hosts with H750 cards.

The key bit of information is this:

Which is from this page of the PERC11 User Guide.

I deleted both the RAID1 and the RAID10 virtual drives, then recreated them in the order:

1. RAID10
2. RAID1

Now when the server boots the SFF RAID1 drives are successfully detected as /dev/sda and the LFF RAID10 drives are detected as /dev/sdb

I can add some information to https://wikitech.wikimedia.org/wiki/Raid_setup but I thought I'd mention it here first, because I know that some people are now installing these servers and running up against the drive ordering issue.
cc @RobH and @Andrew and, I think, @Papaul.

Change 809641 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] testing h750 recipes

https://gerrit.wikimedia.org/r/809641

In T297913#8037888, @gerritbot wrote:

Change 809641 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] testing h750 recipes

https://gerrit.wikimedia.org/r/809641

Bah, this was meant to reference the dumpsdata install not this task, my bad.

So post dumpsdata1007 install it fails puppet due to megaraid monitoring items it seems?

In T297913#8038074, @RobH wrote:

So post dumpsdata1007 install it fails puppet due to megaraid monitoring items it seems?

That's expected, we still need to adapt the "raid" fact in Puppet so that it installs perccli (but for that we needed a running system with Perc controller, so that we can figure out the device names which allow Puppet to detect the controller). Just leave the system in that state and we'll use dumpsdata1007 for that?

In T297913#8038091, @MoritzMuehlenhoff wrote:

In T297913#8038074, @RobH wrote:

So post dumpsdata1007 install it fails puppet due to megaraid monitoring items it seems?

That's expected, we still need to adapt the "raid" fact in Puppet so that it installs perccli (but for that we needed a running system with Perc controller, so that we can figure out the device names which allow Puppet to detect the controller). Just leave the system in that state and we'll use dumpsdata1007 for that?

Works for me, I'll put this comment reference on the setup task there. Thanks!

Change 809913 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Extend custom raid fact to support Perc 750

https://gerrit.wikimedia.org/r/809913

There was one more issue to address with these servers, which (thanks once again to @fgiunchedi) we have now identified and overcome.

It was related to the enumeration/ordering of the devices so when configured to boot from the RAID controller which logical disk would be selected for boot.

After deleting and re-creating the logical disks in reverse order (T297913#8037638) it seems that the first disk created was being selected. This corresponds to /dev/sdb as far as the operating system is concerned, which is not where grub was being installed.

We pressed F2 to get into into System Setup and then selected the following options:

-> Device Settings
-> -> RAID Controller in Slot 6: Dell PERC H750 Adapter Configuration Utility
-> -> -> Main Menu
-> -> -> -> Controller Management
-> -> -> -> -> Select Boot Device

At this point we could choose which device would be presented to the O/S as the boot device and could change the default setting to the RAID1 SFF drives.

I'll update https://wikitech.wikimedia.org/wiki/Raid_setup with this information. I don't know if this is something that could be done with the automatic setup or whether it would have to be a manual step

Interestingly, I couldn't find a way to configure this setting from the HTTPS user interface.

We also had to change the default boot sequence to use the HDD first, although this is less of an obscure setting.

Change 811667 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] Add Nagios script for monitoring the Dell PERC RAID controller.

https://gerrit.wikimedia.org/r/811667

Change 812250 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] c:raid::perccli add PowerEdge RAID Controller monitoring to Icinga.

https://gerrit.wikimedia.org/r/812250

Change 811667 merged by Slyngshede:

[operations/puppet@production] Add Nagios script for monitoring the Dell PERC RAID controller.

https://gerrit.wikimedia.org/r/811667

Change 809913 abandoned by Muehlenhoff:

[operations/puppet@production] Extend custom raid fact to support Perc 750

Reason:

Obsoleted by the new raid_mgmt_tools fact

https://gerrit.wikimedia.org/r/809913

Change 809913 restored by Slyngshede:

[operations/puppet@production] Extend custom raid fact to support Perc 750

https://gerrit.wikimedia.org/r/809913

Change 809913 abandoned by Slyngshede:

[operations/puppet@production] Extend custom raid fact to support Perc 750

Reason:

https://gerrit.wikimedia.org/r/809913

Change 825728 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] c:raid::perccli add PowerEdge RAID Controller monitoring to Icinga.

https://gerrit.wikimedia.org/r/825728

Change 812250 abandoned by Slyngshede:

[operations/puppet@production] c:raid::perccli add PowerEdge RAID Controller monitoring to Icinga.

Reason:

See: 825728

https://gerrit.wikimedia.org/r/812250

Change 825728 merged by Slyngshede:

[operations/puppet@production] c:raid::perccli add PowerEdge RAID Controller monitoring to Icinga.

https://gerrit.wikimedia.org/r/825728