@herron, it might be worth renaming these hosts! E.g. kafka-main200[4-8] would be more appropriate! :)

Good point! And if we number from 200[1-5] it should simplify mapping of broker IDs between old and new hosts too. I updated the description to reflect this, but if you think its best to keep the 200[4-8] suffix happy to go that route instead.

Awesome, ya that works!

@Ottomata can you please provide me with the partman recipe for those systems and for the RAID config do you want user 256KB for RAID Stripe size?

can you please provide me with the partman recipe for those systems

raid10-gpt-srv-lvm-ext4.cfg would work, but it uses only disks. I think you should be able to make a new modified one to use all 8.

do you want user 256KB for RAID Stripe size?

Hm, hadn't thought about this. I'd think a largeish stripe size would be good for Kafka. Reads are usually very sequential. Writes are done in batches, but the sizes can vary based on the volume of particular topic-partitions. I do see quite a few smallish files for indexes.

I just looked at kafka1001, but I'm not exactly sure if I found the right thing. Is mdadm 'chunk size' the same as the stripe size? If so, it is set at 512K there. I don't think we changed it intentionally for other kafka hosts, so perhaps mdadm's default is fine?

@Ottomata thanks for the updates. Can you please make the necessaries modifications needed on raid10-gpt-srv-lvm-ext4.cfg .

Please note these are showing as an error state of staged in netbox, when they are not yet installed with an OS and have not yet run puppet.

I have changed all of the kafka-main200[1-5] to state 'planned' in netbox. Once the OS is installed (and initial puppet run is completed immediately after OS installation), it can change to staged.

Can you please make the necessaries modifications needed on raid10-gpt-srv-lvm-ext4.cfg .

Ping @herron on this one, SRE is handling these. :)

Change 511952 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] partman: add 8 disk raid10 layout

https://gerrit.wikimedia.org/r/511952

Change 511952 merged by Herron:
[operations/puppet@production] partman: add 8 disk raid10 layout

https://gerrit.wikimedia.org/r/511952

Hey @Papaul, I added a raid10-gpt-srv-lvm-ext4-8disks.cfg for the initial installs on these.

Once they are up and running I'll do a little benchmarking to try and see if there isn't a better sweet spot in terms of stripe/chunk sizes, and update the config if needed. Thanks!

Change 512069 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add mgmt and production DNS for kafka-main200[1-5]

https://gerrit.wikimedia.org/r/512069

Change 512069 merged by Herron:
[operations/dns@master] DNS: Add mgmt and production DNS for kafka-main200[1-5]

https://gerrit.wikimedia.org/r/512069

Change 512236 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address for kafka-main2001

https://gerrit.wikimedia.org/r/512236

Change 512236 merged by Dzahn:
[operations/puppet@production] DHCP: Add MAC address for kafka-main2001

https://gerrit.wikimedia.org/r/512236

@herron can't find a partman recipe

────────────────────────┤ [!!] Partition disks ├─────────────────────────┐

│                                                                         │   
│ The installer can guide you through partitioning a disk (using          │   
│ different standard schemes) or, if you prefer, you can do it            │   
│ manually. With guided partitioning you will still have a chance later   │   
│ to review and customise the results.                                    │   
│                                                                         │   
│ If you choose guided partitioning for an entire disk, you will next     │   
│ be asked which disk should be used.                                     │   
│                                                                         │   
│ Partitioning method:                                                    │   
│                                                                         │   
│          Guided - use entire disk                                       │   
│          Guided - use entire disk and set up LVM                        │   
│          Guided - use entire disk and set up encrypted LVM              │   
│          Manual                                                         │   
│                                                                         │   
│     <Go Back>                                                           │   
│

ok, no worries I'll poke at this for a bit and try to get it installed

Change 512307 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] netboot: assign kafka-main[12]00[1-5] 8 disk raid10 partman config

https://gerrit.wikimedia.org/r/512307

Change 512307 merged by Herron:
[operations/puppet@production] netboot: assign kafka-main[12]00[1-5] 8 disk raid10 partman config

https://gerrit.wikimedia.org/r/512307

Kafka-main2001 is installed. I updated the netboot config to assign the partman config to these hostnames, and switched the hardware controller to HBA mode. Then it completed the install using the 8 disk md raid10 config. Now to do some testing!

@herron also after the OS install please remember to change Netbox status to "staged"

I did some testing of various software and hardware raid configurations and wrote up a summary at https://wikitech.wikimedia.org/wiki/Kafka/Kafka-main-raid-performance-testing-2019

The tldr is that indeed 512K chunk size tested to provide quite good performance, when combined with the noop scheduler. Surprisingly (or maybe not so much since we standardized on a bit older generation raid cards) hardware raid10 tested quite a bit worse than software raid10. The difference was significant enough that I think it justifies moving forward with sw raid.

So, IMO the optimal path forward for these hosts will be raid cards in HBA mode, and OS installed using the raid10-gpt-srv-lvm-ext4-8disks.cfg partman config. Barring objections I'll complete the builds using this layout.

@Papaul could you have a look at kafka-main2002? It seems to be stuck, at least I'm not able to open a console or power cycle.

/admin1-> racadm serveraction powercycle
Unable to perform requested operation.
/admin1-> racadm serveraction powerstatus
Server power status: ON
/admin1-> racadm serveraction powerdown
Unable to perform requested operation.
/admin1-> racadm serveraction hardreset
Unable to perform requested operation.

2. kafka-main200[2345] are not yet able to net boot

13:25 < papaul> herron: the problem is i added just kafka-main2001 to the DHCP file and

not the others that is why you can not boot the other servers

13:25 < papaul> since you wanted to do only the test on 2001

I would expect though that the DHCP requests would make it to the install servers, with or without entries in the dhcp config file.

kafak-main2002
after power drain

/admin1-> racadm serveraction powercycle
Server power operation initiated successfully
/admin1->

In T223493#5230429, @herron wrote:

I would expect though that the DHCP requests would make it to the install servers, with or without entries in the dhcp config file.

Depending on which interface you selecting for pxe boot by defaut those servers will boot on the 1GB interface and there is nothing plugged into the 1Gb interface. You will have to go in the BIOS disable the 1Gb interfaces. I am doing all that so need to bother.

Ok, thanks!

Change 514030 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address entries for kafka-main200[2-5]

https://gerrit.wikimedia.org/r/514030

Change 514030 merged by Herron:
[operations/puppet@production] DHCP: Add MAC address entries for kafka-main200[2-5]

https://gerrit.wikimedia.org/r/514030

Change 514107 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] install_server: tweak raid10 8disk partman layout

https://gerrit.wikimedia.org/r/514107

Change 514107 merged by Herron:
[operations/puppet@production] install_server: tweak raid10 8disk partman layout

https://gerrit.wikimedia.org/r/514107

@herron fixed

That did the trick! All of the new codfw kafka-main hosts are now installed and ready for service setup

Tracking service implementation in T225005