Page MenuHomePhabricator
Create Task
Maniphest T342463

Write new partman recipe for cloudelastic
Closed, ResolvedPublic
Actions

Description

In the last round of hardware upgrades ( T341214 ), we decided to use RAID0 instead of RAID 10 for cloudelastic hosts. This means we'll need to write a new partman recipe for the upcoming cloudelastic hosts (cloudelastic1007-10).

To do this, we'll need to:

  • Write a new partman recipe and save to ./modules/install_server/files/autoinstall/partman in the puppet repo
  • Add an entry to modules/install_server/files/autoinstall/netboot.cfg in the puppet repo.

The new recipe will need RAID-1 for the OS and RAID-0 for the remaining space.

AC:

  • new cloudelasticservers are configured with the new recipe
  • Elastic is running stably, with acceptable performance.

Details

Other Assignee
RKemper
SubjectRepoBranchLines +/-
cloudelastic: include raid0.cfg in netboot.cfgoperations/puppetproduction+1 -1
partman: fix raid0-3dev.cfgoperations/puppetproduction+3 -3
cloudelastic: new partman recipeoperations/puppetproduction+22 -0
partman: remove all traces of cloudelasticoperations/puppetproduction+0 -22
cloudelastic: new partman recipeoperations/puppetproduction+22 -0
cloudelastic: correct partman recipeoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Jul 21 2023, 7:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2023, 7:26 PM
bking mentioned this in Unknown Object (Task).Jul 21 2023, 7:31 PM
bking added a comment.Jul 21 2023, 8:40 PM

Haven't looked closely, but I'm guessing the following recipes from modules/install_server/files/autoinstall/netboot.cfg could work (or would be the easiest to adapt):

restbase1019|restbase102[0-8]|restbase103[0-3]) echo reuse-parts.cfg  partman/custom/reuse-cassandrahosts-3ssd-jbod.cfg ;; \
restbase2009|restbase201[12]) echo reuse-parts.cfg partman/custom/reuse-cassandrahosts-4ssd-jbod.cfg ;; \
restbase201[3-9]|restbase202[0-6]) echo reuse-parts.cfg partman/custom/reuse-cassandrahosts-3ssd-jbod.cfg ;; \
bking updated the task description. (Show Details)Jul 21 2023, 8:41 PM
RobH mentioned this in T342538: Q1:rack/setup/install cloudelastic10[07-10].wikimedia.org.Jul 24 2023, 12:59 PM
Gehel removed a project: Discovery-Search.Jul 24 2023, 3:35 PM
bking renamed this task from Write new partman recipe for cloudelastic (jbod) to Write new partman recipe for cloudelastic (jbod) and update relevant Elastic config.Jul 24 2023, 3:37 PM
bking updated the task description. (Show Details)
Gehel updated the task description. (Show Details)Jul 25 2023, 3:47 PM
Gehel triaged this task as Medium priority.Jul 25 2023, 3:50 PM
BTullis moved this task from Incoming to Misc on the Data-Platform-SRE board.Aug 23 2023, 8:54 AM
bking claimed this task.Sep 19 2023, 9:19 PM
bking moved this task from Misc to In Progress on the Data-Platform-SRE board.

This is blocking T342538 , so I'm starting on it now.

bking updated Other Assignee, added: RKemper.Sep 19 2023, 9:21 PM
bking added a parent task: T231010: Change partitioning scheme for elasticsearch from RAID to JBOD.Sep 21 2023, 7:54 PM
gerritbot added a comment.Sep 22 2023, 5:35 PM

Change 960114 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: new partman recipe

https://gerrit.wikimedia.org/r/960114

gerritbot added a project: Patch-For-Review.Sep 22 2023, 5:35 PM
bking renamed this task from Write new partman recipe for cloudelastic (jbod) and update relevant Elastic config to Write new partman recipe for cloudelastic.Sep 22 2023, 5:53 PM
bking updated the task description. (Show Details)
fgiunchedi added subscribers: Muehlenhoff, fgiunchedi.Sep 25 2023, 9:07 AM

Please excuse the drive-by comment, I've worked with @Muehlenhoff on standardizing our partman recipes and I'm wondering if the standard raid0 recipes (i.e. raid1 for / and raid0 for /srv) would work in this case?

In other words the same as elastic* (and adjusting for the number of devices as needed)

elastic*) echo partman/standard.cfg partman/raid0.cfg partman/raid0-2dev.cfg ;; \
MoritzMuehlenhoff subscribed.Sep 25 2023, 9:47 AM
bking added a comment.Sep 25 2023, 8:00 PM
In T342463#9194350, @fgiunchedi wrote:

Please excuse the drive-by comment, I've worked with @Muehlenhoff on standardizing our partman recipes and I'm wondering if the standard raid0 recipes (i.e. raid1 for / and raid0 for /srv) would work in this case?

In other words the same as elastic* (and adjusting for the number of devices as needed)

elastic*) echo partman/standard.cfg partman/raid0.cfg partman/raid0-2dev.cfg ;; \

You must be psychic! I was just looking at the recipes again and thinking the same thing. I'll try starting with partman/raid0-3dev.cfg instead of a custom recipe and work from there.

fgiunchedi added a comment.Sep 26 2023, 7:26 AM

Sounds great, please reach out and/or send reviews if sth is amiss with the standard recipes

gerritbot added a comment.Sep 26 2023, 5:04 PM

Change 960114 merged by Bking:

[operations/puppet@production] cloudelastic: new partman recipe

https://gerrit.wikimedia.org/r/960114

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2023, 5:11 PM
gerritbot added a comment.Sep 26 2023, 6:56 PM

Change 961186 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: correct partman recipe

https://gerrit.wikimedia.org/r/961186

gerritbot added a project: Patch-For-Review.Sep 26 2023, 6:56 PM

Change 961186 merged by Bking:

[operations/puppet@production] cloudelastic: correct partman recipe

https://gerrit.wikimedia.org/r/961186

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2023, 7:11 PM
gerritbot added a comment.Sep 26 2023, 9:30 PM

Change 961245 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] partman: remove all traces of cloudelastic

https://gerrit.wikimedia.org/r/961245

gerritbot added a project: Patch-For-Review.Sep 26 2023, 9:30 PM
gerritbot added a comment.Sep 26 2023, 9:33 PM

Change 961245 merged by Bking:

[operations/puppet@production] partman: remove all traces of cloudelastic

https://gerrit.wikimedia.org/r/961245

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2023, 10:10 PM
bking mentioned this in T347434: Apply partman recipe patch again and see if it affects unrelated reimages.Sep 26 2023, 10:18 PM
bking added a subtask: T347434: Apply partman recipe patch again and see if it affects unrelated reimages.Sep 26 2023, 10:22 PM
bking added a comment.EditedSep 26 2023, 10:24 PM

Still broken. It's also possible (but extremely unlikely) that the change affected other builds, see T347434 .

Confirmed that the change did not affect other image operations.

bking closed subtask T347434: Apply partman recipe patch again and see if it affects unrelated reimages as Resolved.Sep 27 2023, 5:12 PM
gerritbot added a comment.Sep 27 2023, 9:35 PM

Change 961478 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: new partman recipe

https://gerrit.wikimedia.org/r/961478

gerritbot added a project: Patch-For-Review.Sep 27 2023, 9:35 PM
gerritbot added a comment.Sep 28 2023, 5:09 PM

Change 961478 merged by Bking:

[operations/puppet@production] cloudelastic: new partman recipe

https://gerrit.wikimedia.org/r/961478

Maintenance_bot removed a project: Patch-For-Review.Sep 28 2023, 5:11 PM
bking added a comment.Sep 28 2023, 6:10 PM

The new recipe still errs with message`Failed to load ldlinux.c32`. That doesn't sound like a partitioning problem. Will attempt a firmware update and get back.

bking added a comment.Oct 4 2023, 2:44 PM

OK, I got the host to boot, now we're getting partman errors (fetched from /var/log/partman via install-console)

/lib/partman/init.d/25md-devices: *******************************************************
/lib/partman/init.d/30parted: *******************************************************
parted_server: ======= Starting the server
parted_server: main_loop: iteration 1
parted_server: Opening infifo
parted_server: Read command: PARTITIONS
parted_server: The device =dev=sda is not opened.
parted_server: Line 1418. CRITICAL ERROR!!!  EXITING.
/lib/partman/init.d/30parted: IN: OPEN =dev=md0 /dev/md0

Will compare the recipes again and adjust as necessary.

gerritbot added a comment.Oct 4 2023, 2:49 PM

Change 963328 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] partman: fix raid0-3dev.cfg

https://gerrit.wikimedia.org/r/963328

gerritbot added a project: Patch-For-Review.Oct 4 2023, 2:49 PM
fgiunchedi added a comment.Oct 4 2023, 2:50 PM

The merged partman recipe contains multiple errors, as pointed out in https://gerrit.wikimedia.org/r/c/operations/puppet/+/961478/comments/92a7a209_4f5f5f84

gerritbot added a comment.Oct 4 2023, 2:54 PM

Change 963328 merged by Bking:

[operations/puppet@production] partman: fix raid0-3dev.cfg

https://gerrit.wikimedia.org/r/963328

Maintenance_bot removed a project: Patch-For-Review.Oct 4 2023, 3:11 PM
gerritbot added a comment.Oct 4 2023, 3:13 PM

Change 963334 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: include raid0.cfg in netboot.cfg

https://gerrit.wikimedia.org/r/963334

gerritbot added a project: Patch-For-Review.Oct 4 2023, 3:13 PM

Change 963334 merged by Bking:

[operations/puppet@production] cloudelastic: include raid0.cfg in netboot.cfg

https://gerrit.wikimedia.org/r/963334

Maintenance_bot removed a project: Patch-For-Review.Oct 4 2023, 3:30 PM
bking updated the task description. (Show Details)Oct 4 2023, 4:33 PM

Confirmed that the partman recipe is working via install-console:

root@cloudelastic1007:~# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda       8:0    0  1.7T  0 disk
├─sda1    8:1    0  285M  0 part
├─sda2    8:2    0 74.5G  0 part
│ └─md0   9:0    0 74.4G  0 raid1 /
├─sda3    8:3    0  977M  0 part
│ └─md1   9:1    0  976M  0 raid1 [SWAP]
└─sda4    8:4    0  1.7T  0 part
  └─md2   9:2    0    5T  0 raid0 /srv
sdb       8:16   0  1.7T  0 disk
├─sdb1    8:17   0  285M  0 part
├─sdb2    8:18   0 74.5G  0 part
│ └─md0   9:0    0 74.4G  0 raid1 /
├─sdb3    8:19   0  977M  0 part
│ └─md1   9:1    0  976M  0 raid1 [SWAP]
└─sdb4    8:20   0  1.7T  0 part
  └─md2   9:2    0    5T  0 raid0 /srv
sdc       8:32   0  1.7T  0 disk
├─sdc1    8:33   0  285M  0 part
├─sdc2    8:34   0 74.5G  0 part
│ └─md0   9:0    0 74.4G  0 raid1 /
├─sdc3    8:35   0  977M  0 part
│ └─md1   9:1    0  976M  0 raid1 [SWAP]
└─sdc4    8:36   0  1.7T  0 part
  └─md2   9:2    0    5T  0 raid0 /srv
bking closed this task as Resolved.Oct 4 2023, 4:41 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits