Page MenuHomePhabricator
Create Task
Maniphest T437516

sessionstore: Debian Bookworm upgrades
Closed, ResolvedPublic
Actions

Description

Upgrade hosts to Debian Bookworm,

hosthardwarecommentsstatus
sessionstore1004Dell r450✅
sessionstore1005Dell r450✅
sessionstore1006Dell r450✅
hosthardwarecommentsstatus
sessionstore2004Dell r450✅
sessionstore2005Dell r450✅
sessionstore2006Dell r450✅

✅ = Completed

Steps

Locally, ensure that none of the JBOD devices have RAID metadata on them...

locally
$ sudo mdadm --zero-superblock /dev/sdX3

Do this for every device that is NOT a part of the RAID array. For example, if the array uses /dev/sda3 & /dev/sdb3, then do this for each of /dev/sd[c-h]3 (use mdadm -detail /dev/md0 to check).

cumin
HOST=sessionstore2005
DOMAIN=codfw.wmnet
TICKET=T437516
sudo cookbook sre.hosts.downtime -r "Firmware upgrades — $TICKET" -t $TICKET -H 1 $HOST.$DOMAIN
sudo cookbook sre.hardware.upgrade-firmware $HOST.$DOMAIN
sudo cookbook sre.hosts.provision $HOST --no-users --no-dhcp --no-switch --legacy
sudo cookbook sre.hosts.reimage -t $TICKET --os bookworm $HOST

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Revert "sessionstore1005: partition tables lost; full reimage needed"Eevansoperations/puppetproduction+1 -4
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT357791 Cassandra upgrades to Debian Bookworm
 ResolvedEevansT437516 sessionstore: Debian Bookworm upgrades

Event Timeline

Eevans created this task.Sep 9 2026, 8:31 PM
Eevans triaged this task as High priority.
Eevans updated the task description. (Show Details)Sep 9 2026, 8:33 PM
Eevans added a comment.Sep 9 2026, 9:05 PM

Upgraded sessionstore2004 DRAC to 7.30.30.51, and BIOS to 1.20.2.

ops-monitoring-bot added a comment.Sep 9 2026, 9:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1003 for host sessionstore2004.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 9 2026, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1003 for host sessionstore2004.codfw.wmnet with OS bookworm executed with errors:

  • sessionstore2004 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sessionstore2004.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 9 2026, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1003 for host sessionstore2004.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 9 2026, 9:59 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1003 for host sessionstore2004.codfw.wmnet with OS bookworm executed with errors:

  • sessionstore2004 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sessionstore2004.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 9 2026, 10:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1003 for host sessionstore2004.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 9 2026, 10:45 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1003 for host sessionstore2004.codfw.wmnet with OS bookworm completed:

  • sessionstore2004 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609092224_eevans_4030177_sessionstore2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Eevans updated the task description. (Show Details)Sep 9 2026, 11:17 PM
Eevans added a comment.Sep 9 2026, 11:25 PM

sessionstore2004's install went pear-shaped with a missing volume for the root filesystem. It would seem that the disks we added to the JBOD (quite some time ago) came from another machine with software raid, and the superblock was not zeroed. d-i wasn't able to assemble md0, which was the single pv that the volumes were built on. I was able to get past this by zeroing the superblock on the affected disks and re-running the reimage cookbook.

Moving forward with the remaining hosts I think we should: sudo mdadm --zero-superblock /dev/sdX3, for every value of X that isn't a device that is a part of the md0.

ops-monitoring-bot added a comment.Sep 10 2026, 8:01 PM

Icinga downtime and Alertmanager silence (ID=48e590a6-254f-42e9-b4b4-128790a61ca7) set by eevans@cumin1003 for 1:00:00 on 1 host(s) and their services with reason: Firmware upgrades — T437516

sessionstore2005.codfw.wmnet
Stashbot added a comment.Sep 10 2026, 8:01 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-10T20:01:46Z] <eevans@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore2005.codfw.wmnet with reason: Firmware upgrades — T437516

ops-monitoring-bot added a comment.Sep 10 2026, 8:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1003 for host sessionstore2005.codfw.wmnet with OS bookworm

Eevans updated the task description. (Show Details)Sep 10 2026, 8:53 PM
Eevans updated the task description. (Show Details)Sep 10 2026, 9:00 PM
ops-monitoring-bot added a comment.Sep 10 2026, 9:14 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1003 for host sessionstore2005.codfw.wmnet with OS bookworm completed:

  • sessionstore2005 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609102054_eevans_1036_sessionstore2005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 10 2026, 9:19 PM

Icinga downtime and Alertmanager silence (ID=6544f2b5-5657-43d4-8b32-a3f35bab38e0) set by eevans@cumin1003 for 1:00:00 on 1 host(s) and their services with reason: Firmware upgrades — T437516

sessionstore2006.codfw.wmnet
Stashbot added a comment.Sep 10 2026, 9:19 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-10T21:19:17Z] <eevans@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore2006.codfw.wmnet with reason: Firmware upgrades — T437516

Eevans updated the task description. (Show Details)Sep 10 2026, 9:19 PM
ops-monitoring-bot added a comment.Sep 10 2026, 9:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1003 for host sessionstore2006.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 10 2026, 10:00 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1003 for host sessionstore2006.codfw.wmnet with OS bookworm executed with errors:

  • sessionstore2006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sessionstore2006.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 10 2026, 10:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1003 for host sessionstore2006.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 10 2026, 10:44 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1003 for host sessionstore2006.codfw.wmnet with OS bookworm completed:

  • sessionstore2006 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609102223_eevans_12707_sessionstore2006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Eevans updated the task description. (Show Details)Sep 10 2026, 10:55 PM
ops-monitoring-bot added a comment.Mon, Sep 14, 1:40 PM

Icinga downtime and Alertmanager silence (ID=738547aa-f863-4d0a-b2c0-93de8e2b1edd) set by eevans@cumin1004 for 1:00:00 on 1 host(s) and their services with reason: Firmware upgrades — T437516

sessionstore1004.eqiad.wmnet
Stashbot added a comment.Mon, Sep 14, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-14T13:40:37Z] <eevans@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore1004.eqiad.wmnet with reason: Firmware upgrades — T437516

ops-monitoring-bot added a comment.Mon, Sep 14, 2:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1004 for host sessionstore1004.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Sep 14, 2:59 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1004 for host sessionstore1004.eqiad.wmnet with OS bookworm completed:

  • sessionstore1004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609141438_eevans_2110684_sessionstore1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Sep 14, 3:06 PM

Host sessionstore1004.eqiad.wmnet rebooted by eevans@cumin1004 with reason: Post-disk config reboot test — T437516

Eevans updated the task description. (Show Details)Mon, Sep 14, 3:08 PM
ops-monitoring-bot added a comment.Mon, Sep 14, 3:17 PM

Icinga downtime and Alertmanager silence (ID=4fc1518e-1fea-4baa-84d9-3a714c172327) set by eevans@cumin1004 for 1:00:00 on 1 host(s) and their services with reason: Firmware upgrades — T437516

sessionstore1005.eqiad.wmnet
Stashbot added a comment.Mon, Sep 14, 3:17 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-14T15:17:30Z] <eevans@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore1005.eqiad.wmnet with reason: Firmware upgrades — T437516

ops-monitoring-bot added a comment.Mon, Sep 14, 3:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Sep 14, 3:47 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm executed with errors:

  • sessionstore1005 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sessionstore1005.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, Sep 14, 3:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm

Eevans mentioned this in T437915: sessionstore1005.eqiad.wmnet is down (again).Mon, Sep 14, 3:56 PM
ops-monitoring-bot added a comment.Mon, Sep 14, 4:51 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm executed with errors:

  • sessionstore1005 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sessionstore1005.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, Sep 14, 4:52 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Sep 14, 6:49 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm executed with errors:

  • sessionstore1005 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sessionstore1005.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, Sep 14, 7:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Sep 14, 7:55 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm executed with errors:

  • sessionstore1005 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sessionstore1005.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, Sep 14, 7:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Sep 14, 9:26 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm executed with errors:

  • sessionstore1005 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sessionstore1005.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, Sep 14, 9:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Sep 14, 10:22 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm executed with errors:

  • sessionstore1005 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sessionstore1005.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, Sep 14, 10:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Sep 14, 11:01 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1004 for host sessionstore1005.eqiad.wmnet with OS bookworm executed with errors:

  • sessionstore1005 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sessionstore1005.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Sep 15, 5:56 PM

Icinga downtime and Alertmanager silence (ID=78f0a785-bfcc-467f-b822-39c1987ab418) set by eevans@cumin1004 for 1:00:00 on 1 host(s) and their services with reason: Firmware upgrades — T437516

sessionstore1006.eqiad.wmnet
ops-monitoring-bot added a comment.Tue, Sep 15, 6:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1004 for host sessionstore1006.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Sep 15, 7:05 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1004 for host sessionstore1006.eqiad.wmnet with OS bookworm completed:

  • sessionstore1006 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609151844_eevans_2843474_sessionstore1006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Eevans updated the task description. (Show Details)Tue, Sep 15, 7:23 PM
gerritbot added a comment.Tue, Sep 15, 7:25 PM

Change #1342017 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] Revert "sessionstore1005: partition tables lost; full reimage needed"

https://gerrit.wikimedia.org/r/1342017

gerritbot added a project: Patch-For-Review.Tue, Sep 15, 7:25 PM
gerritbot added a comment.Tue, Sep 15, 7:31 PM

Change #1342017 merged by Eevans:

[operations/puppet@production] Revert "sessionstore1005: partition tables lost; full reimage needed"

https://gerrit.wikimedia.org/r/1342017

Eevans added a comment.Tue, Sep 15, 7:36 PM

This needs to turned into a cookbook and/or properly documented on Wikitech, but for posterity sake, here are the full steps I used post-install.

locally
sudo chown -R cassandra: /srv/cassandra/cassandra-a/{commitlog,hints,saved_caches,system}
sudo umount /srv/sda4 && sudo rmdir /srv/sda4
sudo umount /srv/sdb4 && sudo rmdir /srv/sdb4
sudo emacs /etc/fstab  # remove /srv/sd[a-b]4 entries

# Have a look, everything sane?
lsblk -o NAME,TYPE,LABEL,MOUNTPOINT

sudo ./cassandra-drive-configurator -v reuse
sudo chown -R cassandra: /srv/storage-[0-9]/cassandra-a

# Add the remaining devices as RAID-1 spares
for dev in `ls /dev/sd[a-h]3 | grep -vf <(sudo mdadm --detail /dev/md0 |egrep "[[:space:]]+active[[:space:]]+.+/dev/sd[a-h]3" |awk '{ print $7; }')`; do
	sudo mdadm --manage /dev/md0 --add $dev
done

# Grub all the things
for dev in /dev/sd[a-h]; do sudo grub-install $dev; done

sudo touch /etc/cassandra-a/service-enabled    # create systemd unit guard file
sudo systemctl start cassandra-a               # start cassandra...
sudo tail -f /var/log/cassandra/system-a.log   # watch startup progress
cumin
# Just to be sure...
sudo cookbook sre.hosts.reboot-single -t $TICKET -r "Post-disk config reboot test — $TICKET" $HOST.$DOMAIN
Eevans closed this task as Resolved.Tue, Sep 15, 7:37 PM
Eevans claimed this task.

Done.

Maintenance_bot removed a project: Patch-For-Review.Tue, Sep 15, 8:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits