Both can be turned off at the same time.
They will require new IPs:
public: addr: "10.64.148.16" iface: "ens2f0np0" cluster: addr: "192.168.5.8" prefix: "24" iface: "ens2f1np1"
public: addr: "10.64.149.14" iface: "ens3f0np0" cluster: addr: "192.168.6.6" prefix: "24" iface: "ens3f1np1"
For the record, the final process used was:
- wmcs.ceph.osd.depool_and_destroy cookbook (remove all the osds from the host and remove CRUSH entries for the)
- Move the hosts to the new racks
- In puppet, edit hieradata/eqiad/profile/cloudceph/osd.yaml with new IPs on the new ranges (public and cluster networks) if needed (search the range in Netbox for the next free IP in the range)
- Half-follow https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Rename_while_reimaging
- Move from DECOMMISSIONING to PLANNED
- Add only the public IP to the main interface
- flag that intefrace as primary
- Add also an fqdn to that new IP
- Add also an fqdn to the mgmt IP (if not there)
- Run the sre.dns.netbox
- Merge the patch with the new IPs
- Upgrade the idrac firmware (cookbook sre.hardware.upgrade-firmware -n -c idrac cloudcephosd1004)
- Upgrade the nic firmware (cookbook sre.hardware.upgrade-firmware -n -c nic cloudcephosd1004)
- Reimage the host (cookbook sre.hosts.reimage --os bullseye --new -t T329502 cloudcephosd1004)
- Repeat the reimage until it works (puppet might timeout, etc., you can check the console by sshing to root@<hostname>.mgmt.eqiad.wmnet, use mgmt pass)
- Put the host back in ceph (wmcs.ceph.osd.bootstrap_and_add), it might take a while to finish the rebalancing
EDIT (cmooney): For the record I believe the best process to follow for these kind of moves is the one outlined below: