@Ottomata this should be easy to do right? Or is there any special consideration to make to avoid fireworks? :D

Should be easy peasy, there's nothing special or fancy here, and the hosts are HA LB-ed, so you should be able to just depool, reinstall, puppetize, repool.

If I got it correctly the 4 schema hosts are VM on ganeti, so we cannot upgrade them in place. We could create 4 new VMs identical but with Buster (schema[12]00[34]), apply to them the correct puppet roles and then flip all the configs using the old nodes to the new ones. Finally we'd drop the old VMs and the upgrade would be completed.

When you do that, please use row B/D in eqiad and row C/D in codfw to better balance out our Ganeti groups.

Vms created, next steps:

• check that vms are working fine etc.. (I checked that specs are correct but @Ottomata should verify that all the envoy services are good on all VMs just to be sure)
• add new VMs to LVS
• depool old ones from LVS
• decommission old VMs

Change 622567 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add new schema[12]00[34] vms to the related LVS endpoints

https://gerrit.wikimedia.org/r/622567

Change 622567 merged by Elukey:
[operations/puppet@production] Add new schema[12]00[34] vms to the related LVS endpoints

https://gerrit.wikimedia.org/r/622567

New vms added to the LVS endpoints:

https://config-master.wikimedia.org/pybal/eqiad/schema
https://config-master.wikimedia.org/pybal/codfw/schema

If everything works correctly, we'll need to:

• remove the old VMs from LVS
• decommission them

Change 622587 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove schema[12]00[12] from their LVS endpoint configs

https://gerrit.wikimedia.org/r/622587

Change 622587 merged by Elukey:
[operations/puppet@production] Remove schema[12]00[12] from their LVS endpoint configs

https://gerrit.wikimedia.org/r/622587

Change 622754 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove old schema[12]00[12] from puppet

https://gerrit.wikimedia.org/r/622754

Change 622754 merged by Elukey:
[operations/puppet@production] Remove old schema[12]00[12] from puppet

https://gerrit.wikimedia.org/r/622754

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: schema1001.eqiad.wmnet

• schema1001.eqiad.wmnet (PASS)
  • Downtimed host on Icinga
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: schema1002.eqiad.wmnet

• schema1002.eqiad.wmnet (PASS)
  • Downtimed host on Icinga
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: schema2001.codfw.wmnet

• schema2001.codfw.wmnet (PASS)
  • Downtimed host on Icinga
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: schema2002.codfw.wmnet

• schema2002.codfw.wmnet (PASS)
  • Downtimed host on Icinga
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed

Change 622755 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Remove schema[12]00[12] records (VMs decommissioned)

https://gerrit.wikimedia.org/r/622755

Change 622755 merged by Elukey:
[operations/dns@master] Remove schema[12]00[12] records (VMs decommissioned)

https://gerrit.wikimedia.org/r/622755

All done!