Page MenuHomePhabricator

Migrate eqiad Ganeti cluster to Buster
Open, MediumPublic

Description

  • Add component/ganeti216 to all nodes
  • Upgrade all nodes using "cumin A:ganeti-eqiad 'apt-get install -y ganeti'"
  • cumin A:ganeti-eqiad 'chown gnt-metad /var/log/ganeti/meta-daemon.log'
  • sudo gnt-cluster renew-crypto --new-cluster-certificate --new-rapi-certificate --new-spice-certificate
  • sudo gnt-cluster upgrade --to 2.16
  • sudo gnt-cluster renew-crypto --new-node-certificates (might not be needed for future updates if the hashing algo stays the same)

Following that empty/reimage/readd the following virtualisation servers:

  • ganeti1005.eqiad.wmnet
  • ganeti1006.eqiad.wmnet
  • ganeti1007.eqiad.wmnet
  • ganeti1008.eqiad.wmnet
  • ganeti1009.eqiad.wmnet
  • ganeti1010.eqiad.wmnet
  • ganeti1011.eqiad.wmnet
  • ganeti1012.eqiad.wmnet
  • ganeti1013.eqiad.wmnet
  • ganeti1014.eqiad.wmnet
  • ganeti1015.eqiad.wmnet
  • ganeti1016.eqiad.wmnet
  • ganeti1017.eqiad.wmnet
  • ganeti1018.eqiad.wmnet
  • ganeti1019.eqiad.wmnet
  • ganeti1020.eqiad.wmnet
  • ganeti1021.eqiad.wmnet
  • ganeti1022.eqiad.wmnet

The following instances were temporarily converted from plain disk storage to DRBD:

  • kubestagetcd1004
  • kubestagetcd1005
  • kubestagetcd1006
  • kubetcd1004
  • kubetcd1005
  • kubetcd1006
  • ml-etcd1001
  • ml-etcd1002
  • ml-etcd1003

Related Objects

Event Timeline

Change 742730 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point back irc.wikimedia.org to irc2001

https://gerrit.wikimedia.org/r/742730

Change 742730 merged by Muehlenhoff:

[operations/dns@master] Point back irc.wikimedia.org to irc2001

https://gerrit.wikimedia.org/r/742730

herron triaged this task as Medium priority.Dec 3 2021, 7:08 PM

Change 754540 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Enable ganeti 2.16 in eqiad

https://gerrit.wikimedia.org/r/754540

Change 754540 merged by Muehlenhoff:

[operations/puppet@production] Enable ganeti 2.16 in eqiad

https://gerrit.wikimedia.org/r/754540

Mentioned in SAL (#wikimedia-operations) [2022-01-18T09:50:01Z] <moritzm> installing ganeti 2.16.0-1~bpo9+1+wmf1 on ganeti/eqiad servers T296721

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1018.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1018.eqiad.wmnet with OS buster executed with errors:

  • ganeti1018 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1018.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1018.eqiad.wmnet with OS buster completed:

  • ganeti1018 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201200950_jmm_3211404_ganeti1018.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB