The clusters should be migrated to bullseye and PKI before upgrading the whole clusters to k8s 1.23.
The idea is to do one reimage at the time, doing remove/add member for each of them to allow etcd to bootstrap correctly. Finally we'll just apply PKI settings.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | JMeybohm | T307943 Update Kubernetes clusters to v1.23 | |||
| Resolved | elukey | T324542 Upgrade ML clusters to Kubernetes 1.23 | |||
| Declined | None | T330662 Upgrade the ml-etcd clusters to bullseye and PKI |
Cookbook cookbooks.sre.ganeti.reimage was started by elukey@cumin1001 for host ml-etcd2001.codfw.wmnet with OS bullseye
Change 892462 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] role::etcd::v3::ml_etcd: set cluster status to existing
Change 892462 merged by Elukey:
[operations/puppet@production] role::etcd::v3::ml_etcd: set cluster status to existing
Change 892466 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] role::etcd::v3::ml_etcd: set the new discovery records
Change 892466 merged by Elukey:
[operations/puppet@production] role::etcd::v3::ml_etcd: set the new discovery records
Tried with 2001 but failed to make it work. The new etcd version, on bullseye, requires a new TLS san in every etcd daemon's certificate to be able to run leader elections. Since the other two nodes in the ensemble, 2002 and 2003, are still running Buster, they don't have the new SAN and 2001 fails to trust them.
We could change the puppet code for etcd to allow this use case, but since the code is shared by a lot of important clusters I'd just reimage all the etcd nodes as part of the upgrade to k8s 1.23.
Cookbook cookbooks.sre.ganeti.reimage started by elukey@cumin1001 for host ml-etcd2001.codfw.wmnet with OS bullseye executed with errors: