After updating all clusters to 1.23 we should get rid of the infrastructure_users tokens in favor of client certificates wherever possible. This will auto-fix T290963: Drop the use of nonexisting groups in kubernetes infrastructure_users.
There might be some more gems here, I already know of:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | JMeybohm | T307943 Update Kubernetes clusters to v1.23 | |||
| Open | JMeybohm | T328291 Post Kubernetes v1.23 cleanup | |||
| Open | None | T325268 Migrate use of infrastructure_users tokens to client certificates |
Promehteus 2.24.1 (bullseye) does not support client cert auth for kubernetes_sd, we would need to have 2.33.5 from bullseye-backports
We build our own Prometheus debs anyway since Debian doesn't ship with kubernetes_sd support. We're using client cert auth on Toolforge and it works fine.
Change 897365 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/puppet@production] calico/kubernetes: Replace istio_cni_token with client cert
Change 897361 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/puppet@production] calico/kubernetes: Replace calico cni and ctl tokens with client certs
Mentioned in SAL (#wikimedia-operations) [2023-03-14T09:56:43Z] <jayme> disabling puppet on P:calico::kubernetes for T325268
Change 897365 merged by JMeybohm:
[operations/puppet@production] calico/kubernetes: Replace istio_cni_token with client cert
Change 897361 merged by JMeybohm:
[operations/puppet@production] calico/kubernetes: Replace calico cni and ctl tokens with client certs
Mentioned in SAL (#wikimedia-operations) [2023-03-14T10:15:45Z] <jayme> enabling puppet on P:calico::kubernetes for T325268