Page MenuHomePhabricator
Create Task
Maniphest T292238

Figure out certificate generation for admission webhooks before we lose the certificates/v1beta1
Closed, ResolvedPublic
Actions

Description

We are currently generating certificates for the admission webhooks in Toolforge using the v1beta1 certificates API because it allows the unknown-legacy signer. That is not valid in the certificates/v1 API, and v1beta1 gets dropped in 1.22.

We should not upgrade to this level until this is resolved. (noticed in T289390) and confirmed as an issue in upstream Github issues. A new signer may be added by the upstream project for https://github.com/kubernetes/kubernetes/issues/63732, but we'd need that to be used first. We just need a trusted TLS cert (that the control plane pod trusts) to run those controllers.

webhooks:

  • registry-admission
  • ingress-admission
  • volume-admission
  • buildpack-admission

internal stuff:

  • metrics-server

external stuff:

  • jobs-api

Details

SubjectRepoBranchLines +/-
deployment: Simplify helm chart + add cert-manager certslabs/tools/registry-admission-webhookmaster+204 -542
kubeadm: update wmcs-k8s-get-cert for certificates/v1operations/puppetproduction+14 -16
kubeadm: drop wmcs-k8s-secret-for-certoperations/puppetproduction+1 -134
update to remove k8s side cacloud/toolforge/ingress-admission-controllermaster+52 -96
deployment: Use cert-manager to generate webhook certificatescloud/toolforge/volume-admission-controllermain+34 -113
Use cert-manager to generate webhook certificatescloud/toolforge/volume-admission-controllermain+71 -212
deployment: Use cert-manager to generate webhook certificatescloud/toolforge/ingress-admission-controllermaster+31 -107
toolsbeta: Exclude more namespaceslabs/tools/registry-admission-webhookmaster+10 -1
Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
k8s: refresh wmcs-k8s-get-cert.sh scriptrepos/cloud/toolforge/lima-kilo!23aborreroarturo-k8s-refresh-wmcs-k8s-gemain
Update metrics-server and overhaul its deploymentrepos/cloud/toolforge/wmcs-k8s-metrics!2taaviupdate-metrics-servermain
Customize query in GitLab

Related Objects
Search...

Event Timeline

• Bstorm created this task.Sep 30 2021, 6:57 PM
• Bstorm added a comment.Sep 30 2021, 7:03 PM

The overall issue is that existing certificates/v1 signers don't include a pod serving signer. You cannot make it use the kubelet serving signer (which is the closest you can come). This should not be an issue for maintain-kubeusers since there's a signer for that use case. The certs that signer makes cannot be serving certs, though. They can only be used for client auth.

Some docs on the signers is https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers

They provide no API to create a new signer. You have to put in a pull request to do that :). There is an issue where that may very well be done: https://github.com/kubernetes/kubernetes/issues/63732

We don't need auto-approval, but it would be nice!

• Bstorm mentioned this in T286856: Upgrade Toolforge Kubernetes to latest 1.22.Sep 30 2021, 7:05 PM
taavi added a comment.EditedOct 6 2021, 6:44 PM

Alternative solution would be to use certificates signed by something else than Kubernetes. Bonus points if that is combined with full creation + renewal automation. https://cert-manager.io/docs/concepts/ca-injector/ is likely a good alternative.

taavi mentioned this in T291915: toolforge: automate how we deploy custom k8s components.Oct 6 2021, 6:44 PM
taavi added a comment.Oct 9 2021, 10:53 AM

I took a look into using cert-manager for webhook certs today, and my only concern is that our webhook controllers don't pick up certificate changes from disk. The cert-manager documentation suggests something like https://github.com/wave-k8s/wave, I'm a bit worried about additional complexity but a similar feature could be useful elsewhere too.

gerritbot added a comment.Oct 12 2021, 7:33 AM

Change 730144 had a related patch set uploaded (by Majavah; author: Majavah):

[labs/tools/registry-admission-webhook@master] toolsbeta: Exclude more namespaces

https://gerrit.wikimedia.org/r/730144

gerritbot added a project: Patch-For-Review.Oct 12 2021, 7:33 AM
gerritbot added a comment.Oct 16 2021, 7:15 AM

Change 730144 merged by jenkins-bot:

[labs/tools/registry-admission-webhook@master] toolsbeta: Exclude more namespaces

https://gerrit.wikimedia.org/r/730144

taavi mentioned this in rLTRAf75c3faed562: toolsbeta: Exclude more namespaces.Oct 16 2021, 7:33 AM
Stashbot added a comment.Oct 16 2021, 7:47 AM

Mentioned in SAL (#wikimedia-cloud) [2021-10-16T07:47:36Z] <majavah> deployed cert-manager and wave as a test for automating T292238

taavi added a comment.EditedOct 16 2021, 7:54 AM

I deployed cert-manager and wave to toolsbeta using the following commands as a root on a k8s-control node:

helm repo add jetstack https://charts.jetstack.io
helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update


kubectl create -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: reloader
  labels:
    name: reloader
---
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager
  labels:
    name: cert-manager
EOF

helm install -n reloader reloader stakater/reloader --version v0.0.109
helm install -n cert-manager cert-manager jetstack/cert-manager --version v1.5.4 --set installCRDs=true --set global.podSecurityPolicy.enabled=true

kubectl create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: reloader-psp
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
- kind: ServiceAccount
  name: reloader-reloader
  namespace: reloader
EOF

I am a bit concerned that the wave chart is using deprecated apis (notably rbac.authorization.k8s.io/v1beta1), I'll definitely want to test wave with k8s 1.22 before considering it a production-grade dependency we want to rely on.

Maintenance_bot removed a project: Patch-For-Review.Oct 16 2021, 8:10 AM
rook subscribed.Oct 27 2021, 7:40 PM
gerritbot added a comment.Oct 29 2021, 6:11 PM

Change 735681 had a related patch set uploaded (by Michael DiPietro; author: Michael DiPietro):

[cloud/toolforge/ingress-admission-controller@master] update to remove k8s side ca

https://gerrit.wikimedia.org/r/735681

gerritbot added a project: Patch-For-Review.Oct 29 2021, 6:11 PM
rook added a comment.Oct 29 2021, 6:16 PM

The code in https://gerrit.wikimedia.org/r/c/cloud/toolforge/ingress-admission-controller/+/735681 does away with CertificateSigningRequest. Is there a reason that this approach would not be reasonable?

taavi added a comment.Nov 6 2021, 1:14 PM
In T292238#7468725, @mdipietro wrote:

The code in https://gerrit.wikimedia.org/r/c/cloud/toolforge/ingress-admission-controller/+/735681 does away with CertificateSigningRequest. Is there a reason that this approach would not be reasonable?

That approach looks good if we want to continue using a roughly similar process as what we've done before. I'm still going to at least see if cert-manager would work, since that'd be one less manual admin action we have to do sometimes.

gerritbot added a comment.Nov 6 2021, 2:22 PM

Change 737171 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/volume-admission-controller@main] Use cert-manager to generate webhook certificates

https://gerrit.wikimedia.org/r/737171

taavi added a comment.Mar 16 2022, 3:23 PM
In T292238#7433645, @Majavah wrote:

I am a bit concerned that the wave chart is using deprecated apis (notably rbac.authorization.k8s.io/v1beta1), I'll definitely want to test wave with k8s 1.22 before considering it a production-grade dependency we want to rely on.

https://github.com/stakater/Reloader is actively maintained and does the same thing.

Stashbot added a comment.Mar 16 2022, 3:23 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-16T15:23:53Z] <taavi> deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a T292238 test to toolsbeta

taavi added a comment.Mar 16 2022, 3:36 PM
In T292238#7487308, @gerritbot wrote:

Change 737171 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/volume-admission-controller@main] Use cert-manager to generate webhook certificates

https://gerrit.wikimedia.org/r/737171

I deployed this to toolsbeta and it works fine.

taavi mentioned this in T308402: toolforge: Refresh certs that are not controlled by kubeadm (mid 2022 edition).May 15 2022, 12:31 PM
dcaro mentioned this in T309782: toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition).Jun 2 2022, 12:42 PM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:46 PM
fnegri moved this task from Kanban to Inbox on the cloud-services-team board.
aborrero subscribed.Jan 26 2023, 10:22 AM
taavi added a subtask: T329453: Deploy cert-manager to Toolforge.Feb 12 2023, 4:47 PM
gerritbot added a comment.Feb 14 2023, 8:43 PM

Change 889225 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/ingress-admission-controller@master] deployment: Use cert-manager to generate webhook certificates

https://gerrit.wikimedia.org/r/889225

taavi closed subtask T329453: Deploy cert-manager to Toolforge as Resolved.Feb 15 2023, 10:15 AM
taavi updated the task description. (Show Details)Feb 15 2023, 11:32 AM
gerritbot added a comment.Feb 16 2023, 12:29 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager/-/merge_requests/1

reloaded: deploy in the cert-manager namespace

gerritbot added a comment.Feb 16 2023, 1:42 PM

Change 889225 merged by jenkins-bot:

[cloud/toolforge/ingress-admission-controller@master] deployment: Use cert-manager to generate webhook certificates

https://gerrit.wikimedia.org/r/889225

gerritbot added a comment.Feb 16 2023, 3:25 PM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager/-/merge_requests/1

reloader: deploy in the cert-manager namespace

taavi updated the task description. (Show Details)Feb 16 2023, 4:08 PM
gerritbot added a comment.Feb 16 2023, 4:12 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics/-/merge_requests/2

Update metrics-server and overhaul its deployment

gerritbot added a comment.Feb 17 2023, 10:55 AM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics/-/merge_requests/2

Update metrics-server and overhaul its deployment

taavi updated the task description. (Show Details)Feb 17 2023, 11:23 AM
gerritbot added a comment.Feb 17 2023, 9:03 PM

Change 890027 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/volume-admission-controller@main] deployment: Use cert-manager to generate webhook certificates

https://gerrit.wikimedia.org/r/890027

gerritbot added a comment.Feb 18 2023, 6:56 PM

Change 737171 abandoned by Majavah:

[cloud/toolforge/volume-admission-controller@main] Use cert-manager to generate webhook certificates

Reason:

in favour of https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/890027/

https://gerrit.wikimedia.org/r/737171

gerritbot added a comment.Feb 20 2023, 11:02 AM

Change 890027 merged by jenkins-bot:

[cloud/toolforge/volume-admission-controller@main] deployment: Use cert-manager to generate webhook certificates

https://gerrit.wikimedia.org/r/890027

taavi mentioned this in rCTVA7fd13ac8ecd6: deployment: Use cert-manager to generate webhook certificates.Feb 20 2023, 11:03 AM
Stashbot added a comment.Feb 20 2023, 11:24 AM

Mentioned in SAL (#wikimedia-cloud) [2023-02-20T11:24:08Z] <taavi> redeploy volume-admission with helm and cert-manager certificates T329530 T292238

Stashbot mentioned this in T329530: Convert all Toolforge custom components to standardized Helm based deployment.Feb 20 2023, 11:24 AM
taavi updated the task description. (Show Details)Feb 20 2023, 11:24 AM
gerritbot added a comment.Feb 20 2023, 3:20 PM

Change 890468 had a related patch set uploaded (by Majavah; author: Majavah):

[labs/tools/registry-admission-webhook@master] deployment: Simplify helm chart + add cert-manager certs

https://gerrit.wikimedia.org/r/890468

gerritbot added a comment.Feb 20 2023, 7:23 PM

Change 735681 abandoned by Majavah:

[cloud/toolforge/ingress-admission-controller@master] update to remove k8s side ca

Reason:

in favour of https://gerrit.wikimedia.org/r/c/cloud/toolforge/ingress-admission-controller/+/889225/

https://gerrit.wikimedia.org/r/735681

gerritbot added a comment.Feb 20 2023, 7:24 PM

Change 890501 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] kubeadm: drop wmcs-k8s-secret-for-cert

https://gerrit.wikimedia.org/r/890501

gerritbot added a comment.Feb 20 2023, 7:24 PM

Change 890502 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] kubeadm: update wmcs-k8s-get-cert for certificates/v1

https://gerrit.wikimedia.org/r/890502

taavi updated the task description. (Show Details)Feb 23 2023, 9:40 AM
gerritbot added a comment.Mar 2 2023, 11:25 AM

Change 890501 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] kubeadm: drop wmcs-k8s-secret-for-cert

https://gerrit.wikimedia.org/r/890501

gerritbot added a comment.Mar 3 2023, 12:30 PM

Change 890502 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] kubeadm: update wmcs-k8s-get-cert for certificates/v1

https://gerrit.wikimedia.org/r/890502

gerritbot added a comment.Mar 3 2023, 12:46 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/23

k8s: refresh wmcs-k8s-get-cert.sh script

taavi updated the task description. (Show Details)Mar 3 2023, 1:38 PM
gerritbot added a comment.Mar 5 2023, 3:33 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/23

k8s: refresh wmcs-k8s-get-cert.sh script

gerritbot added a comment.Mar 6 2023, 12:28 PM

Change 890468 merged by jenkins-bot:

[labs/tools/registry-admission-webhook@master] deployment: Simplify helm chart + add cert-manager certs

https://gerrit.wikimedia.org/r/890468

taavi mentioned this in rLTRAe916feec7ac1: deployment: Simplify helm chart + add cert-manager certs.Mar 6 2023, 12:29 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 6 2023, 12:31 PM
taavi closed this task as Resolved.Mar 6 2023, 12:50 PM
taavi claimed this task.
taavi updated the task description. (Show Details)
aborrero mentioned this in T333059: Spread Toolforge tools to multiple Kubernetes clusters.Apr 10 2023, 3:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL