Page MenuHomePhabricator

Provision TLS certificates for k8s services in istio-system namespace
Closed, ResolvedPublic

Description

With cert-manager & cfss-issuer set up (T294560) we're able to automatically create and refresh TLS certificates. The Isio-Ingressgateway (T290966) however requires those certificates to be created in the istio-system namespace.

There are multiple ways we could achieve that:

1. Allow deploy users to write Certificate objects to istio-system namespace

I think this is a really bad idea. We could potentially limit the scope to only create but that would lead to weird helm issues when destroying releases etc. I've not thought about this option very much.

2. Syncing secrets (with kubed)

The cert-manager recommendation on how to solve this is an additional K8s controller (kubed) that synchronizes Secret objects between Namespaces. That way, the creation of the Certificate object could be part of the services helm chart and the resulting TLS Secret (in the services namespace) is then synchronized to istio-system. However, this requires yet another tool to run in the cluster, docker images to be build etc.

3. Create Certificate objects during Namespace creation (in admin_ng)

We could provision default Certificate objects during creation of Namespace objects in admin_ng. The hostnames for services are known upfront anyways and I suspect them not to change very much over time. So it seems link an okay trade off to have them provisioned by SRE instead of the deployer/service.
The only downsides I see here are:

  • We have to reference the hosts a service is reachable at twice, in admin_ng (for the Certificate) and at the helm chart of the service.
  • We potentially create the "same" certificate twice in the future. One for the Istio-Ingressgateway and one for the service itself (for service-proxy, TLS terminating envoy).

I've decided to go with option 3. for now as it is the easiest to implement and does not requite any extra tooling.

Event Timeline

Change 737975 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] dmin_ng: Create Certificates for ingressgateway

https://gerrit.wikimedia.org/r/737975

Change 737975 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: Create Certificates for ingressgateway

https://gerrit.wikimedia.org/r/737975

3. Create Certificate objects during Namespace creation (in admin_ng) it is