Change Details

Currently it's not possible to address Kubernets API servers easily from Calico NetworkPolicy, but starting with T287443 we now need to. We have a hack in place to allow namespace-tillers to connect to them, but that is far from ideal. The reason behind this is that Calico does not model the Endpoints of the k8s service `kubernetes.default.svc.cluster.local` as WorkloadEndpoints as they are not backed by Pods. When running kubernetes API servers as nodes, we could potentially select them in NetworkPolicies directly, but for now I would propose adding a [[ https://docs.projectcalico.org/reference/resources/globalnetworkset | GlobalNetworkSet ]] per cluster with the correct IPs. The downside is that we'd need to deploy that via helmfile.d, so it's an extra step when we add/replace API servers.

Currently it's not possible to address Kubernets API servers easily from NetworkPolicy, but starting with T287443 we now need to. We have a hack in place to allow namespace-tillers to connect to them, but that is far from ideal. The reason behind this is that Calico does not model the Endpoints of the k8s service `kubernetes.default.svc.cluster.local` as WorkloadEndpoints as they are not backed by Pods. When running kubernetes API servers as nodes, we could potentially select them in NetworkPolicies directly, but for now I would propose adding a [[ https://docs.projectcalico.org/reference/resources/globalnetworkset | GlobalNetworkSet ]] per cluster with the correct IPs. The downside is that we'd need to deploy that via helmfile.d, so it's an extra step when we add/replace API servers. @elukey has added the ml master IPs to `helmfile.d/admin_ng/values/*/values.yaml` to cure the immediate pain. To have them auto-generated I thought it might be possible to read them from `service::catalog`, but: * that does not contain staging (we could add staging masters as non-LVS services I guess) * that does not contain IPv6 addresses * there is no way to know which service entry is a kubermaster one (`lvs.conftool.service == kubemaster` might be an option, but only for LVS services)

Currently it's not possible to address Kubernets API servers easily from Calico NetworkPolicy, but starting with T287443 we now need to. We have a hack in place to allow namespace-tillers to connect to them, but that is far from ideal. The reason behind this is that Calico does not model the Endpoints of the k8s service `kubernetes.default.svc.cluster.local` as WorkloadEndpoints as they are not backed by Pods. When running kubernetes API servers as nodes, we could potentially select them in NetworkPolicies directly, but for now I would propose adding a [[ https://docs.projectcalico.org/reference/resources/globalnetworkset | GlobalNetworkSet ]] per cluster with the correct IPs. The downside is that we'd need to deploy that via helmfile.d, so it's an extra step when we add/replace API servers. @elukey has added the ml master IPs to `helmfile.d/admin_ng/values/*/values.yaml` to cure the immediate pain. To have them auto-generated I thought it might be possible to read them from `service::catalog`, but: * that does not contain staging (we could add staging masters as non-LVS services I guess) * that does not contain IPv6 addresses * there is no way to know which service entry is a kubermaster one (`lvs.conftool.service == kubemaster` might be an option, but only for LVS services)