Page MenuHomePhabricator
Create Task
Maniphest T267653

Refactor calico deploy strategy
Closed, ResolvedPublic
Actions

Description

Our current deploy strategy is:

  • Deploy via debian packages (calicoctl, calico-cni)
  • Deploy calico-node (as docker container, launched by systemd) via puppet (modules/calico/manifests/init.pp)
  • Deploy calico-policy-controller via helmfile.d/admin (internal_charts/wmf-calico-policy-controller/)

With upcoming k8s and calico updates it would be nice to have this less scattered, like:

  • Deploy via debian packages (calicoctl, calico-cni)
  • Deploy CDRs, RBAC, calico-node, typha, calico-policy-controller via a helm chart and helmfile.d/admin

Unfortunately this is not easily possible with helm2 & tiller as there is a catch-22 in accessing the k8s API from tiller prior to having the policy-controller running. Also, deploying calico-node as daemonset would require us to run the pod privileged which we currently prohibit globally I suppose.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

JMeybohm created this task.Nov 10 2020, 4:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 10 2020, 4:20 PM
JMeybohm triaged this task as High priority.Nov 10 2020, 4:20 PM
JMeybohm added a comment.Nov 10 2020, 4:22 PM

To solve the catch-22 we could deploy the to-be calico helm chart via helm3. Which would require us to invest into helm3 integration earlier than we hoped for (at least for the helmfile.d/admin part).

gerritbot added a comment.Nov 17 2020, 7:19 PM

Change 641511 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/debs/kubernetes@future] Add kubernetes-addon-manager

https://gerrit.wikimedia.org/r/641511

gerritbot added a project: Patch-For-Review.Nov 17 2020, 7:19 PM
gerritbot added a comment.Nov 18 2020, 11:19 AM

Change 641711 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] kubernetes: Add profile to install addon-manager on masters

https://gerrit.wikimedia.org/r/641711

JMeybohm added a comment.EditedNov 18 2020, 5:20 PM

@akosiaris and me discussed this further and we initially decided to give the kubernetes addon-manager a try for rolling out calico components to a freshly bootstrapped cluster. I created a binary package from kubernetes source package that sets up the addon-manager and added corresponding puppet code.

While the very static manifests like CRDs and RBAC rules could easily be packaged (from calico source) and installed via addon-manager, the trouble starts with the calico-node DaemonSet and typha Deployment:

  • In "Calico the hard way" certificates are used to authenticate communication between calico-node and typha. If we want to do so as well, we would need to integrate them somehow which would potentially duplicate puppet code we have for helmfile.d already.
  • We need to override the ENV variables for the kubernetes service (kubernetes API) for all calico specific containers because of the IP SAN limitation of puppet CA. So we would need to template the manifests again, which is a bad thing to to in puppet (and we already have helm to fight against in this area).

Another open question is if we want to switch from running calico-node manually (docker run via systemd) to the recommended way of running it as a Daemonset within the cluster.
Personally I would prefer to run it the recommended way as it seems more straight forward and the benefit of separating it from k8s gets even smaller when we use the kubernetes datastore. The downside of this is that calico-node needs to run privileged and we have privileged containers disabled. We also lack proper limitations regarding privileged containers in production, see: T228967.

MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.Nov 18 2020, 5:21 PM
JMeybohm mentioned this in T268434: Refactor our helmfile.d dir structure for admin.Nov 23 2020, 7:47 AM
JMeybohm added a comment.Nov 23 2020, 2:01 PM

I did a bit of testing and it looks as if it is totally possible to switch helmfile.d/admin to use helm3 and get rid of tiller there (e.g. catch-22) while keeping helm2 + tiller for helmfile.d/services for now (see T268434).

Taking that route instead of addon-manager would also keep the benefit of having everything "inside" the cluster deployed via deployment-charts repo.

gerritbot added a comment.Nov 24 2020, 2:42 PM

Change 641711 abandoned by JMeybohm:
[operations/puppet@production] kubernetes: Add profile to install addon-manager on masters

Reason:

https://gerrit.wikimedia.org/r/641711

gerritbot added a comment.Nov 24 2020, 2:42 PM

Change 641511 abandoned by JMeybohm:
[operations/debs/kubernetes@future] Add kubernetes-addon-manager

Reason:

https://gerrit.wikimedia.org/r/641511

gerritbot added a comment.Nov 27 2020, 5:08 PM

Change 643974 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] Add charts for calico and calico-crds

https://gerrit.wikimedia.org/r/643974

gerritbot added a comment.Dec 1 2020, 9:15 AM

Change 644462 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] Add calico helm chart

https://gerrit.wikimedia.org/r/644462

gerritbot added a comment.Dec 2 2020, 8:32 AM

Change 643974 merged by jenkins-bot:
[operations/deployment-charts@master] Add helm chart for calico CRDs

https://gerrit.wikimedia.org/r/643974

JMeybohm claimed this task.Dec 3 2020, 2:15 PM
gerritbot added a comment.Dec 4 2020, 11:27 AM

Change 645317 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] Split out RBAC rules and service accoutns for typa and CNI

https://gerrit.wikimedia.org/r/645317

gerritbot added a comment.Dec 4 2020, 6:08 PM

Change 645408 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] calico: Bind the calico-cni Role to the calico-cni user

https://gerrit.wikimedia.org/r/645408

gerritbot added a comment.Dec 4 2020, 6:16 PM

Change 645412 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[labs/private@master] k8s_infrastructure_users: add calico-cni

https://gerrit.wikimedia.org/r/645412

gerritbot added a comment.Dec 4 2020, 6:33 PM

Change 645412 merged by JMeybohm:
[labs/private@master] k8s_infrastructure_users: add calico-cni

https://gerrit.wikimedia.org/r/645412

JMeybohm mentioned this in rLPRI621cfa62e8bb: k8s_infrastructure_users: add calico-cni.Dec 4 2020, 6:33 PM
JMeybohm mentioned this in rLPRI0e40d1ecdba0: k8s_infrastructure_users: add calicoctl.Dec 4 2020, 7:21 PM
gerritbot added a comment.Dec 4 2020, 7:32 PM

Change 645417 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] calico: Add support for calico 3.x with kubernetes datastore

https://gerrit.wikimedia.org/r/645417

gerritbot added a comment.Dec 4 2020, 8:13 PM

Change 645426 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[labs/private@master] Add tokens for calico::kubernetes cni and ctl

https://gerrit.wikimedia.org/r/645426

gerritbot added a comment.Dec 4 2020, 8:14 PM

Change 645426 merged by JMeybohm:
[labs/private@master] Add tokens for calico::kubernetes cni and ctl

https://gerrit.wikimedia.org/r/645426

JMeybohm mentioned this in rLPRI09d85e7162a1: Add tokens for calico::kubernetes cni and ctl.Dec 4 2020, 8:16 PM
gerritbot added a comment.Dec 7 2020, 3:37 PM

Change 646740 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] calico: Remove calico/data

https://gerrit.wikimedia.org/r/646740

gerritbot added a comment.Dec 7 2020, 5:06 PM

Change 646740 merged by Alexandros Kosiaris:
[operations/puppet@production] calico: Remove calico/data

https://gerrit.wikimedia.org/r/646740

gerritbot added a comment.Dec 9 2020, 10:57 AM

Change 644462 merged by jenkins-bot:
[operations/deployment-charts@master] Add calico helm chart

https://gerrit.wikimedia.org/r/644462

gerritbot added a comment.Dec 9 2020, 10:57 AM

Change 645317 merged by jenkins-bot:
[operations/deployment-charts@master] Split out RBAC rules and service accounts for typha and CNI

https://gerrit.wikimedia.org/r/645317

gerritbot added a comment.Dec 9 2020, 10:57 AM

Change 645408 merged by jenkins-bot:
[operations/deployment-charts@master] calico: Bind the calico-cni Role to the calico-cni user

https://gerrit.wikimedia.org/r/645408

JMeybohm added a comment.Dec 9 2020, 11:14 AM

The new calico chart is merged, thanks @akosiaris

What is missing currently is a proper RoleBinding for the calicoctl user as I was not sure yet what permissions he's going to need.
We should be not using the tool for changing calico config, that's to be done via the helm chart now. But we will want to keep the analyze functionality intact. Could not find any docs on that by know so we will maybe just have to figure it out when we have a node in staging-codfw

akosiaris added a comment.Dec 9 2020, 3:06 PM
In T267653#6678721, @JMeybohm wrote:

The new calico chart is merged, thanks @akosiaris

What is missing currently is a proper RoleBinding for the calicoctl user as I was not sure yet what permissions he's going to need.
We should be not using the tool for changing calico config, that's to be done via the helm chart now. But we will want to keep the analyze functionality intact. Could not find any docs on that by know so we will maybe just have to figure it out when we have a node in staging-codfw

That's fine, but since the tool is also used to run some diagnostics and will only be run from the kubernetes nodes by an SRE, it's probably ok to use the network-admin role that is defined in https://docs.projectcalico.org/getting-started/kubernetes/hardway/end-user-rbac

JMeybohm added a comment.Dec 9 2020, 6:08 PM
In T267653#6679363, @akosiaris wrote:
In T267653#6678721, @JMeybohm wrote:

The new calico chart is merged, thanks @akosiaris

What is missing currently is a proper RoleBinding for the calicoctl user as I was not sure yet what permissions he's going to need.
We should be not using the tool for changing calico config, that's to be done via the helm chart now. But we will want to keep the analyze functionality intact. Could not find any docs on that by know so we will maybe just have to figure it out when we have a node in staging-codfw

That's fine, but since the tool is also used to run some diagnostics and will only be run from the kubernetes nodes by an SRE, it's probably ok to use the network-admin role that is defined in https://docs.projectcalico.org/getting-started/kubernetes/hardway/end-user-rbac

Yeah. I thought it might be smart to create a read-only role for that (if possible) to prevent (untracked) changes to the config by human error. Let's see when we get there.

gerritbot added a comment.Dec 10 2020, 3:45 PM

Change 645417 merged by JMeybohm:
[operations/puppet@production] calico: Add support for calico 3.x with kubernetes datastore

https://gerrit.wikimedia.org/r/645417

JMeybohm added a comment.Dec 10 2020, 3:54 PM

[puppet-private] (487bdca0) (jayme) Add calicoctl and calico-cni kubernetes users

gerritbot added a comment.Dec 11 2020, 9:52 AM

Change 648143 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] Add calico releases to admin_ng helmfile

https://gerrit.wikimedia.org/r/648143

gerritbot added a comment.Dec 11 2020, 10:03 AM

Change 648143 merged by jenkins-bot:
[operations/deployment-charts@master] Add calico releases to admin_ng helmfile

https://gerrit.wikimedia.org/r/648143

gerritbot added a comment.Dec 11 2020, 2:32 PM

Change 648245 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] helmfile needs: parameter requires a release namespace

https://gerrit.wikimedia.org/r/648245

gerritbot added a comment.Dec 11 2020, 2:36 PM

Change 648245 merged by jenkins-bot:
[operations/deployment-charts@master] helmfile needs: parameter requires a release namespace

https://gerrit.wikimedia.org/r/648245

JMeybohm closed this task as Resolved.Dec 18 2020, 1:56 PM

This is done with calico deployed now via puppet (CNI plugins and calicoctl) as well as helm3 (helmfile.d/admin_ng).
Everything is under version control and there are no catch-22's anymore during cluster bootstrapping.

JMeybohm reopened this task as Open.Jan 28 2021, 2:22 PM

Looks like we're missing an RBAC rule for calico-node:

2021-01-28 14:17:19.139 [INFO][31] tunnel-ip-allocator/ipam.go 1325: Releasing all IPs with handle 'wireguard-tunnel-addr-kubestage2002.codfw.wmnet'
2021-01-28 14:17:19.142 [FATAL][31] tunnel-ip-allocator/allocateip.go 537: Error releasing address by handle Handle="wireguard-tunnel-addr-kubestage2002.codfw.wmnet" IP="" error=connection is unauthorized: ipamhandles.crd.projectcalico.org "wireguard-tunnel-addr-kubestage2002.codfw.wmnet" is forbidden: User "system:serviceaccount:kube-system:calico-node" cannot get resource "ipamhandles" in API group "crd.projectcalico.org" at the cluster scope type="wireguardTunnelAddress"
Calico node failed to start
gerritbot added a comment.Jan 28 2021, 2:44 PM

Change 659259 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] calico-node needs read access to ipamhandles resource

https://gerrit.wikimedia.org/r/659259

gerritbot added a comment.Jan 28 2021, 3:02 PM

Change 659259 merged by jenkins-bot:
[operations/deployment-charts@master] calico-node needs read access to ipamhandles resource

https://gerrit.wikimedia.org/r/659259

JMeybohm closed this task as Resolved.Jan 28 2021, 3:19 PM

fixed with helm chart version: 0.1.9

JMeybohm reopened this task as Open.Jan 30 2021, 3:51 PM

2021-01-30 15:45:27.743 [ERROR][8] lookup.go 63: Failed to get Typha endpoint from Kubernetes error=endpoints "calico-typha" is forbidden: User "system:serviceaccount:kube-system:calico-typha" cannot get resource "endpoints" in API group "" in the namespace "kube-system"

gerritbot added a comment.Jan 30 2021, 3:57 PM

Change 660399 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] calico: Typha needs to get endpoints to discover it's instances

https://gerrit.wikimedia.org/r/660399

gerritbot added a comment.Feb 16 2021, 3:22 PM

Change 660399 merged by jenkins-bot:
[operations/deployment-charts@master] calico: Typha needs to get endpoints to discover it's instances

https://gerrit.wikimedia.org/r/660399

JMeybohm closed this task as Resolved.Feb 22 2021, 3:29 PM

I think it's safe to say this is done now with the admin_ng using helm3 and the updates to puppet.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL