I typically prefer if we rebuild images from dockerfiles, using our base images. That gives us a tad more control over upgrading in case of a disaster security hole in e.g. alpine linux.

In T266893#6594832, @Joe wrote:

I typically prefer if we rebuild images from dockerfiles, using our base images. That gives us a tad more control over upgrading in case of a disaster security hole in e.g. alpine linux.

I've not looked at it in detail, but it seems that at least node, kube-controller and typha are released in scratch containers.
The node container, though does look pretty complicated and basically includes all of registry.access.redhat.com/ubi8/ubi-minimal:8.1.

What's in the box (release tarball):
Docker images

• calico/node
  • Looks like a scratch container, but actually copies / of a registry.access.redhat.com/ubi8/ubi-minimal:8.1 base plus modifications
  • Dockerfile is quite complex, building/copying specific versions of bpftool, iptables, bird. Some of them from other images like calico/bird:latest and calico/bpftool:v5.3-amd64
• calico/typha
  • Recommended for clusters with 50+ nodes (which I guess we target) and kubernetes datastore
  • scratch container, some config, binary and tini (typha is not designed to run as PID1)
• calico/cni
  • By default this is run together with calico/node in a daemonset
    • That just copies the binaries to the host FS. We can do that via debs as well.
  • scratch container with a bunch of binaries (networking and IPAM)
• calico/kube-controllers
  • scratch container with two binaries
• calico/dikastes
  • Not sure yet what this is, but it seems like a component to secure pod-to-pod communication (like envoy/istio sidecar?). So we probably don't need it.
• calico/pod2daemon-flexvol
  • Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes to communicate with Felix over the Policy Sync API.
  • So probably not needed when we don't need/use dikastes
• calico/flannel-migration-controller
  • No idea, not checked as name suggests we don't need it.

Binaries

• calicoctl
  • We will need a debian package for that to be installed on all nodes. Single go binary, so pretty easy.

So for new builds it looks as if we can go with:

• Import docker images
  • calico/node
  • calico/typha
  • calico/kube-controllers
• Build debs:
  • calicoctl (with calicoctl)
  • calico-cni (with cni-plugin and cni-ipam)

Investigation is needed on how we get the CRD manifests/RBAC roles etc to kubernetes:

• Kubernetes releases ship some calico manifests as addons (have not looked into what that exactly is)
• calico releases contain a lot of manifests itself

calico/node is the only more-than-slightly-worrisome thing here. for everything else we're probably ok using their builds for the time being. It's also true that as far as external images go, a redhat certified image is probably one of the most "secure" options we can find:

https://catalog.redhat.com/software/containers/ubi8/ubi-minimal/5c359a62bed8bd75a2c3fba8

I am starting to think we could just import their images into our registry, as they will live under the calico/ namespace anyways.

As for the manifests, if we need them, they should be in hellfile.d/admin I guess?

In T266893#6605376, @Joe wrote:

As for the manifests, if we need them, they should be in hellfile.d/admin I guess?

The thing with that is that it relies on other stuff to work. At least right now it requites tiller to be installed, DNS to function etc.
When deployed as cluster addon, we can bypass all this and have mandatory components of our stack deployed/reconceiled directly when the cluster is set up (shipping manifests via debian packages or generate via puppet).
While that sounds appealing to me, I'm not sure this is still the "correct" way. The upstream addons are called legacy for a couple of years now (https://github.com/kubernetes/kubernetes/commit/43276035734ba8a5914977b13da86ec2548fa745) ....

Change 640094 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/debs/calico@master] Initial release of upstream binary packages

https://gerrit.wikimedia.org/r/640094

Change 640095 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/debs/calico@master] Build a calico-images package

https://gerrit.wikimedia.org/r/640095

In T266893#6605404, @JMeybohm wrote:

When deployed as cluster addon, we can bypass all this and have mandatory components of our stack deployed/reconceiled directly when the cluster is set up (shipping manifests via debian packages or generate via puppet).
While that sounds appealing to me, I'm not sure this is still the "correct" way. The upstream addons are called legacy for a couple of years now (https://github.com/kubernetes/kubernetes/commit/43276035734ba8a5914977b13da86ec2548fa745) ....

I tried to track that down a bit and it seems like a relic from my GKE times now. I was unable to find many usages of addon manager outside of GKE (where is it still in use: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/manifests/kube-addon-manager.yaml). I still think it's kind of clever to be able to ship manifests via debian packages or maybe puppet code directly. This is probably nothing we want to change during a "regular" k8s upgrade...

Change 641445 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] aptrepo: add component for future calico packages

https://gerrit.wikimedia.org/r/641445

After discussing with @akosiaris we decided to keep building the calico-images package but only use it as kind of artifact and a way to get the images out of the pbuilder environment. After building, the package can be extracted and the images imported to the registry via a script added to the package.
While not ideal, it's an okay solution for now and I've documented it at: https://wikitech.wikimedia.org/wiki/Calico

Change 641445 merged by JMeybohm:
[operations/puppet@production] aptrepo: add component for future calico packages

https://gerrit.wikimedia.org/r/641445

Change 640094 merged by JMeybohm:
[operations/debs/calico@master] Initial release of upstream binary packages

https://gerrit.wikimedia.org/r/640094

Change 640095 merged by JMeybohm:
[operations/debs/calico@master] Build a calico-images package

https://gerrit.wikimedia.org/r/640095

Change 643274 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/debs/calico@master] Update to v3.17.0

https://gerrit.wikimedia.org/r/643274

Change 643274 merged by JMeybohm:
[operations/debs/calico@master] Update to v3.17.0

https://gerrit.wikimedia.org/r/643274

Change 645078 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] Remove calico::builder

https://gerrit.wikimedia.org/r/645078

Change 645078 merged by JMeybohm:
[operations/puppet@production] Remove calico::builder

https://gerrit.wikimedia.org/r/645078