Page MenuHomePhabricator

Build new kubernetes packages
Closed, ResolvedPublic

Description

We decided to build packages from the binary releases instead of compiling ourselves.
We also figured we need to update to k8s 1.16.x first, so we need packages for that.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 29 2020, 10:38 AM
JMeybohm triaged this task as High priority.Oct 29 2020, 10:45 AM

Change 637463 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] aptrepo: add component for future kubernetes packages

https://gerrit.wikimedia.org/r/637463

JMeybohm added a subscriber: Joe.Oct 30 2020, 1:20 PM

With Kubernetes 1.19.3 things changed a bit and we now need a docker version supporting the --platform flag for FROM.
The envoy builder host would support that but lacks enough space on /var/lib/docker to hold all the intermediate images used for build.

Talking with @Joe we thought it might be about time to switch to building debs from the binary releases instead going through the hurdles of the kubernetes build system (frequently). With the complexity of the build system itself, we don't really add any vetting to the build anyways.

Unfortunately there are no signed releases of k8s as of now (https://github.com/kubernetes/release/issues/914) so the best we could get is the binary tar balls plus sha512 from the official GCS bucket and package those.

Missing from tar balls are the man pages at least, which we would still need to build ourselves from the kubernetes sources (or leave them out).

With Kubernetes 1.19.3 things changed a bit and we now need a docker version supporting the --platform flag for FROM.
The envoy builder host would support that but lacks enough space on /var/lib/docker to hold all the intermediate images used for build.

Oh 1.19.3 tries to build for all architectures in 1 go? Ouch.

Talking with @Joe we thought it might be about time to switch to building debs from the binary releases instead going through the hurdles of the kubernetes build system (frequently). With the complexity of the build system itself, we don't really add any vetting to the build anyways.

That's not entirely true. Supply chain attacks can happen and they are far more difficult to detect in binaries than a git repo (even if we aren't the ones doing the detecting), so even if we don't do any vetting, we aren't subjected to some attack vectors. Now, whether that's worth it, it's an entirely different question. It quite possibly isn't by now.

Unfortunately there are no signed releases of k8s as of now (https://github.com/kubernetes/release/issues/914) so the best we could get is the binary tar balls plus sha512 from the official GCS bucket and package those.

So relying on e.g. https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#downloads-for-v11810 (from what I see the server binaries tar is enough as it contains whatever the node and client tars have) and some thin debian/rules to fetch/extract and put in the right place before building the deb so that we don't have to change our puppet code as well? That could work.

Missing from tar balls are the man pages at least, which we would still need to build ourselves from the kubernetes sources (or leave them out).

It probably is ok to just ditch them all together. The man pages were added at the request of WMCS and they no longer rely on our packages.

Unfortunately there are no signed releases of k8s as of now (https://github.com/kubernetes/release/issues/914) so the best we could get is the binary tar balls plus sha512 from the official GCS bucket and package those.

So relying on e.g. https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#downloads-for-v11810 (from what I see the server binaries tar is enough as it contains whatever the node and client tars have) and some thin debian/rules to fetch/extract and put in the right place before building the deb so that we don't have to change our puppet code as well? That could work.

Indeed. It's probably a even slightly better to fetch the download link and expected hash from git's CHANGELOG-*.md rater than using https://dl.k8s.io/v1.18.10/kubernetes-server-linux-arm64.tar.gz.sha512 directly.

Unfortunately, the current version of calico is not tested against kubernetes 1.19.x yet (https://docs.projectcalico.org/getting-started/kubernetes/requirements#kubernetes-requirements).
While it might still work, we should stick to a tested/supported version (at least for production) and there is no release timeline I found for k8s 1.19 support in calico. So no idea if we get a calico version supporting 1.19 anytime "soon".

An additional alternative could be to switch to a build without docker like the debian upspream does. That would maybe require us to backport golang-1.15 but would remove the build-dependency to docker with all it's space and network requirements.

Change 638115 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/debs/kubernetes@future] Package binary kubernetes releases

https://gerrit.wikimedia.org/r/638115

JMeybohm renamed this task from Build kubernetes 1.19 to Build new kubernetes packages.Nov 3 2020, 3:23 PM
JMeybohm updated the task description. (Show Details)

Change 638595 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/debs/kubernetes@master] Update to 1.16.15

https://gerrit.wikimedia.org/r/638595

Change 638595 abandoned by JMeybohm:
[operations/debs/kubernetes@master] Update to 1.16.15

Reason:
This was submitted for the wrong branch and move is disabled.

https://gerrit.wikimedia.org/r/638595

Change 639192 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/debs/kubernetes@future] Update to 1.16.15

https://gerrit.wikimedia.org/r/639192

Change 637463 merged by JMeybohm:
[operations/puppet@production] aptrepo: add component for future kubernetes packages

https://gerrit.wikimedia.org/r/637463

Change 638115 merged by JMeybohm:
[operations/debs/kubernetes@future] Package binary kubernetes releases

https://gerrit.wikimedia.org/r/638115

Change 639192 merged by JMeybohm:
[operations/debs/kubernetes@future] Update to 1.16.15

https://gerrit.wikimedia.org/r/639192

Change 639604 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/debs/kubernetes@future] Make the build pdebuild compatible

https://gerrit.wikimedia.org/r/639604

Change 639604 merged by JMeybohm:
[operations/debs/kubernetes@future] Make the build pdebuild compatible

https://gerrit.wikimedia.org/r/639604

JMeybohm closed this task as Resolved.Nov 5 2020, 6:51 PM

1.16.15 released to stretch-wikimedia component/kubernetes-future