Page MenuHomePhabricator

Package/copy kubeadm, kubelet, docker-ce and kubectl to Toolforge Aptly or Reprepro
Closed, ResolvedPublic

Description

These packages need to end up in toolforge's aptly so we can install the upgraded k8s at the version we want. All packages work fine with stretch in my tests. Kubernetes' core repos no longer support Debian directly, and instead they use Xenial as a proxy (Ubuntu is the reference platform for Kubernetes).

The apt repo they live in is https://apt.kubernetes.io/ kubernetes-xenial main

hi  kubeadm                           1.13.1-00                      amd64        Kubernetes Cluster Bootstrapping Tool
hi  kubectl                           1.13.1-00                      amd64        Kubernetes Command Line Tool
hi  kubelet                           1.13.1-00                      amd64        Kubernetes Node Agent
ii  kubernetes-cni                    0.6.0-00                       amd64        Kubernetes CNI

As a dependency, they will also need:

ii  docker-ce                         5:18.09.0~3-0~debian-stretch   amd64        Docker: the open-source application container engine
ii  docker-ce-cli                     5:18.09.0~3-0~debian-stretch   amd64        Docker CLI: the open-source application container engine

From https://download.docker.com/linux/debian stretch/stable

This is a core task blocking progress on our k8s upgrade. We should be able to use straight package copies effectively, since our customizations of k8s shouldn't be required.

Event Timeline

Bstorm triaged this task as High priority.Feb 12 2019, 10:52 PM
Bstorm created this task.
Bstorm renamed this task from Package/copy kubeadm, kubelet, docker-ce and kubectl to toolforge aptly from to Package/copy kubeadm, kubelet, docker-ce and kubectl to Toolforge Aptly.Feb 12 2019, 11:56 PM
Bstorm renamed this task from Package/copy kubeadm, kubelet, docker-ce and kubectl to Toolforge Aptly to Package/copy kubeadm, kubelet, docker-ce and kubectl to Toolforge Aptly or Reprepro.Feb 22 2019, 3:23 PM
Bstorm moved this task from Inbox to Important on the cloud-services-team (Kanban) board.

There is some preference here for a mirror/sync vs. manually managed packages for minor version updates/patches, I think.

As a PoC, I mirrored the kubernetes-xenial repository (~5GB) as follows.

Update aptly:

root@tools-sge-services-03:~# apt install -t stretch-backports aptly

Make sure keys are imported (note: aptly doesn't work very well with gpg2):

root@tools-sge-services-03:~# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

root@tools-sge-services-03:~# gpg1 --no-default-keyring --keyring trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 6A030B21BA07F4FB

Create mirror:

root@tools-sge-services-03:~# aptly mirror create -architectures=amd64 kubernetes-xenial https://apt.kubernetes.io/ kubernetes-xenial main
Downloading https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease...
gpgv: Signature made Sat 09 Mar 2019 09:03:47 PM UTC
gpgv:                using RSA key 6A030B21BA07F4FB
gpgv: Good signature from "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"

Mirror [kubernetes-xenial]: https://apt.kubernetes.io/ kubernetes-xenial successfully added.
You can run 'aptly mirror update kubernetes-xenial' to download repository contents.

Update mirror (fetch packages):

root@tools-sge-services-03:~# aptly mirror update kubernetes-xenial
Downloading https://apt.kubernetes.io/dists/kubernetes-xenial/InRelease...
gpgv: Signature made Sat 09 Mar 2019 10:49:51 PM UTC using RSA key ID BA07F4FB
gpgv: Good signature from "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Downloading & parsing package files...
Downloading https://apt.kubernetes.io/dists/kubernetes-xenial/main/binary-amd64/Packages.gz...
Building download queue...
Download queue: 336 items (5.05 GiB)
Downloading https://apt.kubernetes.io/pool/kubeadm_1.7.3-01_amd64_c4bac3fcbc1a820523a3617495707aff0adab0db52ae02f3a5ee5001ff0a1e74.deb...
Downloading https://apt.kubernetes.io/pool/kubeadm_1.13.4-00_amd64_1094a7c75b7764a1b43d1009e76532f5a30e940e5a8154ccb82ea0abf8792685.deb...
Downloading https://apt.kubernetes.io/pool/kubectl_1.10.0-00_amd64_e391c19fa377b84587676c5577222ceb5d8fcbde442c79a9cd55d1f344293834.deb...
Downloading https://apt.kubernetes.io/pool/kubectl_1.6.0-00_amd64_a068b847837acec84e1922277dc46f6fee2b0c2f930405ff232cf9ac7e3473e5.deb...
[...]
Mirror `kubernetes-xenial` has been successfully updated.

Create a snapshot:

root@tools-sge-services-03:~# aptly snapshot create kubernetes-xenial-current from mirror kubernetes-xenial
Snapshot kubernetes-xenial-current successfully created.
You can run 'aptly publish snapshot kubernetes-xenial-current' to publish snapshot as Debian repository.

Publish repository based on snapshot:

root@tools-sge-services-03:~# aptly publish --skip-signing snapshot -distribution=kubernetes-xenial kubernetes-xenial-current 
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...

Snapshot kubernetes-xenial-current has been successfully published.

Then on a worker node:

root@tools-worker-1010:~# cat /etc/apt/sources.list.d/project-aptly.list 
deb [trusted=yes] http://tools-sge-services-03.tools.eqiad.wmflabs/repo jessie-tools main
deb [trusted=yes] http://tools-sge-services-03.tools.eqiad.wmflabs/repo kubernetes-xenial main

root@tools-worker-1010:/etc/apt/sources.list.d# apt-cache policy kubeadm
kubeadm:
  Installed: (none)
  Candidate: 1.13.4-00
  Version table:
     1.13.4-00 0
       1500 http://tools-sge-services-03.tools.eqiad.wmflabs/repo/ kubernetes-xenial/main amd64 Packages
     1.13.3-00 0
       1500 http://tools-sge-services-03.tools.eqiad.wmflabs/repo/ kubernetes-xenial/main amd64 Packages

To get newer packages, update the mirror and publish a new snapshot:

root@tools-sge-services-03:~# aptly mirror update kubernetes-xenial
Mirror `kubernetes-xenial` has been successfully updated.

root@tools-sge-services-03:~# aptly snapshot create kubernetes-xenial-new from mirror kubernetes-xenial
Snapshot kubernetes-xenial-new successfully created.


root@tools-sge-services-03:~# aptly publish --skip-signing switch kubernetes-xenial kubernetes-xenial-new
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Cleaning up prefix "." components main...

Publish for snapshot ./kubernetes-xenial (origin: kubernetes-xenial) [amd64] publishes {main: [kubernetes-xenial-20190310]: Snapshot from mirror [kubernetes-xenial]: https://apt.kubernetes.io/ kubernetes-xenial} has been successfully switched to new snapshot.

# aptly snapshot drop kubernetes-xenial-current
Snapshot `kubernetes-xenial-current` has been dropped.

root@tools-sge-services-03:~# aptly snapshot rename kubernetes-xenial-new kubernetes-xenial-current

Snapshot kubernetes-xenial-new -> kubernetes-xenial-current has been successfully renamed.

@Bstorm if this looks acceptable, I can work on a script to do it automatically (and also mirror the Docker repository).

aborrero added a subscriber: MoritzMuehlenhoff.EditedMar 12 2019, 6:32 PM

We talked about evaluating if we should use production reprepro instead. We didn't do any in-deep conversation though. CC @MoritzMuehlenhoff

GTirloni removed a subscriber: GTirloni.Mar 21 2019, 9:06 PM

@MoritzMuehlenhoff would be OK to import DEB packages from https://apt.kubernetes.io/ into our reprepro? Let's talk on IRC for the details if you want.

aborrero claimed this task.Mar 26 2019, 4:47 PM
aborrero moved this task from Needs discussion to Doing on the cloud-services-team (Kanban) board.

I just talked on IRC with @MoritzMuehlenhoff We may need reprepro if we want, as long as we use clearly identified repo components, like "whatever-k8s" or something like that, to avoid confusion with prod k8s packages.

aborrero lowered the priority of this task from High to Low.Apr 29 2019, 9:27 AM

Lowering priority of this task. Last time I talked with this with @Bstorm we agreed on reconsidering using same approach as prod k8s.

aborrero raised the priority of this task from Low to High.Jun 27 2019, 4:07 PM

Raising priority. I'm doing this now, since we will be trying kubeadm this week and the next.

Change 519449 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: add stretch-wikimedia/component/kubeadm-k8s

https://gerrit.wikimedia.org/r/519449

Change 519449 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: add stretch-wikimedia/thirdparty/kubeadm-k8s

https://gerrit.wikimedia.org/r/519449

Mentioned in SAL (#wikimedia-operations) [2019-06-27T17:21:04Z] <arturo> imported gpg keys 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 and 54A647F9048D5688D7DA2ABE6A030B21BA07F4FB into install1002 for T215975

Change 519463 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: fix wrong component name for thirdparty/kubeadm-k8s

https://gerrit.wikimedia.org/r/519463

Change 519463 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: fix wrong component name for thirdparty/kubeadm-k8s

https://gerrit.wikimedia.org/r/519463

Change 519470 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: reprepro config file format fixes for thirdparty/kubeadm-k8s

https://gerrit.wikimedia.org/r/519470

Change 519470 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: reprepro config file format fixes for thirdparty/kubeadm-k8s

https://gerrit.wikimedia.org/r/519470

Change 519472 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: fix missing -e flag in grep-dctrl for thirparty/kubeadm-k8s

https://gerrit.wikimedia.org/r/519472

Change 519472 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: fix missing -e flag in grep-dctrl for thirparty/kubeadm-k8s

https://gerrit.wikimedia.org/r/519472

Change 519480 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: k8s: add basic kubeadm infra

https://gerrit.wikimedia.org/r/519480

Change 519480 merged by Bstorm:
[operations/puppet@production] toolforge: k8s: add basic kubeadm infra

https://gerrit.wikimedia.org/r/519480

Change 519696 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: the kubeadm repo can't be labeled trusted in puppet apparently

https://gerrit.wikimedia.org/r/519696

Change 519696 merged by Bstorm:
[operations/puppet@production] toolforge: the kubeadm repo can't be labeled trusted in puppet apparently

https://gerrit.wikimedia.org/r/519696

Apparently, I missed a dependency (or perhaps this is a new one):

The following packages have unmet dependencies:
 docker-ce : Depends: containerd.io (>= 1.2.2-3) but it is not installable

Change 519726 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] aptrepo: fix the kubeadm packages to include containerd.io

https://gerrit.wikimedia.org/r/519726

Change 519726 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: fix the kubeadm packages to include containerd.io

https://gerrit.wikimedia.org/r/519726

aborrero closed this task as Resolved.Jul 1 2019, 10:25 AM

This should be done now :-)

See toolforge::k8s::kubeadmrepo for reference.

Change 519991 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] aptrepo: thirdparty/kubeadm-k8s: add cri-tools

https://gerrit.wikimedia.org/r/519991

Change 519991 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] aptrepo: thirdparty/kubeadm-k8s: add cri-tools

https://gerrit.wikimedia.org/r/519991