Page MenuHomePhabricator
Create Task
Maniphest T300682

contint1001 and contint2001 need a newer version of Docker installed
Closed, ResolvedPublic
Actions

Description

The docker package installed on our contint* hosts is terribly out of date relative to the rest of the CI infrastructure and needs a newer package installed from thirdparty/ci.

The migration of contint1001 and contint2001 to buster effectively downgraded its docker package, since we'd been using thirdparty/ci for stretch (version 19.03) but started using the distro package with buster (version 18.09). Our runners are now running bullseye and docker 20.10. Since contint servers are the only runners allowed to publish images, their docker packages need to be kept as up-to-date as possible with other runners.

Details

SubjectRepoBranchLines +/-
contint: Bump docker 20.10 version for thirdparty/ci on busteroperations/puppetproduction+1 -1
Revert "Revert "contint: Install docker 20.10 from thirdparty/ci on buster""operations/puppetproduction+14 -10
contint: Install docker 20.10 from thirdparty/ci on busteroperations/puppetproduction+14 -10
Add missing update config for thirdparty/docker-ci-busteroperations/puppetproduction+1 -0
aptrepo: add docker packages to thirdparty/ci for busteroperations/puppetproduction+9 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

dduvall created this task.Feb 1 2022, 11:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 1 2022, 11:25 PM
dduvall added a comment.Feb 1 2022, 11:26 PM

Note this issue is currently blocking T296046: Allow build time control of effective UID/GID for runtime in Blubber generated Dockerfile. See T296046#7668655

gerritbot added a comment.Feb 1 2022, 11:52 PM

Change 758986 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] aptrepo: add docker packages to thirdparty/ci for buster

https://gerrit.wikimedia.org/r/758986

gerritbot added a project: Patch-For-Review.Feb 1 2022, 11:52 PM

Change 758987 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] contint: Install docker 20.10 from thirdparty/ci on buster

https://gerrit.wikimedia.org/r/758987

Legoktm added a parent task: T296046: Allow build time control of effective UID/GID for runtime in Blubber generated Dockerfile.Feb 2 2022, 4:24 AM
Dzahn added a project: serviceops.Feb 11 2022, 9:57 PM
Stashbot added a comment.Feb 14 2022, 6:44 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-14T18:44:43Z] <mutante> contint2001 - disabling puppet, try replacing docker version (docker-io -> docker-ce), contint1001 first which is currently NOT the active server - gerrit:758987 T300682

gerritbot added a comment.Feb 16 2022, 8:21 AM

Change 758986 merged by Muehlenhoff:

[operations/puppet@production] aptrepo: add docker packages to thirdparty/ci for buster

https://gerrit.wikimedia.org/r/758986

gerritbot added a comment.Feb 16 2022, 8:34 AM

Change 763184 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add missing update config for thirdparty/docker-ci-buster

https://gerrit.wikimedia.org/r/763184

gerritbot added a comment.Feb 16 2022, 8:39 AM

Change 763184 merged by Muehlenhoff:

[operations/puppet@production] Add missing update config for thirdparty/docker-ci-buster

https://gerrit.wikimedia.org/r/763184

gerritbot added a comment.Feb 16 2022, 4:48 PM

Change 758987 merged by Dzahn:

[operations/puppet@production] contint: Install docker 20.10 from thirdparty/ci on buster

https://gerrit.wikimedia.org/r/758987

Stashbot added a comment.Feb 16 2022, 4:51 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-16T16:51:15Z] <mutante> contint2001 - temp disabled puppet (active CI server) - contint1001 - attempting to install newer docker version (gerrit:758987 T300682)

Maintenance_bot removed a project: Patch-For-Review.Feb 16 2022, 5:10 PM
dduvall added a subscriber: Muehlenhoff.Feb 16 2022, 5:48 PM

@Muehlenhoff for some reason, it seems the docker-ce package did not make it into the thirdparty/ci component after https://gerrit.wikimedia.org/r/c/operations/puppet/+/758986 was merged.

Specifically:

I'm not sure whether I did something wrong in the reprepro configuration or there was an issue in deployment. Let me know if there's something I can do to fix it. Thanks!

dduvall added a subscriber: Dzahn.Feb 16 2022, 5:49 PM

And many thanks to @Dzahn for troubleshooting following the puppet apply of https://gerrit.wikimedia.org/r/758987 this morning.

MoritzMuehlenhoff subscribed.Feb 17 2022, 8:12 AM
In T300682#7715571, @dduvall wrote:

@Muehlenhoff for some reason, it seems the docker-ce package did not make it into the thirdparty/ci component after https://gerrit.wikimedia.org/r/c/operations/puppet/+/758986 was merged.

Specifically:

I'm not sure whether I did something wrong in the reprepro configuration or there was an issue in deployment. Let me know if there's something I can do to fix it. Thanks!

https://gerrit.wikimedia.org/r/c/operations/puppet/+/758986 only adds a config where to pull from, but the sync still needs to be run by the reprepro command, I assume that didn't happen before https://gerrit.wikimedia.org/r/758987 was merged. I can import the packages in a bit.

MoritzMuehlenhoff added a comment.Feb 17 2022, 8:37 AM

These have now been imported:

jmm@gin:~$ curl -s https://apt.wikimedia.org/wikimedia/dists/buster-wikimedia/thirdparty/ci/binary-amd64/Packages | grep -e ^Package -e ^Version
Package: containerd.io
Version: 1.4.12-1
Package: docker-ce
Version: 5:20.10.12~3-0~debian-buster
Package: docker-ce-cli
Version: 5:20.10.12~3-0~debian-buster
Package: jenkins
Version: 2.319.3
gerritbot added a comment.Mar 7 2022, 6:40 PM

Change 768774 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] Revert "Revert "contint: Install docker 20.10 from thirdparty/ci on buster""

https://gerrit.wikimedia.org/r/768774

gerritbot added a project: Patch-For-Review.Mar 7 2022, 6:40 PM
gerritbot added a comment.May 5 2022, 6:15 PM

Change 768774 merged by Dzahn:

[operations/puppet@production] Revert "Revert "contint: Install docker 20.10 from thirdparty/ci on buster""

https://gerrit.wikimedia.org/r/768774

gerritbot added a comment.May 5 2022, 6:31 PM

Change 789668 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] contint: Bump docker 20.10 version for thirdparty/ci on buster

https://gerrit.wikimedia.org/r/789668

gerritbot added a comment.May 5 2022, 6:33 PM

Change 789668 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] contint: Bump docker 20.10 version for thirdparty/ci on buster

https://gerrit.wikimedia.org/r/789668

gerritbot added a comment.May 5 2022, 6:35 PM

Change 789668 merged by Dzahn:

[operations/puppet@production] contint: Bump docker 20.10 version for thirdparty/ci on buster

https://gerrit.wikimedia.org/r/789668

Stashbot added a comment.May 5 2022, 6:42 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-05T18:42:01Z] <mutante> contitn2001 - apt-get remove --purge docker.io after docker-ce was installed by puppet for T300682

Stashbot added a comment.May 5 2022, 6:51 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-05T18:51:31Z] <mutante> contitn1001 - apt-get remove --purge docker.io after docker-ce was installed by puppet for T300682 (different behaviour from contint2001 since it did not have /var/lib/docker)

Dzahn added a comment.May 5 2022, 6:55 PM
root@contint1001:/srv/docker# docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:37 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:46 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
Dzahn closed this task as Resolved.May 5 2022, 6:57 PM
Dzahn claimed this task.

docker-ce replaced docker.io on both contint* machines andis on 20.10.12 now.

bd808 mentioned this in T296046: Allow build time control of effective UID/GID for runtime in Blubber generated Dockerfile.May 5 2022, 7:23 PM
bd808 mentioned this in rGBLBR6331215531de: Revert "Revert "feature: build-time arguments for lives & runs user config"".Jun 29 2022, 7:36 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL