Page MenuHomePhabricator
Create Task
Maniphest T342346

Refresh integration/zuul/deploy to work on Debian Bullseye
Closed, ResolvedPublic
Actions

Description

Zuul is deployed by scap from integration/zuul/deploy which hold all the python dependencies in a tarball. Refreshing the dependencies is done via a Makefile and a docker image. As part of upgrading the underlying OS from Buster → Bullseye (T334517) we need to make sure the system still works properly and adjust it for Bullseye.

There are a few steps:

  • We need a python2.7 build image based on Bullseye
  • Build artifacts for Bullseye
  • The python2.7 provided by Bullseye is a minimal one, we might be missing modules?
  • Test it on a WMCS instance! In the end we tested the generated Python wheels using containers. We managed to install and start a basic Zuul server/merger pair on bullseye containers

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
zuul: require python2.7operations/puppetproduction+1 -1
Dockerfile to "run" Zuul with bullseyeintegration/zuul/deploymaster+71 -0
generate Python2 wheels for bullseye targetsintegration/zuul/deploymaster+107 -11
jjb: simplify Zuul deploy jobsintegration/configmaster+7 -26
use `pip` of current Python installation in common `run.sh`operations/docker-images/production-imagesmaster+9 -4
jjb: overhaul Zuul deploy jobsintegration/configmaster+24 -59
python-build: ensure frozen-requirements is exhaustiveoperations/docker-images/production-imagesmaster+21 -1
python-build: set date of source files in the wheelsoperations/docker-images/production-imagesmaster+7 -1
python-build: add lsb-release, make and virtualenvoperations/docker-images/production-imagesmaster+5 -5
python-build: default to run as nobody from /deployoperations/docker-images/production-imagesmaster+20 -0
support Zuul v2 on bullseye contint hostsoperations/puppetproduction+3 -0
python-build: rebuild images for recent changesoperations/docker-images/production-imagesmaster+32 -0
Add wheels for Debian Bullseyeintegration/zuul/deploymaster+2 -2
python-build: provide a python2 Bullseye imageoperations/docker-images/production-imagesmaster+45 -0
Output diff of the list of wheels held in artifactsintegration/zuul/deploymaster+6 -0
Add missing frozen requirementsintegration/zuul/deploymaster+4 -0
Add frozen-requirements.txt as a make targetintegration/zuul/deploymaster+1 -1
Add support to build with podmanintegration/zuul/deploymaster+7 -0
Add job for integration/zuul/deployintegration/configmaster+99 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Jul 20 2023, 1:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 20 2023, 1:30 PM
hashar added a parent task: T334517: upgrade contint servers to bullseye.Jul 20 2023, 1:31 PM
hashar added a subtask: T259611: The python-build images regenerate wheels even when matching ones are already available.Jul 20 2023, 1:37 PM
hashar reopened subtask T259611: The python-build images regenerate wheels even when matching ones are already available as Open.
Jdforrester-WMF renamed this task from Refresh integration/zuul/deploy makefile to Refresh integration/zuul/deploy makefile to work on Debian Bullseye.Jul 20 2023, 2:11 PM
hashar renamed this task from Refresh integration/zuul/deploy makefile to work on Debian Bullseye to Refresh integration/zuul/deploy to work on Debian Bullseye.Jul 20 2023, 2:17 PM
gerritbot added a comment.Jul 20 2023, 2:17 PM

Change 940157 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/docker-images/production-images@master] python-build: set date of source files in the wheel

https://gerrit.wikimedia.org/r/940157

gerritbot added a project: Patch-For-Review.Jul 20 2023, 2:17 PM
gerritbot added a comment.Jul 20 2023, 2:25 PM

Change 940161 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/docker-images/production-images@master] python-build: provide a python2 Bullseye image

https://gerrit.wikimedia.org/r/940161

gerritbot added a comment.Jul 20 2023, 2:41 PM

Change 940167 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/zuul/deploy@master] Add missing frozen requirements

https://gerrit.wikimedia.org/r/940167

gerritbot added a comment.Jul 20 2023, 2:41 PM

Change 940168 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/zuul/deploy@master] Add support to build with podman

https://gerrit.wikimedia.org/r/940168

gerritbot added a comment.Jul 20 2023, 2:41 PM

Change 940169 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/zuul/deploy@master] Output diff of the list of wheels held in artifacts

https://gerrit.wikimedia.org/r/940169

gerritbot added a comment.Jul 20 2023, 2:41 PM

Change 940170 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/zuul/deploy@master] Add frozen-requirements.txt as a target

https://gerrit.wikimedia.org/r/940170

gerritbot added a comment.Jul 24 2023, 2:17 PM

Change 940950 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Add job for integration/zuul/deploy

https://gerrit.wikimedia.org/r/940950

gerritbot added a comment.Jul 25 2023, 6:16 AM

Change 940950 merged by jenkins-bot:

[integration/config@master] Add job for integration/zuul/deploy

https://gerrit.wikimedia.org/r/940950

gerritbot added a comment.Jul 25 2023, 1:35 PM

Change 940168 merged by jenkins-bot:

[integration/zuul/deploy@master] Add support to build with podman

https://gerrit.wikimedia.org/r/940168

gerritbot added a comment.Jul 25 2023, 2:38 PM

Change 940170 merged by jenkins-bot:

[integration/zuul/deploy@master] Add frozen-requirements.txt as a make target

https://gerrit.wikimedia.org/r/940170

gerritbot added a comment.Jul 25 2023, 2:40 PM

Change 940167 merged by jenkins-bot:

[integration/zuul/deploy@master] Add missing frozen requirements

https://gerrit.wikimedia.org/r/940167

gerritbot added a comment.Jul 25 2023, 2:45 PM

Change 940169 merged by jenkins-bot:

[integration/zuul/deploy@master] Output diff of the list of wheels held in artifacts

https://gerrit.wikimedia.org/r/940169

gerritbot added a comment.Jul 25 2023, 3:33 PM

Change 941443 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/docker-images/production-images@master] python-build: ensure frozen-requirements is exhaustive

https://gerrit.wikimedia.org/r/941443

gerritbot added a comment.Jul 25 2023, 3:33 PM

Change 941444 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/docker-images/production-images@master] python-build: rebuild images for recent changes

https://gerrit.wikimedia.org/r/941444

hashar added a comment.Jul 26 2023, 9:23 AM

https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/940161 creates a Python 2 Docker image based on Bullseye. Since Debian Bullseye no more endorses Python 2.7, pip is no more packaged there. That makes it impossible to ship the python modules Zuul depends on:

apt install python-pip
2023-07-26 10:59:32 [docker-pkg-build] INFO - Package python-pip is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  python3-pip (drivers.py:106)

There is at least one dependency (pycrypto, dependency of Paramiko) which requires compilation against system libs.

gerritbot added a comment.Jul 26 2023, 12:15 PM

Change 941904 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/zuul/deploy@master] Add wheels for Debian Bullseye

https://gerrit.wikimedia.org/r/941904

Vgutierrez mentioned this in T344330: acme-chief should support debian bookworm.Aug 16 2023, 10:49 AM
thcipriani changed the task status from Open to In Progress.Aug 16 2023, 3:34 PM
thcipriani triaged this task as Medium priority.
thcipriani edited projects, added Release-Engineering-Team (Priority Backlog 📥); removed Release-Engineering-Team.
gerritbot added a comment.Oct 5 2023, 9:45 AM

Change 940161 merged by EoghanGaffney:

[operations/docker-images/production-images@master] python-build: provide a python2 Bullseye image

https://gerrit.wikimedia.org/r/940161

LSobanski subscribed.Oct 12 2023, 2:12 PM
gerritbot added a comment.Jan 10 2024, 8:42 AM

Change 940157 abandoned by Hashar:

[operations/docker-images/production-images@master] python-build: set date of source files in the wheels

Reason:

https://gerrit.wikimedia.org/r/940157

gerritbot added a comment.Jan 10 2024, 8:42 AM

Change 941443 abandoned by Hashar:

[operations/docker-images/production-images@master] python-build: ensure frozen-requirements is exhaustive

Reason:

https://gerrit.wikimedia.org/r/941443

gerritbot added a comment.Jan 10 2024, 8:42 AM

Change 941444 abandoned by Hashar:

[operations/docker-images/production-images@master] python-build: rebuild images for recent changes

Reason:

https://gerrit.wikimedia.org/r/941444

gerritbot added a comment.Jan 10 2024, 8:42 AM

Change 941904 abandoned by Hashar:

[integration/zuul/deploy@master] Add wheels for Debian Bullseye

Reason:

https://gerrit.wikimedia.org/r/941904

Maintenance_bot removed a project: Patch-For-Review.Jan 10 2024, 9:30 AM
gerritbot added a comment.Feb 8 2024, 2:26 PM

Change 941904 restored by Hashar:

[integration/zuul/deploy@master] Add wheels for Debian Bullseye

https://gerrit.wikimedia.org/r/941904

gerritbot added a project: Patch-For-Review.Feb 8 2024, 2:26 PM
gerritbot added a comment.Feb 12 2024, 2:11 PM

Change 941904 abandoned by Hashar:

[integration/zuul/deploy@master] Add wheels for Debian Bullseye

Reason:

In favor of https://gerrit.wikimedia.org/r/c/integration/zuul/deploy/+/1002466

https://gerrit.wikimedia.org/r/941904

gerritbot added a comment.Feb 14 2024, 8:31 AM

Change 940157 restored by Hashar:

[operations/docker-images/production-images@master] python-build: set date of source files in the wheels

https://gerrit.wikimedia.org/r/940157

gerritbot added a comment.Feb 14 2024, 8:32 AM

Change 941443 restored by Hashar:

[operations/docker-images/production-images@master] python-build: ensure frozen-requirements is exhaustive

https://gerrit.wikimedia.org/r/941443

gerritbot added a comment.Feb 14 2024, 8:33 AM

Change 941444 restored by Hashar:

[operations/docker-images/production-images@master] python-build: rebuild images for recent changes

https://gerrit.wikimedia.org/r/941444

gerritbot added a comment.Feb 14 2024, 8:44 AM

Change 941444 abandoned by Hashar:

[operations/docker-images/production-images@master] python-build: rebuild images for recent changes

Reason:

https://gerrit.wikimedia.org/r/941444

gerritbot added a comment.Feb 14 2024, 8:50 AM

Change 1002997 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/docker-images/production-images@master] python-build: default to run as nobody from /deploy

https://gerrit.wikimedia.org/r/1002997

gerritbot added a comment.Feb 14 2024, 8:50 AM

Change 1003002 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/docker-images/production-images@master] python-build: add make and virtualenv

https://gerrit.wikimedia.org/r/1003002

gerritbot added a comment.Feb 15 2024, 2:20 PM

Change 1002461 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/puppet@production] support Zuul v2 on bullseye contint hosts

https://gerrit.wikimedia.org/r/1002461

gerritbot added a comment.Feb 15 2024, 2:25 PM

Change 1002461 merged by Muehlenhoff:

[operations/puppet@production] support Zuul v2 on bullseye contint hosts

https://gerrit.wikimedia.org/r/1002461

gerritbot added a comment.Feb 15 2024, 2:26 PM

Change 1003437 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[integration/config@master] jjb: overhaul Zuul deploy jobs

https://gerrit.wikimedia.org/r/1003437

gerritbot added a comment.Feb 15 2024, 4:52 PM

Change 1002466 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[integration/zuul/deploy@master] generate Python2 wheels for bullseye targets

https://gerrit.wikimedia.org/r/1002466

gerritbot added a comment.Feb 16 2024, 11:58 AM

Change 1002997 merged by Clément Goubert:

[operations/docker-images/production-images@master] python-build: default to run as nobody from /deploy

https://gerrit.wikimedia.org/r/1002997

gerritbot added a comment.Feb 16 2024, 11:59 AM

Change 1003002 merged by Clément Goubert:

[operations/docker-images/production-images@master] python-build: add lsb-release, make and virtualenv

https://gerrit.wikimedia.org/r/1003002

gerritbot added a comment.Feb 16 2024, 11:59 AM

Change 940157 merged by Clément Goubert:

[operations/docker-images/production-images@master] python-build: set date of source files in the wheels

https://gerrit.wikimedia.org/r/940157

gerritbot added a comment.Feb 16 2024, 12:00 PM

Change 941443 merged by Clément Goubert:

[operations/docker-images/production-images@master] python-build: ensure frozen-requirements is exhaustive

https://gerrit.wikimedia.org/r/941443

gerritbot added a comment.Feb 19 2024, 2:27 PM

Change 1003437 merged by jenkins-bot:

[integration/config@master] jjb: overhaul Zuul deploy jobs

https://gerrit.wikimedia.org/r/1003437

gerritbot added a comment.Feb 19 2024, 6:21 PM

Change 1004754 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/docker-images/production-images@master] match `pip2` path used by common `run.sh`

https://gerrit.wikimedia.org/r/1004754

gerritbot added a comment.Feb 20 2024, 2:20 PM

Change 1004754 merged by Clément Goubert:

[operations/docker-images/production-images@master] use `pip` of current Python installation in common `run.sh`

https://gerrit.wikimedia.org/r/1004754

Stashbot added a comment.Feb 20 2024, 2:21 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-20T14:21:46Z] <claime> launching build-production-images - T342346

gerritbot added a comment.Feb 20 2024, 3:49 PM

Change 1005116 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/zuul/deploy@master] Dockerfile to "run" Zuul with bullseye

https://gerrit.wikimedia.org/r/1005116

Clement_Goubert subscribed.Feb 20 2024, 5:33 PM

Build is done

gerritbot added a comment.Feb 21 2024, 12:42 PM

Change 1005499 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[integration/config@master] jjb: simplify Zuul deploy jobs

https://gerrit.wikimedia.org/r/1005499

gerritbot added a comment.Feb 21 2024, 1:21 PM

Change 1005499 merged by jenkins-bot:

[integration/config@master] jjb: simplify Zuul deploy jobs

https://gerrit.wikimedia.org/r/1005499

gerritbot added a comment.Feb 21 2024, 1:39 PM

Change 1002466 merged by jenkins-bot:

[integration/zuul/deploy@master] generate Python2 wheels for bullseye targets

https://gerrit.wikimedia.org/r/1002466

jnuche closed this task as Resolved.Feb 21 2024, 2:23 PM
jnuche updated the task description. (Show Details)
gerritbot added a comment.Mar 19 2024, 10:01 AM

Change 1005116 abandoned by Hashar:

[integration/zuul/deploy@master] Dockerfile to "run" Zuul with bullseye

Reason:

This was a local hack to quickly check whether the Zuul scripts at least started.

https://gerrit.wikimedia.org/r/1005116

gerritbot added a comment.Apr 16 2024, 7:27 PM

Change #1020329 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] zuul: require python2.7

https://gerrit.wikimedia.org/r/1020329

gerritbot added a comment.Apr 16 2024, 7:30 PM

Change #1020329 abandoned by Hashar:

[operations/puppet@production] zuul: require python2.7

Reason:

The issue lies in modules/base/manifests/standard_packages.pp which removes the python package but never reinstall them when the feature `remove_python2_on_bullseye` is flipped to `true`.

https://gerrit.wikimedia.org/r/1020329

gerritbot added a comment.Apr 16 2024, 7:35 PM

Change #1020329 restored by Hashar:

[operations/puppet@production] zuul: require python2.7

https://gerrit.wikimedia.org/r/1020329

gerritbot added a comment.Apr 16 2024, 7:46 PM

Change #1020329 merged by Dzahn:

[operations/puppet@production] zuul: require python2.7

https://gerrit.wikimedia.org/r/1020329

hashar closed subtask T259611: The python-build images regenerate wheels even when matching ones are already available as Resolved.May 17 2024, 3:20 PM
hashar mentioned this in T335765: Migrate all CI jobs from buster to bullseye or later and drop buster testing support.Aug 6 2024, 4:17 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits