Page MenuHomePhabricator

Package conflicts for tox causes CI failure
Closed, ResolvedPublicBUG REPORT

Description

The MinT project uses tox for testing. The requirements.txt specifies a tox version. But the Dockerfile generated by blubber already installs the tox package. Since there is no virtualenvironment isolation in the docker image, both versions conflicts.

Example CI failure: https://integration.wikimedia.org/ci/job/machinetranslation-pipeline-test/280/console

Docker file from blubber .pipeline/blubber.yaml production command:


FROM docker-registry.wikimedia.org/python3-build-bullseye AS production
USER 0
ENV HOME="/root"
ENV DEBIAN_FRONTEND="noninteractive"
RUN apt-get update && apt-get install -y "git" "build-essential" "cmake" "wget" "unzip" "wmf-certificates" "ca-certificates" && rm -rf /var/lib/apt/lists/*
RUN python3 "-m" "pip" "install" "-U" "setuptools!=60.9.0" && python3 "-m" "pip" "install" "-U" "wheel" "tox" "pip"
ARG LIVES_AS="somebody"
ARG LIVES_UID=65533
ARG LIVES_GID=65533
RUN (getent group "$LIVES_GID" || groupadd -o -g "$LIVES_GID" -r "$LIVES_AS") && (getent passwd "$LIVES_UID" || useradd -l -o -m -d "/home/$LIVES_AS" -r -g "$LIVES_GID" -u "$LIVES_UID" "$LIVES_AS") && mkdir -p "/app" && chown "$LIVES_UID":"$LIVES_GID" "/app" && mkdir -p "/opt/lib" && chown "$LIVES_UID":"$LIVES_GID" "/opt/lib"
ARG RUNS_AS="runuser"
ARG RUNS_UID=900
ARG RUNS_GID=900
RUN (getent group "$RUNS_GID" || groupadd -o -g "$RUNS_GID" -r "$RUNS_AS") && (getent passwd "$RUNS_UID" || useradd -l -o -m -d "/home/$RUNS_AS" -r -g "$RUNS_GID" -u "$RUNS_UID" "$RUNS_AS")
USER $LIVES_UID
ENV HOME="/home/somebody"
WORKDIR "/app"
COPY --chown=$LIVES_UID:$LIVES_GID ["requirements.txt", "./"]
ENV PIP_FIND_LINKS="file:///opt/lib/python" PIP_WHEEL_DIR="/opt/lib/python"
RUN mkdir -p "/opt/lib/python"
RUN python3 "-m" "pip" "wheel" "-r" "requirements.txt" && python3 "-m" "pip" "install" "--target" "/opt/lib/python/site-packages" "-r" "requirements.txt"
ENV PATH="/opt/lib/python/site-packages/bin:${PATH}" PYTHONPATH="/opt/lib/python/site-packages"
COPY --chown=$LIVES_UID:$LIVES_GID [".", "."]
ENV PIP_NO_INDEX="1"
ENTRYPOINT ["/app/entrypoint.sh"]

LABEL blubber.variant="production" blubber.version="0.9.0+2638669" 

As you can see in line 6 tox is installed. virtualenv is a dependency to it, so that also get installed. Later when the actual tox runs for tests, we get the following error:

AttributeError: module 'virtualenv.create.via_global_ref.builtin.cpython.mac_os' has no attribute 'CPython3macOsBrew'

As per this question at stackoverlow, the error is due to two virtualenv packages

Is there way we can force virtual environment for the application instead of using the system environment? I saw a mention of that in T346090: Blubber: Add workaround for Bookworm's Python pip restrictions

Event Timeline

santhosh renamed this task from Package conflics for tox causes CI failure to Package conflicts for tox causes CI failure.Sep 13 2023, 11:34 AM

Hi, I looked into this issue again. Blubber not using a virtual environment is an important issue. As debian bookworm's pip restrictions is also coming up, we need to fix this.

I did some changes to the python.go in blubber code and got this Dockerfile

FROM docker-registry.wikimedia.org/python3-build-bullseye AS test
USER 0
ENV HOME="/root"
ENV DEBIAN_FRONTEND="noninteractive"
RUN apt-get update && apt-get install -y "git" "build-essential" "cmake" "wget" "unzip" "wmf-certificates" "python3-venv" && rm -rf /var/lib/apt/lists/*
ARG LIVES_AS="somebody"
ARG LIVES_UID=65533
ARG LIVES_GID=65533
RUN (getent group "$LIVES_GID" || groupadd -o -g "$LIVES_GID" -r "$LIVES_AS") && (getent passwd "$LIVES_UID" || useradd -l -o -m -d "/home/$LIVES_AS" -r -g "$LIVES_GID" -u "$LIVES_UID" "$LIVES_AS") && mkdir -p "/app" && chown "$LIVES_UID":"$LIVES_GID" "/app" && mkdir -p "/opt/lib" && chown "$LIVES_UID":"$LIVES_GID" "/opt/lib"
ARG RUNS_AS="runuser"
ARG RUNS_UID=900
ARG RUNS_GID=900
RUN (getent group "$RUNS_GID" || groupadd -o -g "$RUNS_GID" -r "$RUNS_AS") && (getent passwd "$RUNS_UID" || useradd -l -o -m -d "/home/$RUNS_AS" -r -g "$RUNS_GID" -u "$RUNS_UID" "$RUNS_AS")
USER $LIVES_UID
ENV HOME="/home/somebody"
WORKDIR "/app"
COPY --chown=$LIVES_UID:$LIVES_GID ["requirements.txt", "./"]
ENV VIRTUAL_ENV="/app/.venv"
RUN python3 "-m" "venv" "/app/.venv" && python3 "-m" "pip" "install" "-U" "setuptools!=60.9.0" && python3 "-m" "pip" "install" "-U" "wheel" "tox" "pip"
ENV PATH="/app/.venv/bin:${PATH}"
RUN python3 "-m" "pip" "install" "-r" "requirements.txt"
COPY --chown=$LIVES_UID:$LIVES_GID [".", "."]
# ENV PIP_NO_INDEX="1"
ENTRYPOINT ["tox"]

LABEL blubber.variant="test" blubber.version="+"

The main difference is not installing any python package outside a venv. Also some additional changes:

  • Do not use PYTHONPATH
  • Avoid ENV PIP_NO_INDEX="1" since tox require installing packages when tox.ini has deps instructions. If PIP_NO_INDEX="1", tox will not be able to install any packages. And will fail
  • Avoid pip wheel -r requirements.txt - I don't know why it is needed. pip install -r requirements.txt is all we needed. I can understand the need for a pip wheel . to build wheel for the current application, but why build wheel for all items in requirements.txt? Where is it used?
  • PIP_WHEEL_DIR and PIP_FIND_LINKS removed. I don't think they are useful.
  • remove PIP_BREAK_SYSTEM_PACKAGES as we are using venv now

I did not change anything related to RUNS_AS related params. LIVES_AS is the only one used here. Not sure the need of RUNS_AS

With this Dockerfile, I am able to run the tests successfully.

To get this Dockerfile from blubber, I did some changes to python.go, since I am not too familiar with the repo or even the go language, I am providing a diff here so that you may refer:

diff --git a/config/python.go b/config/python.go
index 1a22130..5f192d0 100644
--- a/config/python.go
+++ b/config/python.go
@@ -107,16 +107,6 @@ func (pc PythonConfig) InstructionsForPhase(phase build.Phase) []build.Instructi
 		switch phase {
 		case build.PhasePrivileged:
 			if pc.Requirements != nil || pc.usePoetry() {
-				if pc.Requirements != nil {
-					ins = append(ins, build.Env{map[string]string{
-						"PIP_BREAK_SYSTEM_PACKAGES": "1",
-					}})
-					ins = append(ins, build.RunAll{[]build.Run{
-						{pc.version(), append([]string{"-m", "pip", "install", "-U", "setuptools!=60.9.0"})},
-						{pc.version(), append([]string{"-m", "pip", "install", "-U", "wheel", "tox", pc.pipPackage()})},
-					}})
-				}
-
 				if pc.usePoetry() {
 					ins = append(ins, build.Env{map[string]string{
 						"POETRY_VIRTUALENVS_PATH": PythonPoetryVenvs,
@@ -134,10 +124,16 @@ func (pc PythonConfig) InstructionsForPhase(phase build.Phase) []build.Instructi
 				if !pc.usePoetry() {
 					ins = append(ins, []build.Instruction{
 						build.Env{map[string]string{
-							"PIP_WHEEL_DIR":  PythonLibPrefix,
-							"PIP_FIND_LINKS": "file://" + PythonLibPrefix,
+							"VIRTUAL_ENV": "/app/.venv",
+						}},
+						build.RunAll{[]build.Run{
+							{pc.version(), []string{"-m", "venv", "/app/.venv"}},
+							{pc.version(), append([]string{"-m", "pip", "install", "-U", "setuptools!=60.9.0"})},
+							{pc.version(), append([]string{"-m", "pip", "install", "-U", "wheel", pc.pipPackage()})},
+						}},
+						build.Env{map[string]string{
+							"PATH": "/app/.venv" + "/bin" + ":${PATH}",
 						}},
-						build.CreateDirectory(PythonLibPrefix),
 					}...)
 				}
 
@@ -150,35 +146,23 @@ func (pc PythonConfig) InstructionsForPhase(phase build.Phase) []build.Instructi
 					ins = append(ins, build.Run{"poetry", cmd})
 
 				} else if args := pc.RequirementsArgs(); len(args) > 0 {
-					installCmd := append([]string{"-m", "pip", "install", "--target"}, PythonSitePackages)
+					installCmd := []string{"-m", "pip", "install"}
 					if pc.UseSystemFlag.True {
 						installCmd = InsertElement(installCmd, "--system", PosOf(installCmd, "install")+1)
 					}
 					if pc.UseNoDepsFlag.True {
 						installCmd = InsertElement(installCmd, "--no-deps", PosOf(installCmd, "install")+1)
 					}
-					wheelCmd := append([]string{"-m", "pip", "wheel"})
-					if pc.UseNoDepsFlag.True {
-						wheelCmd = InsertElement(wheelCmd, "--no-deps", PosOf(wheelCmd, "wheel")+1)
-					}
+
 
 					ins = append(ins, build.RunAll{[]build.Run{
-						{pc.version(), append(wheelCmd, args...)},
+
 						{pc.version(), append(installCmd, args...)},
 					}})
 				}
 			}
 
-			if !pc.usePoetry() {
-				// PythonSitePackages and the wheel cache are not used with
-				// Poetry. Instead Poetry is allowed to  manage a venv
-				// containing the installed packages and their related
-				// scripts.
-				ins = append(ins, build.Env{map[string]string{
-					"PYTHONPATH": PythonSitePackages + ":${PYTHONPATH}",
-					"PATH":       PythonSiteBin + ":${PATH}",
-				}})
-			}
+
 
 		case build.PhasePostInstall:
 			if !pc.usePoetry() && pc.Requirements != nil {

@hashar, @dancy , @dduvall Requesting your help in getting this fixed in blubber so that the CI is unbroken for MinT. Thanks!

Change 958578 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/services/machinetranslation@master] Workaround for blubber-installed tox version

https://gerrit.wikimedia.org/r/958578

Change 958584 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/services/machinetranslation@master] Revise blubber.yaml to deal with package conflicts

https://gerrit.wikimedia.org/r/958584

Change 958578 abandoned by Ahmon Dancy:

[mediawiki/services/machinetranslation@master] Workaround for blubber-installed tox version

Reason:

This approach did not work.

https://gerrit.wikimedia.org/r/958578

@santhosh I made a couple of unsuccessful attempts to come up with a quick workaround for you. I'll keep trying tomorrow.

I did a temporary fix in the repository to unblock CI issue so that we are not blocked by issue https://gerrit.wikimedia.org/r/c/mediawiki/services/machinetranslation/+/958599 - directly call pytest instead of calling via tox.
but black, and ruff checks are not present in it.

Change #1035009 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/services/machinetranslation@master] CI: Revive use of tox for tests

https://gerrit.wikimedia.org/r/1035009

Change #958584 abandoned by Ahmon Dancy:

[mediawiki/services/machinetranslation@master] Revise blubber.yaml to deal with package conflicts

Reason:

Change 1035009 does it right.

https://gerrit.wikimedia.org/r/958584

I did a temporary fix in the repository to unblock CI issue so that we are not blocked by issue https://gerrit.wikimedia.org/r/c/mediawiki/services/machinetranslation/+/958599 - directly call pytest instead of calling via tox.
but black, and ruff checks are not present in it.

https://gerrit.wikimedia.org/r/1035009 should get things back on track.

dancy changed the task status from Open to In Progress.May 22 2024, 5:23 PM
dancy claimed this task.
dancy triaged this task as Medium priority.

Change #1035009 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] CI: Revive use of tox for tests

https://gerrit.wikimedia.org/r/1035009

Change #1041372 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update MinT to 2024-06-11-052620-production

https://gerrit.wikimedia.org/r/1041372

Change #1041372 merged by jenkins-bot:

[operations/deployment-charts@master] Update MinT to 2024-06-11-052620-production

https://gerrit.wikimedia.org/r/1041372

Mentioned in SAL (#wikimedia-operations) [2024-06-11T07:40:29Z] <kart_> Updated MinT to 2024-06-11-052620-production (T364122, T346226, T357548)