Page MenuHomePhabricator

Building "web" container failing during `poetry install` in multiple environments
Closed, ResolvedPublicBUG REPORT

Description

First reported by @Slst2020 in team chat and later reproduced by CI builds (wikimedia-toolhub-pipeline-test/1589):

Step 17/25 : RUN poetry "install" "--no-root"
 ---> Running in 035335d6dd1f
Traceback (most recent call last):
  File "/usr/local/bin/poetry", line 5, in <module>
    from poetry.console import main
  File "/usr/local/lib/python3.7/dist-packages/poetry/console/__init__.py", line 1, in <module>
    from .application import Application
  File "/usr/local/lib/python3.7/dist-packages/poetry/console/application.py", line 7, in <module>
    from .commands.about import AboutCommand
  File "/usr/local/lib/python3.7/dist-packages/poetry/console/commands/__init__.py", line 4, in <module>
    from .check import CheckCommand
  File "/usr/local/lib/python3.7/dist-packages/poetry/console/commands/check.py", line 2, in <module>
    from poetry.factory import Factory
  File "/usr/local/lib/python3.7/dist-packages/poetry/factory.py", line 18, in <module>
    from .repositories.pypi_repository import PyPiRepository
  File "/usr/local/lib/python3.7/dist-packages/poetry/repositories/pypi_repository.py", line 33, in <module>
    from ..inspection.info import PackageInfo
  File "/usr/local/lib/python3.7/dist-packages/poetry/inspection/info.py", line 25, in <module>
    from poetry.utils.env import EnvCommandError
  File "/usr/local/lib/python3.7/dist-packages/poetry/utils/env.py", line 23, in <module>
    import virtualenv
  File "/usr/local/lib/python3.7/dist-packages/virtualenv/__init__.py", line 3, in <module>
    from .run import cli_run, session_via_cli
  File "/usr/local/lib/python3.7/dist-packages/virtualenv/run/__init__.py", line 14, in <module>
    from .plugin.creators import CreatorSelector
  File "/usr/local/lib/python3.7/dist-packages/virtualenv/run/plugin/creators.py", line 6, in <module>
    from virtualenv.create.via_global_ref.builtin.builtin_way import VirtualenvBuiltin
  File "/usr/local/lib/python3.7/dist-packages/virtualenv/create/via_global_ref/builtin/builtin_way.py", line 7, in <module>
    from virtualenv.create.creator import Creator
  File "/usr/local/lib/python3.7/dist-packages/virtualenv/create/creator.py", line 15, in <module>
    from virtualenv.discovery.cached_py_info import LogCmd
  File "/usr/local/lib/python3.7/dist-packages/virtualenv/discovery/cached_py_info.py", line 23, in <module>
    _CACHE[Path(sys.executable)] = PythonInfo()
  File "/usr/local/lib/python3.7/dist-packages/virtualenv/discovery/py_info.py", line 86, in __init__
    self.distutils_install = {u(k): u(v) for k, v in self._distutils_install().items()}
  File "/usr/local/lib/python3.7/dist-packages/virtualenv/discovery/py_info.py", line 152, in _distutils_install
    d = dist.Distribution({"script_args": "--no-user-cfg"})  # conf files not parsed so they do not hijack paths
  File "/usr/local/lib/python3.7/dist-packages/setuptools/dist.py", line 456, in __init__
    for ep in metadata.entry_points(group='distutils.setup_keywords'):
  File "/usr/local/lib/python3.7/dist-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 1003, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/usr/local/lib/python3.7/dist-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 453, in load
    ordered = sorted(eps, key=by_group)
  File "/usr/local/lib/python3.7/dist-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 1001, in <genexpr>
    dist.entry_points for dist in unique(distributions())
  File "/usr/local/lib/python3.7/dist-packages/setuptools/_vendor/importlib_metadata/_itertools.py", line 16, in unique_everseen
    k = key(element)
AttributeError: 'PathDistribution' object has no attribute '_normalized_name'

Event Timeline

bd808 changed the task status from Open to In Progress.Feb 14 2022, 4:53 PM
bd808 claimed this task.
bd808 triaged this task as High priority.

This is blocking all of our work in Gerrit so really UBN! for the Toolhub project, but only tagging as High to stay off of various reports about UBN! which are more properly focused on production errors rather than CI/local dev problems.

The python3-buster base image we build on got an update yesterday (2022-02-13) which feels like a pretty suspicious coincidence. The stacktrace shows that the error is being raised from the system package for setuptools which vendors some version of the importlib-metadata library.

@Slst2020 reports in team chat that pinning to python3-buster:0.0.1-20220206 gets her local environment working again.

@Slst2020 reports in team chat that pinning to python3-buster:0.0.1-20220206 gets her local environment working again.

You beat me to it :))

@Slst2020 reports in team chat that pinning to python3-buster:0.0.1-20220206 gets her local environment working again.

You beat me to it :))

Interestingly that pin does not work for me locally. The python3 runtime and library versions are the same inside of the 0.0.1-20220206 and 0.0.1-20220213 base images. The Blubber generated instructions applied to the base image include using easy_install to upgrade the system pip and then using the system pip to upgrade setuptools. Both would try to install the latest versions which work with the python version (3.7.3). Setuptools 60.9.0 has been released in the last 24 hours.

The last version of out container that built in CI was docker-registry.wikimedia.org/wikimedia/wikimedia-toolhub:2022-02-11-140743-production. That container has setuptools 60.8.1 installed.

Looks like https://github.com/pypa/setuptools/issues/3102, though importlib_metadata issues like this have also appeared in virtualenv.

I have a sneaking suspicion that pinning to an older version of the base image worked for Slavina because she had a locally cached layer for the combination of :0.0.1-20220206 and the pip install step (RUN python3 "-m" "easy_install" "pip" && python3 "-m" "pip" "install" "-U" "setuptools" "wheel" "tox" "pip") which ended up with a setuptools < 60.9.0.

Change 762503 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[wikimedia/toolhub@main] [WIP] Will container build with :0.0.1-20220206 base?

https://gerrit.wikimedia.org/r/762503

Change 762503 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[wikimedia/toolhub@main] [WIP] Will container build with :0.0.1-20220206 base?

https://gerrit.wikimedia.org/r/762503

This builds successfully, and I think validates my theory about cached layers:

Step 6/25 : RUN python3 "-m" "easy_install" "pip" && python3 "-m" "pip" "install" "-U" "setuptools" "wheel" "tox" "pip"
 ---> Using cache
 ---> 02886e2c003c

Looks like https://github.com/pypa/setuptools/issues/3102, though importlib_metadata issues like this have also appeared in virtualenv.

I agree that this sounds like the same issue we are hitting.

I guess our near term options are to do one or more of:

  • pin to python3-buster:0.0.1-20220206 base image where it appears CI has a cached layer with a working setuptools and hope that cache sticks around until there is an upstream fix
  • patch blubber to make it avoid installing setuptools==60.9.0
  • twiddle our thumbs until setuptools releases a >60.9.0 version fixing the upstream issue

Change 762503 merged by jenkins-bot:

[wikimedia/toolhub@main] build: Work around setuptools==60.9.0 issues

https://gerrit.wikimedia.org/r/762503

Change 762552 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[blubber@master] python: ban setuptools==60.9.0 from installing

https://gerrit.wikimedia.org/r/762552

bd808 changed the task status from In Progress to Open.Feb 14 2022, 11:06 PM
bd808 lowered the priority of this task from High to Medium.
bd808 added a project: Upstream.

Lowering priority to medium now that we have a working hack to keep our CI pipelines running. Once the blubber patch lands we should partially revert https://gerrit.wikimedia.org/r/762503 by setting the base image version back to "latest".

Change 762552 merged by jenkins-bot:

[blubber@master] python: ban setuptools==60.9.0 from installing

https://gerrit.wikimedia.org/r/762552

Change 762965 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[wikimedia/toolhub@main] build: use python3-buster:latest base image again

https://gerrit.wikimedia.org/r/762965

Change 762965 merged by jenkins-bot:

[wikimedia/toolhub@main] build: use python3-buster:latest base image again

https://gerrit.wikimedia.org/r/762965

Change 770638 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/deployment-charts@master] toolhub: Bump container version to 2022-03-15-002555-production

https://gerrit.wikimedia.org/r/770638

Change 770638 merged by jenkins-bot:

[operations/deployment-charts@master] toolhub: Bump container version to 2022-03-15-002555-production

https://gerrit.wikimedia.org/r/770638