Page MenuHomePhabricator
Create Task
Maniphest T428024

Build spicerack for Trixie
Closed, ResolvedPublic
Actions

Description

Previous task for Bookworm: T389380

This is the current Python deps list:

INSTALL_REQUIRES = [
    "conftool>=6.0.0",
    "cumin>=3.0.2",
    "dnspython~=2.3.0",
    "gitpython>=3.1.30",
    "kafka-python~=2.0.2",
    "kubernetes==22.6.*",  # frozen to the version available on debian bookworm
    "packaging",
    "pymysql>=1.0.2",
    "pynetbox~=7.4",
    "python-etcd~=0.4.5",
    "redis==4.3.*",
    "requests>=2.28.1",
    "wmflib",
]

We map them to:

ii  python3-conftool                     6.0.1+deb12u1                        all          Set of tools to configure the WMF kv config store.
ii  python3-conftool-dbctl               6.0.1+deb12u1                        all          Set of tools to manage MediaWiki databases.
ii  cumin                                6.0.0-1+deb12u1                      all          Automation and orchestration framework written in Python
ii  python3-git                          3.1.30-1+deb12u2                     all          Python library to interact with Git repositories
ii  python3-kafka                        2.0.2-3                              all          Pure Python client for Apache Kafka - Python 3.x
ii  python3-kubernetes                   22.6.0-2                             all          Kubernetes Python client - Python 3.x
ii  python3-packaging                    23.0-1                               all          core utilities for python3 packages
ii  python3-pymysql                      1.0.2-2+deb12u1                      all          Pure-Python MySQL Driver - Python 3.x
ii  python3-pynetbox                     7.4.1-1~wmf12u1                      all          Python API client library for NetBox
ii  python3-etcd                         0.4.5-4                              all          Python Etcd client library - Python3 module
ii  python3-redis                        4.3.4-3                              all          Persistent key-value database with network interface (Python 3 library)
ii  python3-requests                     2.28.1+dfsg-1                        all          elegant and simple HTTP library for Python3, built for human beings
ii  python3-wmflib                       3.0.0-1+deb12u1                      amd64        Generic Python library for common tasks in the WMF production infrastructure
ii  python3-dnspython                    2.3.0-1                              all          DNS toolkit for Python 3

Since Trixie ships with Python 3.13 and Bookworm with 3.11, we can use a trick like https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1141941 to support both.

We should verify the list of changes for the new Trixie package versions, follow up with the teams owning cookbooks or spicerack modules depending on them and rinse/repeat until we have a working version.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
setup.py: install setuptools for Python > 3.11operations/software/spicerackdebian+2 -0
Modify rules to build sphinx on Bookworm and Trixieoperations/software/spicerackdebian+10 -1
Upgrade config and code to allow Trixie buildsoperations/software/spicerackmaster+8 -5
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT427897 Upgrade Cumin hosts to Trixie
 ResolvedelukeyT428024 Build spicerack for Trixie

Event Timeline

elukey created this task.Wed, Jun 3, 7:39 AM
elukey added a comment.EditedWed, Jun 3, 9:30 AM

On Trixie:

  • python3-kafka goes to 2.0.2-9
  • python3-kubernetes goes to 30.1.0-2
  • python3-mysql goes to 1.4.6-2+b5
  • python3-pynetbox goes to 7.4.1-1
  • python3-etcd goes to 0.4.5-6
  • python3-redis goes to 6.1.0-2
  • python3-requests goes to 2.32.3+dfsg-5+deb13u1
  • python3-dnspython goes to 2.7.0-1

We package:

  • python3-conftool
  • python3-conftool-dbctl
  • cumin
  • python3-wmflib
elukey added a comment.Wed, Jun 3, 1:56 PM

Our packages have already their Trixie variant, so we are good. Among the other changes, I'd check:

  • python3-kubernetes - we had some issues in the past when migrating, so it is worth to check the changelog.
  • python3-pymysql - good to check with DP
  • python3-pynetbox - I noticed that we are running the new version on Bookworm, but we have the 7.4.1-1~wmf12u1 suffix variant. I am relatively sure it was just to mark Bookworm, but better to be safe and check if we don't have patches stacked on top.
  • python3-redis - just a quick look even if I don't really think that our basic usage of it will make any difference. We had the same issue when upgrading to Bookworm.

Running unit tests with the updated deps should tell us more, I'll try to create a patch and run a test build.

elukey updated the task description. (Show Details)Wed, Jun 3, 1:58 PM
elukey updated the task description. (Show Details)
elukey added a comment.Wed, Jun 3, 4:23 PM

I tried to build a new version of spicerack on build2001 with Trixie as target, cherry picking https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1297184 and all the tests passed! Great first step :)

If failed with:

SETUPTOOLS_SCM_PRETEND_VERSION="12.7.0-1" python3 setup.py build_sphinx -b man
/usr/lib/python3/dist-packages/setuptools/__init__.py:94: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

        ********************************************************************************
        Requirements should be satisfied by a PEP 517 installer.
        If you are using pip, you can try `pip install --use-pep517`.
        ********************************************************************************

!!
  dist.fetch_build_eggs(dist.setup_requires)
WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/setuptools_scm/_integration/pyproject_reading.py", line 36, in read_pyproject
    section = defn.get("tool", {})[tool_name]
              ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'setuptools_scm'
/usr/lib/python3/dist-packages/setuptools/dist.py:334: InformationOnly: Normalizing '12.7.0-1' to '12.7.0.post1'
  self.metadata.version = self._normalize_version(self.metadata.version)
/usr/lib/python3/dist-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
!!

        ********************************************************************************
        Please consider removing the following classifiers in favor of a SPDX license expression:

        License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)

        See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
        ********************************************************************************

!!
  self._finalize_license_expression()
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'build_sphinx'

That is probably another quick thing to fix, will check tomorrow!

gerritbot added a comment.Thu, Jun 4, 8:44 AM

Change #1297624 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/spicerack@debian] Modify rules to build sphinx on Bookworm and Trixie

https://gerrit.wikimedia.org/r/1297624

gerritbot added a project: Patch-For-Review.Thu, Jun 4, 8:44 AM
gerritbot added a comment.Thu, Jun 4, 9:20 AM

Change #1297184 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/spicerack@master] Upgrade config and code to allow Trixie builds

https://gerrit.wikimedia.org/r/1297184

gerritbot added a comment.Thu, Jun 4, 12:28 PM

Change #1297184 merged by Elukey:

[operations/software/spicerack@master] Upgrade config and code to allow Trixie builds

https://gerrit.wikimedia.org/r/1297184

gerritbot added a comment.Thu, Jun 4, 1:22 PM

Change #1297624 merged by Elukey:

[operations/software/spicerack@debian] Modify rules to build sphinx on Bookworm and Trixie

https://gerrit.wikimedia.org/r/1297624

Maintenance_bot removed a project: Patch-For-Review.Thu, Jun 4, 1:31 PM
elukey added a comment.Thu, Jun 4, 4:06 PM

I was able to build spicerack for Trixie. Side note is that python-release needs a fix:

Traceback (most recent call last):
  File "/home/elukey/Wikimedia/spicerack/setup.py", line 3, in <module>
    from setuptools import find_packages, setup
ModuleNotFoundError: No module named 'setuptools'
2026-06-04 18:04:10,723 [ERROR]: Failed to run subprocess.run(['.tox/py313-lint_unit/bin/python', 'setup.py', 'sdist'], check=True): Command '['.tox/py313-lint_unit/bin/python', 'setup.py', 'sdist']' returned non-zero exit status 1.
>>> What do you want to do? "retry" the last command, "skip" the last command but continue, "abort" the execution, "debug" drop into a REPL shell
> skip
2026-06-04 18:05:14,126 [INFO]: Step 82: subprocess.run(['.tox/py313-lint_unit/bin/python', 'setup.py', 'bdist_wheel'], check=True)
Traceback (most recent call last):
  File "/home/elukey/Wikimedia/spicerack/setup.py", line 3, in <module>
    from setuptools import find_packages, setup
ModuleNotFoundError: No module named 'setuptools'
gerritbot added a comment.Fri, Jun 5, 7:11 AM

Change #1298036 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/spicerack@debian] setup.py: install setuptools for Python > 3.11

https://gerrit.wikimedia.org/r/1298036

gerritbot added a project: Patch-For-Review.Fri, Jun 5, 7:11 AM
gerritbot added a comment.Fri, Jun 5, 7:25 AM

Change #1298036 merged by Elukey:

[operations/software/spicerack@debian] setup.py: install setuptools for Python > 3.11

https://gerrit.wikimedia.org/r/1298036

elukey closed this task as Resolved.Fri, Jun 5, 7:27 AM
elukey claimed this task.

The new spicerack trixie deb has been deployed on cumin2003 (Trixie), since unit tests are passing I am inclined to close this task and re-open later on if anything comes up. People will use cumin2003 to test the new tools on Trixie, including cookbooks.

Maintenance_bot removed a project: Patch-For-Review.Fri, Jun 5, 7:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits