Page MenuHomePhabricator
Create Task
Maniphest T381923

Toolforge Build Service does not support .python-version
Closed, ResolvedPublic
Actions

Description

Help:Toolforge/Building container images (permalink) advises that one can use e.g. “python 3.12, see Specifying a Python runtime”. The linked Heroku documentation advises users to create a .python-version file that should control the Python version. However, the build service does not appear to take this file into account whatsoever. Only specifying a runtime.txt file, as documented in older Heroku documentation, appears to work – I assume we’re using an older version of whatever the relevant component is (pack?).

Presumably for the same reason, Python 3.13 doesn’t appear to be available – trying to select python-3.13.0 in runtime.txt results in:

[step-build] 2024-12-10T18:08:21.777668762Z [Error: Requested Python version is not available]
[step-build] 2024-12-10T18:08:21.777686324Z The requested Python version (3.13.0) is not available for this stack (heroku-22).
[step-build] 2024-12-10T18:08:21.777701510Z 
[step-build] 2024-12-10T18:08:21.777711416Z Please update the version in 'runtime.txt' to a supported Python version, or else
[step-build] 2024-12-10T18:08:21.777721493Z remove the file to instead use the default version (currently Python 3.12.1).
[step-build] 2024-12-10T18:08:21.777729950Z 
[step-build] 2024-12-10T18:08:21.777739389Z For a list of the supported Python versions, see:
[step-build] 2024-12-10T18:08:21.777749236Z https://devcenter.heroku.com/articles/python-support#supported-runtimes
[step-build] 2024-12-10T18:08:21.777758243Z 
[step-build] 2024-12-10T18:08:21.779447421Z ERROR: failed to build: exit status 1

Related Objects
Search...

Event Timeline

LucasWerkmeister created this task.Dec 10 2024, 8:40 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptDec 10 2024, 8:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
LucasWerkmeister mentioned this in T381899: Add support for Python 3.13.Dec 10 2024, 8:41 PM
dcaro subscribed.Dec 12 2024, 8:49 AM

Yep, the documentation shifted and we did not update (yet) the buildpack underneath, this will be solved with the next buildpack upgrade, related tasks:

T374056: [builds-builder] Upgrade python buildpack to v0.17.0 or newer for Poetry support
T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble

fnegri changed the task status from Open to Stalled.Dec 17 2024, 3:18 PM
fnegri triaged this task as Medium priority.
fnegri added subtasks: T374056: [builds-builder] Upgrade python buildpack to v0.17.0 or newer for Poetry support, T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble.
fnegri subscribed.

Setting to "Stalled" until the subtasks are completed.

dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.Jan 7 2025, 1:56 PM
dcaro changed the status of subtask T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble from Open to In Progress.Mar 12 2025, 3:14 PM
DaxServer subscribed.Apr 5 2025, 10:57 AM
dcaro added a comment.May 12 2025, 4:29 PM

@LucasWerkmeister can you try now with toolforge build start --use-latest-versions <url>? That will be using the new runner (ubuntu24) and the newer buildpacks too (python, etc.)

Stashbot added a comment.May 12 2025, 5:42 PM

Mentioned in SAL (#wikimedia-cloud) [2025-05-12T17:42:30Z] <wmbot~lucaswerkmeister@tools-bastion-13> deployed d949e8ee4a (Python 3.13 using --use-latest-versions via T381923)

LucasWerkmeister added a comment.May 12 2025, 5:43 PM

Seems to be working \o/

I feel like the first build took longer to start showing output than usual, but that might just be random noise or even bad memory. (I wasn’t paying attention during the second build.)

dcaro added a comment.May 12 2025, 5:52 PM

Nice :), it might be because it's the first time it uses the builder/runner/etc. images, so it has to pull them anew.

Feel free to close the task once you tested it a bit.

LucasWerkmeister added a comment.May 12 2025, 5:53 PM

I don’t think I have any more Python build service tools at the moment, so I’ll just go ahead and close this ^^

LucasWerkmeister closed this task as Resolved.May 12 2025, 5:53 PM
LucasWerkmeister assigned this task to dcaro.
LucasWerkmeister mentioned this in R2494:d949e8ee4a2f: Update to Python 3.13.May 12 2025, 9:48 PM
dcaro changed the status of subtask T374056: [builds-builder] Upgrade python buildpack to v0.17.0 or newer for Poetry support from Open to In Progress.May 14 2025, 2:04 PM
LucasWerkmeister reopened this task as Open.May 22 2025, 9:37 PM
In T381923#10813247, @LucasWerkmeister wrote:

Seems to be working \o/

I spoke too soon :( I just noticed that the tool has been partially broken since this deployment, with errors like these:

tools.quickcategories@tools-bastion-13:~$ kubectl logs background-runner-6d4b45f48-jj6bv
Traceback (most recent call last):
  File "/layers/heroku_python/python/lib/python3.13/getpass.py", line 173, in getuser
    return pwd.getpwuid(os.getuid())[0]
           ~~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'getpwuid(): uid not found: 53976'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/./background_runner.py", line 15, in <module>
    from database import DatabaseBatchStore, DatabasePreferenceStore
  File "/workspace/database.py", line 10, in <module>
    import pymysql
  File "/layers/heroku_python/venv/lib/python3.13/site-packages/pymysql/__init__.py", line 79, in <module>
    from . import connections  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/venv/lib/python3.13/site-packages/pymysql/connections.py", line 41, in <module>
    DEFAULT_USER = getpass.getuser()
  File "/layers/heroku_python/python/lib/python3.13/getpass.py", line 175, in getuser
    raise OSError('No username set in the environment') from e
OSError: No username set in the environment

The webservice is still alive because Kubernetes dutifully kept the old pod up – yay for health checks! – but that also means that nobody reported the breakage to me.

tools.quickcategories@tools-bastion-13:~$ kubectl get pods                                                                                                                                                                                    
NAME                                READY   STATUS             RESTARTS           AGE                                                                                                                                                         
background-runner-6d4b45f48-52thd   0/1     CrashLoopBackOff   2861 (3m16s ago)   10d                                                                                                                                                         
quickcategories-69958b9845-tn85m    0/1     CrashLoopBackOff   3045 (73s ago)     10d                                                                                                                                                         
quickcategories-9dc6df5d9-qrmct     1/1     Running            0                  45d

(Since then, I tried to deploy another update, so the age and restarts of the broken pods are now much lower.)

I assume this error is due to --use-latest-versions, though I haven’t yet tried to revert that.

bd808 subscribed.May 22 2025, 9:39 PM

A build from https://gitlab.wikimedia.org/toolforge-repos/containers-bnc/-/commit/a937384ad626eb5902910906e37749348caa3757 worked for me.

Stashbot added a comment.May 22 2025, 9:52 PM

Mentioned in SAL (#wikimedia-cloud) [2025-05-22T21:52:35Z] <wmbot~lucaswerkmeister@tools-bastion-13> deployed 74cd3dee83 (install PyMySQL from Git for Python 3.13 compatibility; CC T381923)

LucasWerkmeister closed this task as Resolved.May 22 2025, 9:56 PM

False alarm – turns out this error was caused by an unfortunate confluence of several factors:

  • Build service containers have always had SSSD uninstalled / unconfigured, so the user name was never available, but this was previously not a problem
  • Python 3.13 changed getpass.getuser() to raise an OSError rather than a KeyError if no user name is defined in the environment
  • PyMySQL previously caught ImportError and KeyError when calling getpass.getuser(), but not OSError, so this now crashed the tool
  • PyMySQL merged a fix for this, but has yet to publish it in a new release, so even upgrading all dependencies (as I did earlier today) didn’t fix the error

So for now, I’m installing PyMySQL from Git to include the fix and make the tool work under Python 3.13 again.

LucasWerkmeister mentioned this in R2362:34bb10528872: Install PyMySQL from Git.Jul 12 2025, 8:45 PM
LucasWerkmeister mentioned this in R2422:81a627821b7f: Upgrade to Python 3.13 and Toolforge Build Service.Jul 13 2025, 3:24 PM
LucasWerkmeister mentioned this in R2469:a11bdfe30c70: Upgrade to Python 3.13 and Toolforge Build Service.Jul 19 2025, 3:57 PM
fnegri changed the status of subtask T374056: [builds-builder] Upgrade python buildpack to v0.17.0 or newer for Poetry support from In Progress to Open.Jan 13 2026, 5:09 PM
dcaro reopened subtask T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble as Open.Mar 3 2026, 2:45 PM
aputhin changed the status of subtask T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble from Open to In Progress.Mar 17 2026, 5:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits