That is because pip provided by the Debian package attempts to prevent you from overriding packages that are managed by Debian packages. There is a lot of documentation instructing people to try to sudo pip install which surely end up breaking anything that is managed by Debian. Thus they went to make pip to complain/abort with the message you are seeing.

I don't know what pip install --local does, maybe it installs under /usr/local (rather than system wide to /usr) but one typically does not have write access to it so it makes sense to complain. Instead you can use pip install --user which would install under $HOME/.local in which you have write access (it is your home) but then the Debian version would still complain with the above message.

There are several way to fix it:

Report bug

Complain to upstream Debian to have at least pip install --user to not complain, I am pretty sure that was the case at some point but the patch must have been removed. I am absolutely sure I went through that rabbit hole some years ago and ended up giving up.

Break it!!

In your environment set export PIP_BREAK_SYSTEM_PACKAGES=1, that is the same as always invoking pip with --break-system-packages. That get rid of the blocking message and you would get something such as:

$ PIP_BREAK_SYSTEM_PACKAGES=1 /usr/bin/pip install requests
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (2.28.1)

Note how it states Defaulting to user installation, so that means the pip provided by Debian actually change the config to NOT break the package and that falls back to install to your user directory ($HOME/.local) exactly as if --user had been passed. So you can set PIP_USER=1 to always enable that:

$ PIP_BREAK_SYSTEM_PACKAGES=1 PIP_USER=1 /usr/bin/pip install requests
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (2.28.1)

So you can:

# Default to install to $HOME/.local
export PIP_USER=1

# Work around Ubuntu provided pip not recognizing installations to $HOME
export PIP_BREAK_SYSTEM_PACKAGES=1

Use a virtualenv for the app

When installing dependencies in a shared $HOME/.local, you would eventually face some kind of dependency conflict or sometime can end up using the wrong version. Instead you can use a virtualenv, which is the recommendation given by the message:

$ virtualenv venv
$ ./venv/bin/pip install .
$ ./venv/bin/docker-pkg
...

Which you can then symlink in your $HOME/bin (that is often added to one PATH when the directory exists).

Use another pip/Python

Use https://github.com/pyenv/pyenv to setup several Pythons, they would not come with the Debian specific fix and would have a pip that defaults to do user installation in and thus under $HOME/.local. That opens the benefit to have several python versions in parallel and to be entirely independent from Debian installed python/pip and packages.

Conclusion

After years and years, I highly recommend pyenv, it dramatically simplify the experience and lets you test across several versions which is quite handy.

I do use virtualenv / tox to set up the applications in most case. This way they have their own little dependencies isolated from the other applications you might have.

And since I am sometime lazy/forget about the above, I do have PIP_BREAK_SYSTEM_PACKAGES=1 PIP_USER=1 so that docker-pkg and quibble are installed in my $HOME/.local and share their dependencies. But really I should symlink the venv instead.

I am happy to pair and assist you anytime :)

@hashar Thank you for the detailed explanation, as always :) I had read a bit about the reason behind the debian package being marked as externally managed, and I think they're sensible. I was looking for a quick way to get it up and working, without messing with my python installation. As I mentioned in the task description, pipx seems to work nicely, and it's essentially just a wrapper for virtualenvs. To me that's ideal because it allows me to keep a single python version, not mess up with system packages, and the only thing I need to remember is to add "x" after "pip". Obviously this is partly motivated by the fact that I don't need python for many other things; else I probably would have taken the pyenv route.

Unfortunately though, I don't think we can resolve this task for two reasons:

• The install instructions at https://www.mediawiki.org/wiki/Continuous_integration/Docker are not working in a setup like mine. Since virtualenvs seem to be the preferred way to install things nowadays, maybe that's what the install instructions should mention. As a side note, I don't even know if that page is meant to have the "canonical" installation instructions for docker-pkg. The tool's documentation page says nothing about installing.
• Virtualenvs aside, there's the incompatibility with python 3.12 due to the removal of distutils. I guess I could create a separate task for it if needed, but I don't think that can be resolved without updating the requirements list in setup.py (either requiring docker >= 6, or adding setuptools).

In T380135#10341483, @hashar wrote:

So that is the subject of Debian bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030335 , the reporter complains they can't use pip install -e. The proposed fix was to add a patch to the Debian package which implemented --break-system-packages option https://salsa.debian.org/python-team/packages/python-pip/-/commit/68673ebd85db676cded1fc8a8f34f46bfc4e14d4 . The patch has been upstreamed to pip with pip 23.0.1 / pip 23.1.

Hmmmm I see. So I guess pip install -e --break-system-packages would be fine in this case. That seems subpar UX to me, but then what do I know about python.

Meanwhile, I guess you can try with virtualenv and update the wiki page with the commands that worked for you which really should be the one I pasted in my previous comment:

pipx install -e . worked for me (after I updated the requirements for python 3.12). I've updated the documentation to recommend using a venv, with both methods as examples.

For the python3.12 compatibility, yes please file another task for it. docker-pkg should probably drop setuptools/distutils (for Quibble that was T345093 / and the series of changes leading to https://gerrit.wikimedia.org/r/c/integration/quibble/+/958495 ).

Thanks, I've created T380424.