Page MenuHomePhabricator

Failures updating python packages in dev environment following Linux Docker improvements
Closed, ResolvedPublicBUG REPORT

Description

Follow up to T295318: Fix Toolhub developer environment tooling to work on Linux hosts:

There are additional permissions problems inside the "web" (python) container related to the runtime user change.

  1. make web-shell does not set new runtime $HOME in to environment which causes failures for poetry attempting to cache pip information for operations like poetry add $package.
  2. When the $HOME issue is corrected, poetry add $package fails with a permissions error writing to the virtual environment (venv) at /opt/lib/poetry/toolhub-{random hash}-py3.7 inside the container.

The first is a relatively easy fix of adjusting the Makefile command. The second is a trickier issue to resolve without the ability to sudo in the container to change permissions. We could attempt to work around it via rebuilding the venv as the runtime user in some other location at container startup, but hopefully there is some less computationally expensive solution to be found.

Event Timeline

bd808 changed the subtype of this task from "Task" to "Bug Report".Nov 16 2021, 9:11 PM

A temporary work around for users of Docker Desktop for MacOS (which does not have the volume mount permissions issue of blocking writes when the runtime container user and the host OS user have mismatched effective UID values) is to set LOCAL_UID=65533 and LOCAL_GID=65533 in your .env. This keeps the runtime effective UID the same during container build and container runtime, which in turn eliminates the file ownership issues for files created during container build.

This also hints that it may be possible to make things work better generally by finding a way to vary the UID/GID for the "somebody" user inside the container at build time. That would allow matching build time and runtime UID/GID across platforms.

For linux users (and I suspect everyone else), it is possible to log into the container where you want to install the package using docker container exec -u 0 -it <container name> bash. This allows you to log into the container as root so you can install packages with poetry add <package name>.

This is obviously a hack, but it does help in getting yourself unblocked while we work on a fix for this issue.

bd808 changed the task status from Open to Stalled.Apr 25 2022, 7:32 PM
bd808 triaged this task as Medium priority.
bd808 moved this task from Backlog to Radar on the Toolhub board.
bd808 changed the task status from Stalled to In Progress.Jul 1 2022, 5:18 PM
bd808 moved this task from Radar to Review on the Toolhub board.

Change 810390 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[wikimedia/toolhub@main] dev: set build time effective UID/GID

https://gerrit.wikimedia.org/r/810390

Change 810390 merged by jenkins-bot:

[wikimedia/toolhub@main] dev: set build time effective UID/GID

https://gerrit.wikimedia.org/r/810390

This should be fixed now. Please do reopen if the ARGS integration with docker-compose added in https://gerrit.wikimedia.org/r/c/wikimedia/toolhub/+/810390/ is found to be insufficient.