go: downloading github.com/subosito/gotenv v1.2.0
go: downloading github.com/pelletier/go-toml v1.9.4
failed to collect values: failed to get commit: git: error="exit status 128" stderr=fatal: detected dubious ownership in repository at '/builds/repos/releng/cli'
To add an exception for this directory, call:
	git config --global --add safe.directory /builds/repos/releng/cli
/go/bin/gox-v1.0.1 -output="./_release/latest/mw_latest_{{.OS}}_{{.Arch}}" -osarch='darwin/amd64 darwin/arm64 linux/amd64 linux/386 linux/arm linux/arm64 linux/ppc64le windows/amd64' -ldflags '' gitlab.wikimedia.org/repos/releng/cli
Error reading packages: exit status 1
Stderr: error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
make: *** [Makefile:23: release] Error 1
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

As a result, I want to run git config --global --add safe.directory /builds/repos/releng/cli in my gitlab job.
However, I run into an issue as the nobody user can not create the nonexistent directory needed

nobody@b2cf1f51ff32:/go$ git config --global --add safe.directory /builds/repos/releng/cli
error: could not lock config file /nonexistent/.gitconfig: No such file or directory
nobody@b2cf1f51ff32:/home$ mkdir $HOME
mkdir: cannot create directory ‘/nonexistent’: Permission denied

It would be great if images that use a user such as nobody out of the box all have home directories that actually exist so tooling such as git works

Related Objects

Mentioned Here: T365871: Upstream HOME=/tmp from fresh-node to WMF CI images

Event Timeline

Addshore created this task.Mar 4 2023, 7:43 PM

Addshore moved this task from Inbox to Watching & Blocked on the a Wikimedia CLI board.Mar 4 2023, 8:09 PM

Let me track back one level: why are we getting the error at all?

Looking at the logs, it seems that /builds/repos/releng/cli is not owned by the user nobody, or is world-writable. Can we change that? I'm asking because I don't exactly love to add a /nonexistent directory, and the fact that user nobody has no homedir is a deliberate security measure.

For what it's worth, it's called /nonexistent to point out that it does not exist. it would be completely counter intuitive (as well as breaking the logic it is like that) if that directory suddenly started existing.

the fact that user nobody has no homedir is a deliberate security measure.

Ah, right, in that case let me see if I can find a work around
I imagine such a work around will end up being useful if these images continue being used in CI by folks on gitlab.

Looking at the logs, it seems that /builds/repos/releng/cli is not owned by the user nobody, or is world-writable

I imagine (but didnt look) that it has the ownership of the user that sets up the cloned repo, rather than the user used in this image which then only runs 1 part of the job

On the gitlab runner / CI side of things, it looks like choosing a user for the steps in CI is not possible, per https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2750
(But it might be coming)

If this did land, I would be able to use a user consistently through the CI

Addshore renamed this task from Create /nonexistent direcotry for nobody user in golang images to Create /nonexistent directory for nobody user in golang images.Mar 14 2023, 2:55 PM

A possible workaround could be to add the correct stanza to /etc/gitconfig

Tagging Release-Engineering-Team and Collaboration-Services teams because it involves CI and gitlab.

Clement_Goubert triaged this task as Medium priority.Mar 15 2023, 12:09 PM

Clement_Goubert moved this task from Incoming 🐫 to serviceops-radar on the serviceops-deprecated board.

Clement_Goubert edited projects, added: serviceops-radar; removed: serviceops-deprecated.

LSobanski moved this task from Incoming to Backlog on the Collaboration-Services board.May 8 2023, 3:40 PM

@Addshore I see that https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2750 has been implemented. Could you check if this addresses your use case? Here's the relevant documentation: https://docs.gitlab.com/ee/ci/yaml/index.html#imagedocker.

ping @Addshore This ticket is still open but possibly resolved since 2023. thoughts?

@Dzahn so I don't actually use these images any more, as the golang ones from docker hub are accessible on the runners now.
So I'd be happy to just close this now if noone else wants it open!

See also T365871: Upstream HOME=/tmp from fresh-node to WMF CI images, where we added ENV HOME=/tmp to all node-test and node-test-browser images for a similar reason.

Based on previous comments I am boldly closing this as resolved. Finding out the "if noone else wants it open" usually works best by just doing it.

Several jobs over the years have failed for one reason or another due to our unusual use of HOME=/nonexistent in WMF CI. We work around it by changing HOME in those jobs to a directory that is writable. I don't remember the benefit of /nonexistent, but a growing number of jobs don't benefit from it today, because we accept the workarounds.

Are there jobs where we would deny such workaround?

This task represents an example of such a job. Do we have enough examples to reconsider our stance on HOME=/nonexistent in the base image? If not, I suggest we re-state the benefit we believe it gives us, and perhaps invert it by applying it only to jobs where we need it.

The oppertunity here is to make WMF CI less novel and more welcoming to developers by not being different unless we have a compelling reason to being different. This is more suitable to self-servicing. I worry this may be the kind of thing that prolongs fears around Zuul and vague desires toward GitHub/GitLab. It takes a long time to recover from a bad first impression.

Set writeable HOME for nobody user in golang images instead of /nonexistentOpen, MediumPublicActions

Description

Related Objects

Event Timeline

Set writeable HOME for nobody user in golang images instead of /nonexistent
Open, MediumPublic
Actions