Page MenuHomePhabricator

Deal with git safe.directory in CI Docker images
Closed, ResolvedPublic

Description

Git introduced a feature to prevent arbitrary execution of repositories hook which is controlled by the git config safe.directory T335354.

git is included in the base CI images ci-* which never got rebuild since the git package got updated. When adding rsync to the base image ( https://gerrit.wikimedia.org/r/c/integration/config/+/983892 ), the whole fleet of CI images got rebuild, upgrading git as a side effect and that leads to CI builds failing "unexpectedly" as expected.

An example of a build failure with Quibble:

00:00:11.235 INFO:quibble.commands:<<< Finish: Zuul clone {"cache_dir": "/srv/git", "projects": ["mediawiki/core", "mediawiki/extensions/AbuseFilter", "mediawiki/extensions/AntiSpoof", "mediawiki/extensions/Babel", "mediawiki/extensions/BetaFeatures", "mediawiki/extensions/CheckUser", "mediawiki/extensions/CirrusSearch", "mediawiki/extensions/Cite", "mediawiki/extensions/CiteThisPage", "mediawiki/extensions/CodeEditor", "mediawiki/extensions/ConfirmEdit", "mediawiki/extensions/ContentTranslation", "mediawiki/extensions/Disambiguator", "mediawiki/extensions/Echo", "mediawiki/extensions/Elastica", "mediawiki/extensions/EventBus", "mediawiki/extensions/EventLogging", "mediawiki/extensions/EventStreamConfig", "mediawiki/extensions/FileImporter", "mediawiki/extensions/Gadgets", "mediawiki/extensions/GeoData", "mediawiki/extensions/GlobalCssJs", "mediawiki/extensions/GlobalPreferences", "mediawiki/extensions/Graph", "mediawiki/extensions/GrowthExperiments", "mediawiki/extensions/GuidedTour", "mediawiki/extensions/ImageMap", "mediawiki/extensions/InputBox", "mediawiki/extensions/Interwiki", "mediawiki/extensions/JsonConfig", "mediawiki/extensions/Kartographer", "mediawiki/extensions/Math", "mediawiki/extensions/MobileApp", "mediawiki/extensions/MobileFrontend", "mediawiki/extensions/NavigationTiming", "mediawiki/extensions/PageImages", "mediawiki/extensions/PageTriage", "mediawiki/extensions/PageViewInfo", "mediawiki/extensions/ParserFunctions", "mediawiki/extensions/PdfHandler", "mediawiki/extensions/Poem", "mediawiki/extensions/ProofreadPage", "mediawiki/extensions/SandboxLink", "mediawiki/extensions/Scribunto", "mediawiki/extensions/SiteMatrix", "mediawiki/extensions/SpamBlacklist", "mediawiki/extensions/TemplateData", "mediawiki/extensions/Thanks", "mediawiki/extensions/TimedMediaHandler", "mediawiki/extensions/Translate", "mediawiki/extensions/UniversalLanguageSelector", "mediawiki/extensions/VisualEditor", "mediawiki/extensions/WikiEditor", "mediawiki/extensions/WikiLove", "mediawiki/extensions/Wikibase", "mediawiki/extensions/WikibaseCirrusSearch", "mediawiki/extensions/WikibaseMediaInfo", "mediawiki/extensions/WikimediaMessages", "mediawiki/extensions/cldr", "mediawiki/skins/MinervaNeue", "mediawiki/skins/Vector", "mediawiki/vendor"], "workers": 8, "workspace": "/workspace/src", "zuul_branch": "master", "zuul_project": "mediawiki/core", "zuul_ref": "refs/zuul/master/Z94deb24f0f1a4b7b8d887942a426d5c5", "zuul_url": "git://contint2002.wikimedia.org"}, in 4.403 s
00:00:11.239 Traceback (most recent call last):
00:00:11.239   File "/usr/local/bin/quibble", line 8, in <module>
00:00:11.239     sys.exit(main())
00:00:11.239   File "/usr/local/lib/python3.7/dist-packages/quibble/cmd.py", line 901, in main
00:00:11.239     dry_run=args.dry_run,
00:00:11.240   File "/usr/local/lib/python3.7/dist-packages/quibble/cmd.py", line 554, in execute
00:00:11.240     quibble.commands.execute_command(command)
00:00:11.240   File "/usr/local/lib/python3.7/dist-packages/quibble/commands.py", line 46, in execute_command
00:00:11.240     command.execute()
00:00:11.240   File "/usr/local/lib/python3.7/dist-packages/quibble/commands.py", line 181, in execute
00:00:11.240     self.zuul_url,
00:00:11.240   File "/usr/local/lib/python3.7/dist-packages/quibble/zuul.py", line 97, in clone
00:00:11.240     zuul_cloner.prepareRepo('mediawiki/core', dests['mediawiki/core'])
00:00:11.240   File "/usr/local/lib/python3.7/dist-packages/zuul/lib/cloner.py", line 163, in prepareRepo
00:00:11.240     repo.prune()
00:00:11.240   File "/usr/local/lib/python3.7/dist-packages/zuul/merger/merger.py", line 119, in prune
00:00:11.240     stale_refs = origin.stale_refs
00:00:11.240   File "/usr/local/lib/python3.7/dist-packages/git/remote.py", line 741, in stale_refs
00:00:11.240     for line in self.repo.git.remote("prune", "--dry-run", self).splitlines()[2:]:
00:00:11.240   File "/usr/local/lib/python3.7/dist-packages/git/cmd.py", line 736, in <lambda>
00:00:11.240     return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
00:00:11.240   File "/usr/local/lib/python3.7/dist-packages/git/cmd.py", line 1316, in _call_process
00:00:11.240     return self.execute(call, **exec_kwargs)
00:00:11.240   File "/usr/local/lib/python3.7/dist-packages/git/cmd.py", line 1111, in execute
00:00:11.240     raise GitCommandError(redacted_command, status, stderr_value, stdout_value)
00:00:11.240 git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
00:00:11.240   cmdline: git remote prune --dry-run origin
00:00:11.240   stderr: 'fatal: detected dubious ownership in repository at '/workspace/src'
00:00:11.240 To add an exception for this directory, call:
00:00:11.240 
00:00:11.240 	git config --global --add safe.directory /workspace/src'
00:00:11.698 Build step 'Execute shell' marked build as failure

https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-php74-docker/34254/console

Event Timeline

hashar added a parent task: Restricted Task.Jan 5 2024, 10:49 AM

My guess is the src directory is created on the host and owned by jenkins-deploy / uid 2947 while the container runs git as nobody / uid 65534.

It seems to me the easiest is to entirely disable the feature by adding safe.directory=* to one of the git protected configurations. Most probably /etc/gitconfig which can be done via dockerfiles/ci-common/content/gitconfig.

Change 988408 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: disable git safe.directory

https://gerrit.wikimedia.org/r/988408

Change 988408 merged by jenkins-bot:

[integration/config@master] dockerfiles: disable git safe.directory

https://gerrit.wikimedia.org/r/988408

Mentioned in SAL (#wikimedia-releng) [2024-01-08T14:19:47Z] <James_F> Docker: Publishing new CI images with git safe.directory disabled for T354409

thcipriani assigned this task to hashar.
thcipriani added a subscriber: thcipriani.

This covers images in integration/config, if there are further issues please reopen.