Page MenuHomePhabricator

cloud puppetmasters fail to run git-sync-upstream
Closed, ResolvedPublic

Description

Reported by @SLyngshede-WMF on IRC and later verified by me:

aborrero@cloud-puppetmaster-05:~$ sudo git-sync-upstream 
2022-12-15T12:06:04Z INFO     sync-upstream: Rebasing repository '/var/lib/git/operations/puppet' on top of commit '243b3631783a62d5ab9c2cf4e0c2573b73a6f066'
2022-12-15T12:06:04Z INFO     git.cmd: git diff --abbrev=40 --full-index -M --raw --no-color
Traceback (most recent call last):
  File "/usr/local/bin/git-sync-upstream", line 173, in <module>
    "/var/lib/git/operations/puppet", resp.content.decode("ascii").strip(), gauge_is_up_to_date
  File "/usr/local/bin/git-sync-upstream", line 46, in rebase_repo
    if repo.index.diff(None):
  File "/usr/lib/python3/dist-packages/git/index/util.py", line 76, in check_default_index
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/git/index/base.py", line 1239, in diff
    return super(IndexFile, self).diff(other, paths, create_patch, **kwargs)
  File "/usr/lib/python3/dist-packages/git/diff.py", line 152, in diff
    index = diff_method(self.repo, proc)
  File "/usr/lib/python3/dist-packages/git/diff.py", line 515, in _index_from_raw_format
    handle_process_output(proc, handle_diff_line, None, finalize_process, decode_streams=False)
  File "/usr/lib/python3/dist-packages/git/cmd.py", line 120, in handle_process_output
    return finalizer(process)
  File "/usr/lib/python3/dist-packages/git/util.py", line 333, in finalize_process
    proc.wait(**kwargs)
  File "/usr/lib/python3/dist-packages/git/cmd.py", line 415, in wait
    raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(129)
  cmdline: git diff --abbrev=40 --full-index -M --raw --no-color
aborrero@cloud-puppetmaster-05:/var/lib/git/operations/puppet$ sudo git diff --abbrev=40 --full-index -M --raw --no-color
Not a git repository

And

aborrero@toolsbeta-puppetmaster-04:~$ sudo git-sync-upstream
2022-12-15T12:08:23Z INFO     sync-upstream: Rebasing repository '/var/lib/git/operations/puppet' on top of commit '243b3631783a62d5ab9c2cf4e0c2573b73a6f066'
2022-12-15T12:08:23Z INFO     git.cmd: git diff --abbrev=40 --full-index -M --raw --no-color
Traceback (most recent call last):
  File "/usr/local/bin/git-sync-upstream", line 173, in <module>
    "/var/lib/git/operations/puppet", resp.content.decode("ascii").strip(), gauge_is_up_to_date
  File "/usr/local/bin/git-sync-upstream", line 46, in rebase_repo
    if repo.index.diff(None):
  File "/usr/lib/python3/dist-packages/git/index/util.py", line 76, in check_default_index
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/git/index/base.py", line 1239, in diff
    return super(IndexFile, self).diff(other, paths, create_patch, **kwargs)
  File "/usr/lib/python3/dist-packages/git/diff.py", line 152, in diff
    index = diff_method(self.repo, proc)
  File "/usr/lib/python3/dist-packages/git/diff.py", line 515, in _index_from_raw_format
    handle_process_output(proc, handle_diff_line, None, finalize_process, decode_streams=False)
  File "/usr/lib/python3/dist-packages/git/cmd.py", line 120, in handle_process_output
    return finalizer(process)
  File "/usr/lib/python3/dist-packages/git/util.py", line 333, in finalize_process
    proc.wait(**kwargs)
  File "/usr/lib/python3/dist-packages/git/cmd.py", line 415, in wait
    raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(129)
  cmdline: git diff --abbrev=40 --full-index -M --raw --no-color

aborrero@toolsbeta-puppetmaster-04:/var/lib/git/operations/puppet$ sudo git log
fatal: detected dubious ownership in repository at '/var/lib/git/operations/puppet'
To add an exception for this directory, call:

	git config --global --add safe.directory /var/lib/git/operations/puppet
aborrero@toolsbeta-puppetmaster-04:/var/lib/git/operations/puppet$ ll
total 144
drwxr-xr-x  14 gitpuppet gitpuppet  4096 Dec  5 12:53 ./
drwxr-xr-x   4 root      root       4096 Jun 11  2020 ../
drwxr-xr-x   7 gitpuppet gitpuppet  4096 Mar 28  2022 conftool-data/
-rw-r--r--   1 root      root       3534 Dec  5 12:53 CONTRIBUTORS
-rw-r--r--   1 gitpuppet gitpuppet   323 Jun 11  2020 .ctags
drwxr-xr-x   2 gitpuppet gitpuppet  4096 Jun 11  2020 doc/
drwxr-xr-x   5 gitpuppet gitpuppet  4096 Jun 11  2020 environments/
drwxr-xr-x   3 root      root       4096 Mar 11  2022 examples/
-rw-r--r--   1 root      root       1751 Dec  2 14:31 Gemfile
drwxr-xr-x   8 gitpuppet gitpuppet  4096 Dec 13 23:32 .git/
-rw-r--r--   1 root      root       1339 Aug  8 11:13 .gitignore
-rw-r--r--   1 gitpuppet gitpuppet   665 Jun 11  2020 .gitmessage
-rw-r--r--   1 gitpuppet gitpuppet     0 Jun 11  2020 .gitmodules
-rw-r--r--   1 gitpuppet gitpuppet   101 Jun 11  2020 .gitreview
drwxr-xr-x  12 gitpuppet gitpuppet  4096 Dec 13 23:32 hieradata/
-rw-r--r--   1 gitpuppet gitpuppet    28 Jun 11  2020 .ignored.yaml
-rw-r--r--   1 root      root      11727 Jul 11 08:50 .mailmap
drwxr-xr-x   2 gitpuppet gitpuppet  4096 Dec 13 21:41 manifests/
drwxr-xr-x 280 gitpuppet gitpuppet 12288 Nov 28 11:16 modules/
-rw-r--r--   1 gitpuppet gitpuppet   239 Jun 11  2020 .puppet-lint.rc
-rw-r--r--   1 root      root       5238 Sep 14 15:28 Rakefile
drwxr-xr-x   3 gitpuppet gitpuppet  4096 Nov  2 15:42 rake_modules/
-rw-r--r--   1 root      root        808 Aug  3 13:36 README
-rw-r--r--   1 root      root        231 Jun 24  2021 .rspec_parallel
-rw-r--r--   1 gitpuppet gitpuppet   439 Jun 11  2020 .rubocop_todo.yml
-rw-r--r--   1 root      root       1382 Nov  2 09:04 .rubocop.yml
drwxr-xr-x   3 root      root       4096 Jul 29 12:41 spec/
-rw-r--r--   1 root      root       4872 Nov 14 13:04 tox.ini
-rw-r--r--   1 root      root        577 Jul  5 21:20 typos
drwxr-xr-x   3 gitpuppet gitpuppet  4096 Dec  8 09:36 utils/
drwxr-xr-x   6 root      root       4096 Mar 16  2022 vendor_modules/

Per comments by @MoritzMuehlenhoff on IRC they might have run into the behaviour change imposed by the git update released for buster via LTS: https://lists.debian.org/debian-lts-announce/2022/12/msg00025.html

See also:

Event Timeline

aborrero moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.

I think some actions to solve this could include:

  • make sure the git dir is recursively owned by the gitpuppet user
  • make sure git-sync-upstream uses the gitpuppet user

Change 868400 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] puppetmaster: git-sync-upstream: use the gitpuppet user for git operations

https://gerrit.wikimedia.org/r/868400

I've tested the above patch, and seems to work!

With that patch merged, the course of operations would be to:

  • force permissions via cumin in all puppetmasters (gitpuppet:gitpuppet)
  • force run git-sync-upstream by hand via cumin

Another option might be to setup the git config it says using puppet:

git config --global --add safe.directory /var/lib/git/labs/private
git config --global --add safe.directory /var/lib/git/operations/puppet

Change 868454 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps puppet: allow multiple users to access our puppet git checkout

https://gerrit.wikimedia.org/r/868454

Change 868454 merged by Andrew Bogott:

[operations/puppet@production] cloud-vps puppet: allow multiple users to access our puppet git checkout

https://gerrit.wikimedia.org/r/868454

Change 868460 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps puppet: rework git config safe.dir definition

https://gerrit.wikimedia.org/r/868460

Change 868460 merged by Andrew Bogott:

[operations/puppet@production] cloud-vps puppet: rework git config safe.dir definition

https://gerrit.wikimedia.org/r/868460

Andrew claimed this task.
Andrew subscribed.

I hand-fixed all the self-referential hosts that couldn't pick up this change on their own. There were a couple of outliers which I was unable to reach or which had local puppet issues I couldn't resolve. The only fully broken host still is maps-puppetmaster02.maps-experiments.eqiad1.wikimedia.cloud.

Change 868400 abandoned by Arturo Borrero Gonzalez:

[operations/puppet@production] puppetmaster: git-sync-upstream: use the gitpuppet user for git operations

Reason:

other approach was used.

https://gerrit.wikimedia.org/r/868400