I have looked at Gerrit and nothing significant merged at the time. Eventually I found out git package has been upgraded on the machine:

Start-Date: 2022-12-14  06:06:57
Commandline: /usr/bin/unattended-upgrade
Upgrade: git:amd64 (1:2.20.1-2+deb10u4, 1:2.20.1-2+deb10u5)
End-Date: 2022-12-14  06:07:01

Start-Date: 2022-12-14  06:07:02
Commandline: /usr/bin/unattended-upgrade
Upgrade: git-man:amd64 (1:2.20.1-2+deb10u4, 1:2.20.1-2+deb10u5)
End-Date: 2022-12-14  06:07:07

/srv/mediawiki-staging has a setuid bit:

      v
drwxrwsr-x 22 mwdeploy wikidev 4096 Dec 13 09:13 /srv/mediawiki-staging/
      ^

git (1:2.20.1-2+deb10u5) buster-security; urgency=high
  
  * Non-maintainer upload by the LTS Security Team.
  * CVE-2022-24765: Git not checking the ownership of directories in a
    local multi-user system when running commands specified in the local
    repository configuration.  This allows the owner of the repository to
    cause arbitrary commands to be executed by other users who access the
    repository.
  * The above introduces new 'safe.directory' checks which may cause
    regressions: allow opt-out of such checks with 'safe.directory=*'
  * CVE-2022-29187: an unsuspecting user could still be affected by the
    issue reported in CVE-2022-24765, for example when navigating as root
    into a shared tmp directory that is owned by them, but where an
    attacker could create a git repository.
  * CVE-2022-39253: exposure of sensitive information to a malicious
    actor. When performing a local clone (where the source and target of
    the clone are on the same volume), Git copies the contents of the
    source's `$GIT_DIR/objects` directory into the destination by either
    creating hardlinks to the source contents, or copying them (if
    hardlinks are disabled via `--no-hardlinks`). A malicious actor could
    convince a victim to clone a repository with a symbolic link pointing
    at sensitive information on the victim's machine.
  * CVE-2022-39260: `git shell` improperly uses an `int` to represent the
    number of entries in the array, allowing a malicious actor to
    intentionally overflow the return value, leading to arbitrary heap
    writes. Because the resulting array is then passed to `execv()`, it is
    possible to leverage this attack to gain remote code execution on a
    victim machine.

 -- Sylvain Beucler <beuc@debian.org>  Tue, 13 Dec 2022 15:14:23 +0100

Another note is we have the git repository flagged as shared between user by setting core.sharedRepository=group. In this case, I argue the security check should still allow command if a user is a member of the group owning the .git directory.

The new safe.directory setting does not accept any wildcard setting. Since we can execute git commands in any repository below /srv/mediawiki-staging I am guessing the easiest is to disable that feature as instructed.

Change 868002 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] scap: disable git safe.directory

https://gerrit.wikimedia.org/r/868002

Mentioned in SAL (#wikimedia-releng) [2022-12-14T08:43:11Z] <hashar> deployment-prep: fixed beta-code-update-eqiad Jenkins job by cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002 | T325128

Standalone puppetmasters are also affected by this Git update:

$ git push -f project_puppetmaster HEAD:production
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote: fatal: detected dubious ownership in repository at '/var/lib/git/operations/puppet'
remote: To add an exception for this directory, call:
remote: 
remote: 	git config --global --add safe.directory /var/lib/git/operations/puppet

For WMCS Standalone puppetmasters I am not sure how it should behave.

I have discovered this morning the files are owned by gitpuppet and up until now I have always use root (using: sudo su - → cd /var/lib/git/operations/puppet → git ...).

If we instead use sudo -u gitpuppet git ... that would fix it (local files would probably require some chown -R gitpuppet):

hashar@integration-puppetmaster-02:/var/lib/git/operations/puppet$ sudo -u gitpuppet git log -n1 --oneline
b0f80ae195 (HEAD -> production, tag: snapshot-202212132333, origin/production, origin/HEAD) netmon: Remove the netmon1002 instance as passive node

Though surely we might want to find out a way to let people know about the gitpuppet user and use sudo -u gitpuppet instead of sudo.

Sort of related for standalone puppetmasters: T152059: role::puppetmaster::standalone clones Git repositories as gitpuppet, git-sync-upstream overwrites them as root

Switching from serviceops to SRE for greater visibility within the SRE team, this could break more things.

Mentioned in SAL (#wikimedia-releng) [2023-01-27T08:39:11Z] <hashar> devtools: fix git security warning on deployment server by cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002 on the puppet master | T325128

hashar opened https://gitlab.wikimedia.org/repos/releng/scap3-dev/-/merge_requests/12

Disable git safe.directory in deployment image

jnuche merged https://gitlab.wikimedia.org/repos/releng/scap3-dev/-/merge_requests/12

Disable git safe.directory in deployment image

The same issue occurs on the deployment target as reported by T330394:

nfraison@stat1008:/srv/deployment/analytics/hdfs-tools/deploy$ sudo /usr/bin/git -C /srv/deployment/analytics/hdfs-tools/deploy tag --points-at HEAD
fatal: detected dubious ownership in repository at '/srv/deployment/analytics/hdfs-tools/deploy-cache/revs/0c6e3ca61c094338d821ae7c73e244f1abb5b8bc'
To add an exception for this directory, call:

	git config --global --add safe.directory /srv/deployment/analytics/hdfs-tools/deploy-cache/revs/0c6e3ca61c094338d821ae7c73e244f1abb5b8bc

In T325128#8640737, @hashar wrote:

The same issue occurs on the deployment target as reported by T330394:

nfraison@stat1008:/srv/deployment/analytics/hdfs-tools/deploy$ sudo /usr/bin/git -C /srv/deployment/analytics/hdfs-tools/deploy tag --points-at HEAD
fatal: detected dubious ownership in repository at '/srv/deployment/analytics/hdfs-tools/deploy-cache/revs/0c6e3ca61c094338d821ae7c73e244f1abb5b8bc'
To add an exception for this directory, call:

	git config --global --add safe.directory /srv/deployment/analytics/hdfs-tools/deploy-cache/revs/0c6e3ca61c094338d821ae7c73e244f1abb5b8bc

And JFTR, got fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/891555

@hashar are there still things that need doing on this task? It looks like
https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ is now clear.

[asking because this is on the clinic duty dashboard]

In T325128#8667625, @MatthewVernon wrote:

@hashar are there still things that need doing on this task? It looks like
https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ is now clear.

[asking because this is on the clinic duty dashboard]

The immediate issue got addressed by cherry picking to beta cluster the patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002/ . It globally set the git config safe.directory='*' then I don't think we should set it (I commented on the patch), at least not wide open and we also have use case of executing commands as root which sounds scary if one adds some malicious hook to the git repos.

The issue also happened (and will happen) on a wide range of different systems and triggers as soon as the git package is upgraded on the hosts (typically if one reimage a server and uncarefully upgrade the package). We already had several tasks filed and since this one got filed first, I guess it can be reused as a tracking / parent task.

Part of the fix for the deployment servers and scap is https://gerrit.wikimedia.org/r/c/operations/puppet/+/891555 by @nfraison which is to have Puppet to run the git command as the deployment user rather than root. T330394#8640706 as details about implementing and testing the fix.

Change 868002 abandoned by Hashar:

[operations/puppet@production] scap: disable git safe.directory

Reason:

I have removed this patch from the deployment-prep Puppet master. The proper fix for the deployments server has been made by Icb3e1d3a98e66ec132411b4c0ade56347bc6ce48 via T330394

https://gerrit.wikimedia.org/r/868002

On deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud:

Notice: /Stage[main]/Git::Globalconfig/File[/etc/gitconfig.d/10-disable_safe_directory.gitconfig]/ensure: removed

The scap fix has been done in T330394. For production we still have to update git fleet wide and some bits have been done with @MoritzMuehlenhoff