Page MenuHomePhabricator

git: detected dubious ownership in repository at '/srv/mediawiki-staging'
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ worked at 6:04 UTC and broke at 6:13 UTC

Started by timer
Running as SYSTEM
Building remotely on deployment-deploy03 (BetaClusterBastion) in workspace /srv/jenkins/home/jenkins-deploy/workspace/beta-code-update-eqiad
No emails were triggered.
[beta-code-update-eqiad] $ /bin/bash -xe /tmp/jenkins3520384640553993295.sh
+ /usr/bin/scap prep auto
07:03:08 Started scap prep auto
07:03:08 Update https://gerrit.wikimedia.org/r/operations/mediawiki-config (master branch) in /srv/mediawiki-staging
07:03:08 Finished scap prep auto (duration: 00m 00s)
07:03:08 prep failed: <FailedCommand> Command 'git remote add origin https://gerrit.wikimedia.org/r/operations/mediawiki-config' failed with exit code 128;
stdout:

stderr:
fatal: detected dubious ownership in repository at '/srv/mediawiki-staging'
To add an exception for this directory, call:

	git config --global --add safe.directory /srv/mediawiki-staging

Build step 'Execute shell' marked build as failure
IRC notifier plugin: Sending notification to: #wikimedia-releng
No emails were triggered.
Finished: FAILURE

The beta cluster has the same setup has production which will thus be broken as soon as git is updated. On the deployment server:

deploy1002$ ls -lrdta /srv/mediawiki-staging
drwxrwsr-x 22 mwdeploy deployment 4096 Dec 13 09:10 /srv/mediawiki-staging/

Details

ReferenceSource BranchDest BranchAuthorTitle
repos/releng/scap3-dev!12disable-git-safe-directorymasterhasharDisable git safe.directory in deployment image
Customize query in GitLab

Event Timeline

I have looked at Gerrit and nothing significant merged at the time. Eventually I found out git package has been upgraded on the machine:

/var/log/apt/history.log
Start-Date: 2022-12-14  06:06:57
Commandline: /usr/bin/unattended-upgrade
Upgrade: git:amd64 (1:2.20.1-2+deb10u4, 1:2.20.1-2+deb10u5)
End-Date: 2022-12-14  06:07:01

Start-Date: 2022-12-14  06:07:02
Commandline: /usr/bin/unattended-upgrade
Upgrade: git-man:amd64 (1:2.20.1-2+deb10u4, 1:2.20.1-2+deb10u5)
End-Date: 2022-12-14  06:07:07

/srv/mediawiki-staging has a setuid bit:

      v
drwxrwsr-x 22 mwdeploy wikidev 4096 Dec 13 09:13 /srv/mediawiki-staging/
      ^
git (1:2.20.1-2+deb10u5) buster-security; urgency=high
  
  * Non-maintainer upload by the LTS Security Team.
  * CVE-2022-24765: Git not checking the ownership of directories in a
    local multi-user system when running commands specified in the local
    repository configuration.  This allows the owner of the repository to
    cause arbitrary commands to be executed by other users who access the
    repository.
  * The above introduces new 'safe.directory' checks which may cause
    regressions: allow opt-out of such checks with 'safe.directory=*'
  * CVE-2022-29187: an unsuspecting user could still be affected by the
    issue reported in CVE-2022-24765, for example when navigating as root
    into a shared tmp directory that is owned by them, but where an
    attacker could create a git repository.
  * CVE-2022-39253: exposure of sensitive information to a malicious
    actor. When performing a local clone (where the source and target of
    the clone are on the same volume), Git copies the contents of the
    source's `$GIT_DIR/objects` directory into the destination by either
    creating hardlinks to the source contents, or copying them (if
    hardlinks are disabled via `--no-hardlinks`). A malicious actor could
    convince a victim to clone a repository with a symbolic link pointing
    at sensitive information on the victim's machine.
  * CVE-2022-39260: `git shell` improperly uses an `int` to represent the
    number of entries in the array, allowing a malicious actor to
    intentionally overflow the return value, leading to arbitrary heap
    writes. Because the resulting array is then passed to `execv()`, it is
    possible to leverage this attack to gain remote code execution on a
    victim machine.

 -- Sylvain Beucler <beuc@debian.org>  Tue, 13 Dec 2022 15:14:23 +0100
hashar renamed this task from beta-code-update-eqiad: to git: detected dubious ownership in repository at '/srv/mediawiki-staging'.Dec 14 2022, 7:42 AM
hashar triaged this task as High priority.
hashar added a project: serviceops.
hashar updated the task description. (Show Details)

Another note is we have the git repository flagged as shared between user by setting core.sharedRepository=group. In this case, I argue the security check should still allow command if a user is a member of the group owning the .git directory.

The new safe.directory setting does not accept any wildcard setting. Since we can execute git commands in any repository below /srv/mediawiki-staging I am guessing the easiest is to disable that feature as instructed.

Change 868002 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] scap: disable git safe.directory

https://gerrit.wikimedia.org/r/868002

Mentioned in SAL (#wikimedia-releng) [2022-12-14T08:43:11Z] <hashar> deployment-prep: fixed beta-code-update-eqiad Jenkins job by cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002 | T325128

Standalone puppetmasters are also affected by this Git update:

$ git push -f project_puppetmaster HEAD:production
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote: fatal: detected dubious ownership in repository at '/var/lib/git/operations/puppet'
remote: To add an exception for this directory, call:
remote: 
remote: 	git config --global --add safe.directory /var/lib/git/operations/puppet

For WMCS Standalone puppetmasters I am not sure how it should behave.

I have discovered this morning the files are owned by gitpuppet and up until now I have always use root (using: sudo su -cd /var/lib/git/operations/puppetgit ...).

If we instead use sudo -u gitpuppet git ... that would fix it (local files would probably require some chown -R gitpuppet):

hashar@integration-puppetmaster-02:/var/lib/git/operations/puppet$ sudo -u gitpuppet git log -n1 --oneline
b0f80ae195 (HEAD -> production, tag: snapshot-202212132333, origin/production, origin/HEAD) netmon: Remove the netmon1002 instance as passive node

Though surely we might want to find out a way to let people know about the gitpuppet user and use sudo -u gitpuppet instead of sudo.

akosiaris added a subscriber: akosiaris.

Switching from serviceops to SRE for greater visibility within the SRE team, this could break more things.

I have cherry-picked https://gerrit.wikimedia.org/r/868002 to fix the Beta-Cluster-Infrastructure automatic deployment. It would be needed for production as a prerequisite to the git upgrade as well as some extra fix needed for the puppet master.

I am unassigning myself since the immediate breakage got fixed and there is a bit more work to do (deal with puppetmaster and actually upgrade git fleet wide).

Mentioned in SAL (#wikimedia-releng) [2023-01-27T08:39:11Z] <hashar> devtools: fix git security warning on deployment server by cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002 on the puppet master | T325128

The same issue occurs on the deployment target as reported by T330394:

nfraison@stat1008:/srv/deployment/analytics/hdfs-tools/deploy$ sudo /usr/bin/git -C /srv/deployment/analytics/hdfs-tools/deploy tag --points-at HEAD
fatal: detected dubious ownership in repository at '/srv/deployment/analytics/hdfs-tools/deploy-cache/revs/0c6e3ca61c094338d821ae7c73e244f1abb5b8bc'
To add an exception for this directory, call:

	git config --global --add safe.directory /srv/deployment/analytics/hdfs-tools/deploy-cache/revs/0c6e3ca61c094338d821ae7c73e244f1abb5b8bc

The same issue occurs on the deployment target as reported by T330394:

nfraison@stat1008:/srv/deployment/analytics/hdfs-tools/deploy$ sudo /usr/bin/git -C /srv/deployment/analytics/hdfs-tools/deploy tag --points-at HEAD
fatal: detected dubious ownership in repository at '/srv/deployment/analytics/hdfs-tools/deploy-cache/revs/0c6e3ca61c094338d821ae7c73e244f1abb5b8bc'
To add an exception for this directory, call:

	git config --global --add safe.directory /srv/deployment/analytics/hdfs-tools/deploy-cache/revs/0c6e3ca61c094338d821ae7c73e244f1abb5b8bc

And JFTR, got fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/891555

@hashar are there still things that need doing on this task? It looks like
https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ is now clear.

[asking because this is on the clinic duty dashboard]

@hashar are there still things that need doing on this task? It looks like
https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ is now clear.

[asking because this is on the clinic duty dashboard]

The immediate issue got addressed by cherry picking to beta cluster the patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002/ . It globally set the git config safe.directory='*' then I don't think we should set it (I commented on the patch), at least not wide open and we also have use case of executing commands as root which sounds scary if one adds some malicious hook to the git repos.

The issue also happened (and will happen) on a wide range of different systems and triggers as soon as the git package is upgraded on the hosts (typically if one reimage a server and uncarefully upgrade the package). We already had several tasks filed and since this one got filed first, I guess it can be reused as a tracking / parent task.

Part of the fix for the deployment servers and scap is https://gerrit.wikimedia.org/r/c/operations/puppet/+/891555 by @nfraison which is to have Puppet to run the git command as the deployment user rather than root. T330394#8640706 as details about implementing and testing the fix.

Change 868002 abandoned by Hashar:

[operations/puppet@production] scap: disable git safe.directory

Reason:

I have removed this patch from the deployment-prep Puppet master. The proper fix for the deployments server has been made by Icb3e1d3a98e66ec132411b4c0ade56347bc6ce48 via T330394

https://gerrit.wikimedia.org/r/868002

On deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud:

Notice: /Stage[main]/Git::Globalconfig/File[/etc/gitconfig.d/10-disable_safe_directory.gitconfig]/ensure: removed

The scap fix has been done in T330394. For production we still have to update git fleet wide and some bits have been done with @MoritzMuehlenhoff

hashar claimed this task.

For other use case, looks like that will be done as part {T335354}

I am closing this one since it was primarily for the Beta-Cluster-Infrastructure