Page MenuHomePhabricator

Deploy error: insufficient permission for adding an object to repository database .git/objects
Open, NormalPublic

Description

Happened today to @Addshore while SWATting. Looks like some directories under objects for /srv/mediawiki-staging/php-1.31.0-wmf.20 git repo don't have group write permissions, thus failing a deploy.

I manually fixed tin with chmod -R g+w /srv/mediawiki-staging/php-1.31.0-wmf.20/.git though it would be nice to know what caused this.

Looking at the list of directories with wrong perms there seem to be a pattern:

naos:/srv/mediawiki-staging/php-1.31.0-wmf.20/.git$ find . ! -perm -g+w -type d -ls
2884346    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/08
2884391    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/3d
2884374    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/2a
2884483    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/a3
2884506    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/c0
2884470    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/90
2884401    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/45
2884523    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:00 ./objects/ec
2884492    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ab
2884422    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/5b
2884458    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/7f
2884505    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/bc
2884376    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/2c
2884466    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/88
2884369    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/26
2884513    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/d5
2884488    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/a8
2884526    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/f1
2884451    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/79
2884487    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/a6
2884354    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/14
2884494    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ac
2884393    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/3e
2884426    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/5e
2884468    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/8a
2884474    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/97
2884479    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/9e
2884521    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/e8
2884460    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/80
2884522    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/eb
2884433    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:00 ./objects/60
2884472    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/95
2884372    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/28
2884519    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/e3
2884412    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/4f
2884414    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/53
2884464    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/86
2884342    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/01
2884348    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/0c
2884396    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/3f
2884383    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/34
2884442    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/70
2884362    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/1b
2884477    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/9c
2884420    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/59
2884500    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/b5
2884345    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/07
2884424    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/5c
2884446    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/74
2884462    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/82
2884518    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/df
2884435    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/63
2884431    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/5f
2884508    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/c7
2884528    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/fc
2884380    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/32
2884516    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/dc
2884525    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/f0
2884398    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/44
2884350    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/0e
2884358    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/18
2884360    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/1a
2884403    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/47
2884490    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/a9
2884509    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/c9
2884481    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/a2
2884517    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/dd
2884502    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ba
2884378    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/30
2884507    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/c5
2884496    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/af
2884385    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/38
2884453    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/7a
2884512    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/d1
2884529    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ff
2884524    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ef
2884387    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/3a
2884449    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/78
2884514    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/d7
2884352    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/10
2884498    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/b1
2884520    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/e4
2884504    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/bb
2884410    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/4d
2884527    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/f2
2884418    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/58
2884510    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ca
2884455    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/7b
2884365    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/20
2884437    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/64
2884389    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/3b
2884511    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/cc
2884408    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/4c
2884344    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/04
2884485    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/a5
2884367    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/25
2884356    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/17
2884444    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/71
2884515    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/db
2884416    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:00 ./objects/54
2884439    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/6f
2884343    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/02
2884406    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:00 ./objects/4a
6160396    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/refs/remotes/origin/wmf
2884284    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/objects/2c
2884340    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/objects/be
2884298    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/objects/58
2884341    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/objects/fe
2884292    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/objects/4a
6160395    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/logs/refs/remotes/origin/wmf
6160394    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/MobileFrontend/refs/remotes/origin/wmf
6160392    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/MobileFrontend/logs/refs/remotes/origin/wmf
naos:/srv/mediawiki-staging/php-1.31.0-wmf.20/.git$

Perhaps an umask too restrictive.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 12 2018, 2:29 PM
fgiunchedi triaged this task as Normal priority.Feb 12 2018, 2:31 PM
fgiunchedi added a project: Operations.

I'll hold back fixing naos too to leave time Release-Engineering-Team to inspect the situation.

fgiunchedi renamed this task from insufficient permission for adding an object to repository database .git/objects to Deploy error: insufficient permission for adding an object to repository database .git/objects.Feb 12 2018, 2:31 PM
demon added a subscriber: demon.EditedFeb 12 2018, 4:08 PM

This usually happens for one of two reasons

  1. A root user has come along and stolen ownership to root. This shouldn't happen often in practice, as we've got icinga alerting us (and scap itself yells if you try to run it as geteuid() == 0)
  2. A user has a busted umask

Either way, it's one of those "fix it and move on" scenarios. Go ahead and fix naos, please :)

This usually happens for one of two reasons

  1. A root user has come along and stolen ownership to root. This shouldn't happen often in practice, as we've got icinga alerting us (and scap itself yells if you try to run it as geteuid() == 0)
  2. A user has a busted umask

Either way, it's one of those "fix it and move on" scenarios. Go ahead and fix naos, please :)

Something or someone has already fixed naos !

scap IMO should be warning or refuse to continue if the user has a busted umask to avoid similar situation in the future

demon added a comment.Feb 13 2018, 4:31 PM

scap IMO should be warning or refuse to continue if the user has a busted umask to avoid similar situation in the future

Well, it's not scap's fault....it's the git operations someone does before running scap. I *guess* scap could warn about it so we don't sync it to naos or other locations...

scap IMO should be warning or refuse to continue if the user has a busted umask to avoid similar situation in the future

Well, it's not scap's fault....it's the git operations someone does before running scap. I *guess* scap could warn about it so we don't sync it to naos or other locations...

Yes I think a warning when something is wrong is warranted, especially so that the situation can be acted upon when it happens as opposed to the next deploy by a different user.

Mentioned in SAL (#wikimedia-operations) [2018-02-13T17:20:15Z] <demon@tin> Synchronized README: forcing git config sync, setting core.sharedRepository=group, T187076 (duration: 01m 12s)

FWIW, because we keep the masters in sync as part of scap, so unfortunately (maybe :)) naos was fixed as soon as the sync happened. (details: https://github.com/wikimedia/puppet/blob/production/modules/scap/files/scap-master-sync)

I think a warning might be good, although this situation was created by git and not by scap. Maybe an nrpe check similar to the root-owned-files check? @demon suggested a post-checkout hook. Both seem fine. Warning in scap maybe the wrong place: git wrangling may not be done by the current deployer (although that is typically how it's done).

Setting the setgid bit as part of branch prep ( https://github.com/wikimedia/operations-mediawiki-config/blob/master/scap/plugins/prep.py) might prevent the effects of a bad umask all together. Similar to what we do for all repositories below /srv/deployment/ via https://github.com/wikimedia/puppet/blob/production/modules/scap/lib/puppet/provider/scap_source/default.rb#L80-L106

Some experimentation is needed with the setgid bit part.