Page MenuHomePhabricator
Create Task
Maniphest T187076

Deploy error: insufficient permission for adding an object to repository database .git/objects
Closed, ResolvedPublic
Actions

Description

Happened today to @Addshore while SWATting. Looks like some directories under objects for /srv/mediawiki-staging/php-1.31.0-wmf.20 git repo don't have group write permissions, thus failing a deploy.

I manually fixed tin with chmod -R g+w /srv/mediawiki-staging/php-1.31.0-wmf.20/.git though it would be nice to know what caused this.

Looking at the list of directories with wrong perms there seem to be a pattern:

naos:/srv/mediawiki-staging/php-1.31.0-wmf.20/.git$ find . ! -perm -g+w -type d -ls
2884346    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/08
2884391    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/3d
2884374    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/2a
2884483    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/a3
2884506    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/c0
2884470    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/90
2884401    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/45
2884523    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:00 ./objects/ec
2884492    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ab
2884422    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/5b
2884458    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/7f
2884505    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/bc
2884376    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/2c
2884466    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/88
2884369    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/26
2884513    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/d5
2884488    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/a8
2884526    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/f1
2884451    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/79
2884487    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/a6
2884354    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/14
2884494    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ac
2884393    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/3e
2884426    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/5e
2884468    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/8a
2884474    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/97
2884479    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/9e
2884521    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/e8
2884460    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/80
2884522    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/eb
2884433    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:00 ./objects/60
2884472    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/95
2884372    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/28
2884519    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/e3
2884412    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/4f
2884414    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/53
2884464    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/86
2884342    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/01
2884348    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/0c
2884396    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/3f
2884383    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/34
2884442    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/70
2884362    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/1b
2884477    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/9c
2884420    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/59
2884500    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/b5
2884345    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/07
2884424    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/5c
2884446    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/74
2884462    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/82
2884518    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/df
2884435    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/63
2884431    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/5f
2884508    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/c7
2884528    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/fc
2884380    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/32
2884516    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/dc
2884525    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/f0
2884398    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/44
2884350    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/0e
2884358    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/18
2884360    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/1a
2884403    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/47
2884490    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/a9
2884509    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/c9
2884481    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/a2
2884517    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/dd
2884502    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ba
2884378    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/30
2884507    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/c5
2884496    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/af
2884385    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/38
2884453    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/7a
2884512    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/d1
2884529    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ff
2884524    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ef
2884387    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/3a
2884449    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/78
2884514    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/d7
2884352    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/10
2884498    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/b1
2884520    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/e4
2884504    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/bb
2884410    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/4d
2884527    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/f2
2884418    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./objects/58
2884510    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/ca
2884455    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/7b
2884365    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/20
2884437    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/64
2884389    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/3b
2884511    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/cc
2884408    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/4c
2884344    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/04
2884485    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/a5
2884367    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/25
2884356    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/17
2884444    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/71
2884515    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/db
2884416    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:00 ./objects/54
2884439    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/6f
2884343    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 21:44 ./objects/02
2884406    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:00 ./objects/4a
6160396    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/refs/remotes/origin/wmf
2884284    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/objects/2c
2884340    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/objects/be
2884298    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/objects/58
2884341    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/objects/fe
2884292    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/objects/4a
6160395    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/TextExtracts/logs/refs/remotes/origin/wmf
6160394    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/MobileFrontend/refs/remotes/origin/wmf
6160392    4 drwxr-sr-x   2 tgr      wikidev      4096 Feb  9 22:37 ./modules/extensions/MobileFrontend/logs/refs/remotes/origin/wmf
naos:/srv/mediawiki-staging/php-1.31.0-wmf.20/.git$

Perhaps an umask too restrictive.

Event Timeline

fgiunchedi created this task.Feb 12 2018, 2:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 12 2018, 2:29 PM
fgiunchedi triaged this task as Medium priority.Feb 12 2018, 2:31 PM
fgiunchedi added a project: SRE.

I'll hold back fixing naos too to leave time Release-Engineering-Team to inspect the situation.

fgiunchedi renamed this task from insufficient permission for adding an object to repository database .git/objects to Deploy error: insufficient permission for adding an object to repository database .git/objects.Feb 12 2018, 2:31 PM
zeljkofilipin awarded a token.Feb 12 2018, 2:33 PM
zeljkofilipin subscribed.
• demon subscribed.EditedFeb 12 2018, 4:08 PM

This usually happens for one of two reasons

  1. A root user has come along and stolen ownership to root. This shouldn't happen often in practice, as we've got icinga alerting us (and scap itself yells if you try to run it as geteuid() == 0)
  2. A user has a busted umask

Either way, it's one of those "fix it and move on" scenarios. Go ahead and fix naos, please :)

fgiunchedi added a comment.Feb 13 2018, 3:01 PM
In T187076#3963935, @demon wrote:

This usually happens for one of two reasons

  1. A root user has come along and stolen ownership to root. This shouldn't happen often in practice, as we've got icinga alerting us (and scap itself yells if you try to run it as geteuid() == 0)
  2. A user has a busted umask

Either way, it's one of those "fix it and move on" scenarios. Go ahead and fix naos, please :)

Something or someone has already fixed naos !

scap IMO should be warning or refuse to continue if the user has a busted umask to avoid similar situation in the future

• demon added a comment.Feb 13 2018, 4:31 PM
In T187076#3967607, @fgiunchedi wrote:

scap IMO should be warning or refuse to continue if the user has a busted umask to avoid similar situation in the future

Well, it's not scap's fault....it's the git operations someone does before running scap. I *guess* scap could warn about it so we don't sync it to naos or other locations...

fgiunchedi added a comment.Feb 13 2018, 4:50 PM
In T187076#3967985, @demon wrote:
In T187076#3967607, @fgiunchedi wrote:

scap IMO should be warning or refuse to continue if the user has a busted umask to avoid similar situation in the future

Well, it's not scap's fault....it's the git operations someone does before running scap. I *guess* scap could warn about it so we don't sync it to naos or other locations...

Yes I think a warning when something is wrong is warranted, especially so that the situation can be acted upon when it happens as opposed to the next deploy by a different user.

Stashbot subscribed.Feb 13 2018, 5:20 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-13T17:20:15Z] <demon@tin> Synchronized README: forcing git config sync, setting core.sharedRepository=group, T187076 (duration: 01m 12s)

thcipriani subscribed.Feb 13 2018, 5:20 PM

FWIW, because we keep the masters in sync as part of scap, so unfortunately (maybe :)) naos was fixed as soon as the sync happened. (details: https://github.com/wikimedia/puppet/blob/production/modules/scap/files/scap-master-sync)

I think a warning might be good, although this situation was created by git and not by scap. Maybe an nrpe check similar to the root-owned-files check? @demon suggested a post-checkout hook. Both seem fine. Warning in scap maybe the wrong place: git wrangling may not be done by the current deployer (although that is typically how it's done).

Setting the setgid bit as part of branch prep ( https://github.com/wikimedia/operations-mediawiki-config/blob/master/scap/plugins/prep.py) might prevent the effects of a bad umask all together. Similar to what we do for all repositories below /srv/deployment/ via https://github.com/wikimedia/puppet/blob/production/modules/scap/lib/puppet/provider/scap_source/default.rb#L80-L106

Some experimentation is needed with the setgid bit part.

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:46 PM
dancy closed this task as Resolved.Aug 16 2022, 9:10 PM
dancy claimed this task.
dancy subscribed.

Closing due to age.

zeljkofilipin unsubscribed.Aug 23 2022, 10:47 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL