Page MenuHomePhabricator

[Trebuchet] failed to update scap version on minions
Closed, ResolvedPublic

Description

14:23 < bd808|dep> mutante: Trying to get deploy sync scap from tin with

result of "0/230 minions completed fetch"

Version: wmf-deployment
Severity: blocker

Details

Reference
bz65542

Event Timeline

bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 3:27 AM
bzimport added a project: Deployments.
bzimport set Reference to bz65542.
bzimport added a subscriber: Unknown Object (MLST).

Setting it to Immediate importance.

Opsen: please triage/assign to whoever will debug/diagnose.

14:39 < bd808> mutante: I would say that May 7 22:49 was the last time

`git deploy sync` was used on the scap repo

tin:/srv/deployment/scap/scap (git master)
bd808$ git deploy sync
Repo: scap/scap
Tag: scap/scap-sync-20140520-181227

0/230 minions completed fetch
Continue? ([d]etailed/[C]oncise report,[y]es,[n]o,[r]etry): r
Continue? ([d]etailed/[C]oncise report,[y]es,[n]o,[r]etry): r
Continue? ([d]etailed/[C]oncise report,[y]es,[n]o,[r]etry): d
Repo: scap/scap
Tag: scap/scap-sync-20140520-181227

0/230 minions completed fetch

Details:

mw1068.eqiad.wmnet:

fetch status: 10 [started: 0 mins ago, last-return: 0 mins ago]

[..snip..]
Continue? ([d]etailed/[C]oncise report,[y]es,[n]o,[r]etry): r
Continue? ([d]etailed/[C]oncise report,[y]es,[n]o,[r]etry): c
Repo: scap/scap
Tag: scap/scap-sync-20140520-181227

0/230 minions completed fetch
Continue? ([d]etailed/[C]oncise report,[y]es,[n]o,[r]etry): n
Not continuing to checkout phase. A deployment is still underway, please finish,
sync, or abort.
tin:/srv/deployment/scap/scap (git master)
bd808$ git deploy abort
Deployment aborted.

Last deployed tag seems to be "scap/scap-sync-20140507-224859"

$ ls -ld /srv/deployment/scap/scap
drwxrws--- 6 trebuchet wikidev 4096 May 1 20:07 /srv/deployment/scap/scap/

Directory permissions are borked. The apache user needs to be able to read the directory in order to serve it to the minions. Should be "drwxrwsr-x".

14:59 < Ryan_Lane> chmod o+rx /srv/deployment/scap/scap/
14:59 < mutante> already done

(after test of git-deploy of scap)
15:01 < bd808> greg-g: \o/

Directory permissions were probably broken initially because in the current setup tin is acting as both the deploy server and a minion for the scap deploy. When the minion gets the fetch/checkout commands from salt it runs as root. I sort of wondered if this would cause problems and I think now I have my answer.

We can either try to rearrange the way that classes are defined in operations/puppet.git to eliminate this problem or figure out how to make trebuchet aware of the edge case and avoid updating the deployment server if it is also in the minions list.

greg lowered the priority of this task from Unbreak Now! to Medium.Jan 8 2015, 5:36 PM
greg moved this task from To Triage to Done on the Deployments board.
greg raised the priority of this task from Medium to Unbreak Now!.Jan 8 2015, 5:42 PM
greg set Security to None.