Page MenuHomePhabricator
Create Task
Maniphest T103441

Corrupt /srv/deployment/scap/scap checkouts on WMF prod cluster
Closed, ResolvedPublic
Actions

Description

Every scap and sync-* operation on the WMF prod cluster is emitting soft failure warnings about "@Error: access denied to common from localhost (127.0.0.1)" from mw2187 and mw2080. The Trebuchet managed /srv/deployment/scap/scap git clones on these hosts are corrupted and only partially checked out.

@chasemp ran salt-call deploy.checkout 'scap/scap' on mw2080.codfw.wmnet and saw this failure description:

[16:31]  <  chasemp>	Command u'/usr/bin/git checkout --force --quiet tags/scap/scap-sync-20150622-214637' failed with return code: 128
[16:31]  <  chasemp>	[ERROR   ] output: error: object file .git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a is empty
[16:31]  <  chasemp>	error: object file .git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a is empty
[16:31]  <  chasemp>	fatal: loose object 57a4ca75e191a778f80d06bf1bc331d7db5d7f6a (stored in .git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a) is corrupt
[16:31]  <  chasemp>	local:

When I updated scap via Trebuchet I logged fetch failures from mw1222, mw2086 and virt1000 followed by checkout failures from mw1104, mw1222, mw2009, mw2011, mw2021, mw2028, mw2031, mw2034, mw2069, mw2076, mw2080, mw2086, mw2095, mw2099, mw2120, mw2127, mw2131, mw2136, mw2170, mw2187, mw2189, mw2197 and virt1000.

The fix would be to either manually repair the git clones on these hosts or to remove /srv/deployment/scap/scap entirely and have Trebuchet re-clone the repo via salt-call deploy.fetch 'scap/scap' && salt-call deploy.checkout 'scap/scap'

Event Timeline

bd808 created this task.Jun 22 2015, 11:50 PM
bd808 raised the priority of this task from to Needs Triage.
bd808 updated the task description. (Show Details)
bd808 added projects: acl*sre-team, Deployments.
bd808 added subscribers: bd808, • chasemp.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 22 2015, 11:50 PM
bd808 triaged this task as High priority.Jun 22 2015, 11:51 PM
• chasemp added a project: Release-Engineering-Team.Jun 22 2015, 11:54 PM
• chasemp set Security to None.
fgiunchedi subscribed.Jun 23 2015, 12:06 AM

so, audit time

# salt  --output=txt mw2*.codfw.wmnet cmd.run 'find /srv/deployment/scap/scap/.git -size 0 -ls'
mw2069.codfw.wmnet: 9045165    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/9d/4e17073a4825b2740adb284eaf61ec0193b56e
mw2076.codfw.wmnet: 5243816    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2021.codfw.wmnet: 13769770    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/c8/0507cd77c8c815877ea2d81dfa2aa3fc78dfa0
mw2011.codfw.wmnet: 6164046    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2031.codfw.wmnet: 5636634    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2170.codfw.wmnet: 26739629    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2136.codfw.wmnet: 27133075    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/2f/a6cea97900283be5e0ca68ac19b60bfc1c4086
mw2009.codfw.wmnet: 9570382    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2131.codfw.wmnet: 525229    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2197.codfw.wmnet: 22676662    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/2f/a6cea97900283be5e0ca68ac19b60bfc1c4086
mw2197.codfw.wmnet: 22676658    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/b4/a4348804a8f905e738cdad2f8b0c55561294d0
mw2197.codfw.wmnet: 22676660    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/80/0fe361e10c98f4dc88572794cf437a24a359a0
mw2197.codfw.wmnet: 22676528    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/d6/28c73a4104d55679e162b2f3e398db54714e2f
mw2120.codfw.wmnet: 1069275    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2189.codfw.wmnet: 13894560    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2028.codfw.wmnet: 4066169    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/9d/4e17073a4825b2740adb284eaf61ec0193b56e
mw2080.codfw.wmnet: 14685895    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2034.codfw.wmnet: 9177021    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2099.codfw.wmnet: 11535253    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2187.codfw.wmnet: 14942838    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2127.codfw.wmnet: 24642167    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/57/a4ca75e191a778f80d06bf1bc331d7db5d7f6a
mw2086.codfw.wmnet: 24379571    0 -rw-r--r--   1 root     root            0 Jun 22 23:48 /srv/deployment/scap/scap/.git/FETCH_HEAD
mw2086.codfw.wmnet: 24905380    0 -rw-r--r--   1 root     root            0 Jun 22 23:48 /srv/deployment/scap/scap/.git/objects/a1/1dfacbf4266109e18c332f4d012261fff0a102.temp
mw2086.codfw.wmnet: 24905465    0 -rw-r--r--   1 root     root            0 Jun 22 23:48 /srv/deployment/scap/scap/.git/objects/9d/4e17073a4825b2740adb284eaf61ec0193b56e.temp
mw2086.codfw.wmnet: 24905469    0 -rw-r--r--   1 root     root            0 Jun 22 23:48 /srv/deployment/scap/scap/.git/objects/c8/0507cd77c8c815877ea2d81dfa2aa3fc78dfa0.temp
mw2086.codfw.wmnet: 24905454    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/90/fcaa442e1a41ab9731421973c773885fb9cc39
mw2139.codfw.wmnet: 20449447    0 -rw-r--r--   1 root     root            0 Jun 22 21:47 /srv/deployment/scap/scap/.git/objects/bc/d0850a2dd87325241136358f3f6bfde0e30083
root@palladium:~# salt  --output=txt mw1*.eqiad.wmnet cmd.run 'find /srv/deployment/scap/scap/.git -size 0 -ls'
mw1104.eqiad.wmnet: 12848287    0 -rw-r--r--   1 root     root            0 Mar 17 21:04 /srv/deployment/scap/scap/.git/objects/01/98c33279c2724b8791c7236679d7880f3326df
mw1113.eqiad.wmnet: 8398894    0 -rw-r--r--   1 root     root            0 Mar 17 21:04 /srv/deployment/scap/scap/.git/objects/15/4bf2593c4aa9aa3bb329107b2c5da8002082e3
fgiunchedi added a comment.Jun 23 2015, 12:14 AM

tin logs for mw2086 around that time

tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:22:47:56 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.1" 200 333 "-" "Ruby"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:22:47:56 +0000] "GET /scap/scap/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 3262 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:22:47:56 +0000] "GET /scap/scap/.git/HEAD HTTP/1.1" 200 232 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:22:47:58 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.0" 200 352 "-" "Python-urllib/1.17"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:22:47:58 +0000] "GET /scap/scap/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 3262 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:22:47:58 +0000] "GET /scap/scap/.git/HEAD HTTP/1.1" 200 232 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:22:47:58 +0000] "GET /scap/scap/.git/objects/9d/4e17073a4825b2740adb284eaf61ec0193b56e HTTP/1.1" 200 242 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:22:48:00 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.0" 200 352 "-" "Python-urllib/1.17"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:08:56 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.1" 200 333 "-" "Ruby"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:08:56 +0000] "GET /scap/scap/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 3262 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:08:56 +0000] "GET /scap/scap/.git/HEAD HTTP/1.1" 200 232 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:08:58 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.0" 200 352 "-" "Python-urllib/1.17"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:08:58 +0000] "GET /scap/scap/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 3262 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:08:58 +0000] "GET /scap/scap/.git/HEAD HTTP/1.1" 200 232 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:08:58 +0000] "GET /scap/scap/.git/objects/9d/4e17073a4825b2740adb284eaf61ec0193b56e HTTP/1.1" 200 242 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:09:00 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.0" 200 352 "-" "Python-urllib/1.17"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:27:55 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.1" 200 333 "-" "Ruby"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:27:56 +0000] "GET /scap/scap/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 3262 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:27:56 +0000] "GET /scap/scap/.git/HEAD HTTP/1.1" 200 232 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:27:58 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.0" 200 352 "-" "Python-urllib/1.17"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:27:58 +0000] "GET /scap/scap/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 3262 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:27:58 +0000] "GET /scap/scap/.git/HEAD HTTP/1.1" 200 232 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:27:58 +0000] "GET /scap/scap/.git/objects/9d/4e17073a4825b2740adb284eaf61ec0193b56e HTTP/1.1" 200 242 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:28:00 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.0" 200 352 "-" "Python-urllib/1.17"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:48:26 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.1" 200 333 "-" "Ruby"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:48:26 +0000] "GET /scap/scap/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 3262 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:48:26 +0000] "GET /scap/scap/.git/HEAD HTTP/1.1" 200 232 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:48:28 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.0" 200 352 "-" "Python-urllib/1.17"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:48:28 +0000] "GET /scap/scap/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 3262 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:48:28 +0000] "GET /scap/scap/.git/HEAD HTTP/1.1" 200 232 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:48:28 +0000] "GET /scap/scap/.git/objects/9d/4e17073a4825b2740adb284eaf61ec0193b56e HTTP/1.1" 200 242 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [22/Jun/2015:23:48:30 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.0" 200 352 "-" "Python-urllib/1.17"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [23/Jun/2015:00:07:34 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.1" 200 333 "-" "Ruby"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [23/Jun/2015:00:07:34 +0000] "GET /scap/scap/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 3262 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [23/Jun/2015:00:07:34 +0000] "GET /scap/scap/.git/HEAD HTTP/1.1" 200 232 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [23/Jun/2015:00:07:36 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.0" 200 352 "-" "Python-urllib/1.17"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [23/Jun/2015:00:07:36 +0000] "GET /scap/scap/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 3262 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [23/Jun/2015:00:07:36 +0000] "GET /scap/scap/.git/HEAD HTTP/1.1" 200 232 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [23/Jun/2015:00:07:36 +0000] "GET /scap/scap/.git/objects/9d/4e17073a4825b2740adb284eaf61ec0193b56e HTTP/1.1" 200 242 "-" "git/1.9.1"
tin.eqiad.wmnet_access.log:2620:0:860:102:92b1:1cff:fe25:954d - - [23/Jun/2015:00:07:38 +0000] "GET /scap/scap/.git/deploy/deploy HTTP/1.0" 200 352 "-" "Python-urllib/1.17"
tin.eqiad.wmnet_error.log:[Mon Jun 22 21:47:09 2015] [error] [client 2620:0:860:102:92b1:1cff:fe25:954d] File does not exist: /srv/deployment/scap/scap/.git/objects/info/http-alternates
tin.eqiad.wmnet_error.log:[Mon Jun 22 21:47:09 2015] [error] [client 2620:0:860:102:92b1:1cff:fe25:954d] File does not exist: /srv/deployment/scap/scap/.git/objects/info/alternates
fgiunchedi added a comment.Jun 23 2015, 12:25 AM

I've removed the zero-size files and ran deploy.fetch + deploy.checkout on those machines

bd808 added a comment.Jun 23 2015, 12:40 AM

Some spot checked servers were still missing files in the local checkouts following @fgiunchedi's forced updates. I did a no-op trebuchet deploy to see if that would fix things. All hosts except virt1000 completed the fetch phase but 20 hosts failed the checkout phase:

463/483 minions completed checkout

Details:

mw2197.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2069.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2127.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2189.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2021.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2131.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2076.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2011.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2099.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2009.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2170.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw1104.eqiad.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2187.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2095.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2028.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2031.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2034.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
mw2120.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
virt1000.wikimedia.org:
        checkout status: 0 [started: 729 mins ago, last-return: 729 mins ago]
mw2136.codfw.wmnet:
        checkout status: 30 [started: 0 mins ago, last-return: 0 mins ago]
• MZMcBride subscribed.Jun 23 2015, 12:42 AM
fgiunchedi added a comment.Jun 23 2015, 12:44 AM

running deploy.checkout manually on mw2197 yields a few sha1 files missing

mw2197:~$ sudo salt-call deploy.checkout 'scap/scap'
[INFO    ] Executing command '/usr/bin/git describe --always --tag' in directory '/srv/deployment/scap/scap'
[INFO    ] Executing command u'/usr/bin/git checkout --force --quiet tags/scap/scap-sync-20150623-003433' in directory '/srv/deployment/scap/scap'
[ERROR   ] Command u'/usr/bin/git checkout --force --quiet tags/scap/scap-sync-20150623-003433' failed with return code: 1
[ERROR   ] output: error: unable to read sha1 file of scap/__init__.py (d628c73a4104d55679e162b2f3e398db54714e2f)
error: unable to read sha1 file of scap/config.py (b4a4348804a8f905e738cdad2f8b0c55561294d0)
error: unable to read sha1 file of scap/main.py (800fe361e10c98f4dc88572794cf437a24a359a0)
error: unable to read sha1 file of scap/tasks.py (2fa6cea97900283be5e0ca68ac19b60bfc1c4086)
local:
    ----------
    dependencies:
    repo:
        scap/scap
    status:
        30
    tag:
        scap/scap-sync-20150623-003433
fgiunchedi added a comment.Jun 23 2015, 12:50 AM

that didn't work as we expected, I've removed /srv/deployment/scap from the affected machines and ran deploy.fetch + deploy.checkout again

bd808 closed this task as Resolved.Jun 23 2015, 12:57 AM
bd808 claimed this task.

A third trebuchet run showed only virt1000 failing and SAL lists it as shut down as of today.

greg moved this task from INBOX to Done on the Release-Engineering-Team board.Jul 28 2015, 8:38 PM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 28 2015, 8:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL