Page MenuHomePhabricator

New, mysterious scap failure
Closed, InvalidPublic

Description

I don't see the issue here. But it fails.

awight@deployment-tin:/srv/deployment/ores/deploy$ scap deploy -v -f "Deploy ORES service b67bba7"
17:07:14 <Command u'/usr/bin/git show -s --format=%ct b67bba77acb7c0ffc678201c9f3f54f198da6650'>: starting process
17:07:14 <Command u'/usr/bin/git show -s --format=%ct b67bba77acb7c0ffc678201c9f3f54f198da6650', pid 21369>: process started
17:07:14 <Command u'/usr/bin/git show -s --format=%ct b67bba77acb7c0ffc678201c9f3f54f198da6650', pid 21369>: process completed
17:07:14 <Command u'/usr/bin/git show -s --format=%ct b67bba77acb7c0ffc678201c9f3f54f198da6650', pid 21369>: process completed
17:07:14 <Command u'/usr/bin/git show -s --format=%ct b67bba77acb7c0ffc678201c9f3f54f198da6650', pid 21369>: process completed
17:07:14 <Command u'/usr/bin/git ls-remote --get-url'>: starting process
17:07:14 <Command u'/usr/bin/git ls-remote --get-url', pid 21373>: process started
17:07:14 <Command u'/usr/bin/git ls-remote --get-url', pid 21373>: process completed
17:07:14 <Command u'/usr/bin/git ls-remote --get-url', pid 21373>: process completed
17:07:14 <Command u'/usr/bin/git ls-remote --get-url', pid 21373>: process completed
17:07:14 Started deploy [ores/deploy@b67bba7]
17:07:14 <Command u'/usr/bin/git tag --list scap/sync/2017-12-13/*'>: starting process
17:07:14 <Command u'/usr/bin/git tag --list scap/sync/2017-12-13/*', pid 21377>: process started
17:07:14 <Command u'/usr/bin/git tag --list scap/sync/2017-12-13/*', pid 21377>: process completed
17:07:14 <Command u'/usr/bin/git tag --list scap/sync/2017-12-13/*', pid 21377>: process completed
17:07:14 <Command u'/usr/bin/git tag --list scap/sync/2017-12-13/*', pid 21377>: process completed
17:07:14 <Command u'/usr/bin/git rev-parse --verify HEAD'>: starting process
17:07:14 <Command u'/usr/bin/git rev-parse --verify HEAD', pid 21381>: process started
17:07:14 <Command u'/usr/bin/git rev-parse --verify HEAD', pid 21381>: process completed
17:07:14 <Command u'/usr/bin/git rev-parse --verify HEAD', pid 21381>: process completed
17:07:14 <Command u'/usr/bin/git rev-parse --verify HEAD', pid 21381>: process completed
17:07:14 Deploying Rev: b67bba77acb7c0ffc678201c9f3f54f198da6650
17:07:14 Update DEPLOY_HEAD
17:07:14 Creating /srv/deployment/ores/deploy/.git/DEPLOY_HEAD
17:07:14 <Command u'/usr/bin/git for-each-ref --sort=taggerdate --format=%(refname) refs/tags'>: starting process
17:07:14 <Command u'/usr/bin/git for-each-ref --sort=taggerdate --format=%(refname) refs/tags', pid 21387>: process started
17:07:14 <Command u'/usr/bin/git for-each-ref --sort=taggerdate --format=%(refname) refs/tags', pid 21387>: process completed
17:07:14 <Command u'/usr/bin/git for-each-ref --sort=taggerdate --format=%(refname) refs/tags', pid 21387>: process completed
17:07:14 <Command u'/usr/bin/git for-each-ref --sort=taggerdate --format=%(refname) refs/tags', pid 21387>: process completed
17:07:14 <Command u'/usr/bin/git tag -d scap/sync/2017-11-20/0002'>: starting process
17:07:14 <Command u'/usr/bin/git tag -d scap/sync/2017-11-20/0002', pid 21391>: process started
17:07:14 <Command u'/usr/bin/git tag -d scap/sync/2017-11-20/0002', pid 21391>: process completed
17:07:14 Update server info
17:07:14 <Command u'/usr/bin/git update-server-info'>: starting process
17:07:14 <Command u'/usr/bin/git update-server-info', pid 21395>: process started
17:07:14 <Command u'/usr/bin/git update-server-info', pid 21395>: process completed
17:07:14 <Command u'/usr/bin/git submodule foreach --recursive git update-server-info'>: starting process
17:07:14 <Command u'/usr/bin/git submodule foreach --recursive git update-server-info', pid 21399>: process started
17:07:15 <Command u'/usr/bin/git submodule foreach --recursive git update-server-info', pid 21399>: process completed
17:07:15 Started deploy [ores/deploy@b67bba7]: Deploy ORES service b67bba7
17:07:15 
== WORKER ==
:* deployment-sca03.deployment-prep.eqiad.wmflabs
17:07:15 Running remote deploy cmd ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'ores/deploy', '--force', '-g', 'worker', 'fetch', '--refresh-config']
17:07:15 Using key: /etc/keyholder.d/deploy_service.pub
17:07:22 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'ores/deploy', '--force', '-g', 'worker', 'fetch', '--refresh-config'] on deployment-sca03.deployment-prep.eqiad.wmflabs returned [70]: OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /dev/null
debug1: Connecting to deployment-sca03.deployment-prep.eqiad.wmflabs [10.68.21.183] port 22.
debug1: Connection established.
debug1: identity file /etc/keyholder.d/deploy_service.pub type 1
debug1: key_load_public: No such file or directory
debug1: identity file /etc/keyholder.d/deploy_service.pub-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u3
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-128-etm@openssh.com none
debug1: kex: client->server aes128-ctr umac-128-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 7a:bc:b1:78:59:4d:61:1b:5c:de:5a:25:ec:bb:a3:82
debug1: Host 'deployment-sca03.deployment-prep.eqiad.wmflabs' is known and matches the ECDSA host key.
debug1: Found key in /etc/ssh/ssh_known_hosts:55
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /etc/keyholder.d/deploy_service.pub
debug1: Server accepts key: pkalg ssh-rsa blen 535
debug1: Authentication succeeded (publickey).
Authenticated to deployment-sca03.deployment-prep.eqiad.wmflabs ([10.68.21.183]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending command: /usr/bin/scap deploy-local -v --repo ores/deploy --force -g worker fetch --refresh-config
http://deployment-tin.deployment-prep.eqiad.wmflabs/ores/deploy/.git
/srv/deployment/ores/deploy-cache/cache
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 4180, received 29500 bytes, in 6.2 seconds
Bytes per second: sent 672.1, received 4743.5
debug1: Exit status 70

ores/deploy: fetch stage(s): 100% (ok: 0; fail: 1; left: 0)                     
17:07:22 1 targets had deploy errors
17:07:22 1 targets failed
17:07:22 1 of 1 worker targets failed, exceeding limit
Rollback all deployed groups? [Y/n]:

Event Timeline

awight triaged this task as High priority.Dec 13 2017, 5:09 PM
awight created this task.

/srv is full. Strange that there was no error message during deployment, though...

strange indeed. Full disk can case all sorts of weird behaviors though.

strange indeed. Full disk can case all sorts of weird behaviors though.

+1 This might not be something we can prevent. Sorry to cry wolf!

Meanwhile, kick this if you get the chance: https://gerrit.wikimedia.org/r/#/c/395048/