Page MenuHomePhabricator

Unhandled(?) exceptions in scap3
Open, MediumPublic

Description

We are re-imaging maps-test200[1-4], and I think maps-test2001 is semi-ready, whereas 2-3 are still broken. Scap3 resulted in this log. I'm not sure if it actually caused any problems.

yurik@tin:/srv/deployment/kartotherian/deploy$ scap deploy -v
17:25:58 Started Deploy: kartotherian/deploy
17:25:58 Deploying Rev: a9195c21da382a3662912e0c3377ed7738c62611
17:25:58 Update DEPLOY_HEAD
17:25:58 Creating /srv/deployment/kartotherian/deploy/.git/DEPLOY_HEAD
17:25:58 Update server info
Entering 'src'
17:25:59 
== CANARY ==
:* maps-test2001.codfw.wmnet
17:25:59 Running remote deploy cmd ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'canary', 'fetch', '--refresh-config']
kartotherian/deploy: fetch stage(s): 100% (ok: 1; fail: 0; left: 0)             
17:26:09 Running remote deploy cmd ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'canary', 'config_deploy', '--refresh-config']
kartotherian/deploy: config_deploy stage(s): 100% (ok: 1; fail: 0; left: 0)     
17:26:10 Running remote deploy cmd ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'canary', 'promote', '--refresh-config']
kartotherian/deploy: promote and restart_service stage(s): 100% (ok: 1; fail: 0; left: 0)
canary deploy successful. Continue? [y]es/[n]o/[c]ontinue all groups: y 
17:28:20 
== DEFAULT ==
:* maps-test2002.codfw.wmnet
:* maps2001.codfw.wmnet
:* maps1004.eqiad.wmnet
:* maps1003.eqiad.wmnet
:* maps1001.eqiad.wmnet
:* maps2002.codfw.wmnet
:* maps2004.codfw.wmnet
:* maps-test2004.codfw.wmnet
:* maps-test2003.codfw.wmnet
:* maps2003.codfw.wmnet
17:28:20 Running remote deploy cmd ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'default', 'fetch', '--refresh-config']
17:28:21 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'default', 'fetch', '--refresh-config'] on maps-test2002.codfw.wmnet returned [70]: 17:28:21 WARNING  - Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 253, in run
    app._load_config()
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 69, in _load_config
    overrides = self._get_config_overrides()
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 399, in _get_config_overrides
    cfg.write(yaml.dump(config, default_flow_style=False))
IOError: [Errno 28] No space left on device
17:28:21 ERROR    - deploy-local failed: <IOError> [Errno 28] No space left on device

17:28:21 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'default', 'fetch', '--refresh-config'] on maps-test2003.codfw.wmnet returned [70]: 17:28:21 WARNING  - Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 253, in run
    app._load_config()
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 69, in _load_config
    overrides = self._get_config_overrides()
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 399, in _get_config_overrides
    cfg.write(yaml.dump(config, default_flow_style=False))
IOError: [Errno 28] No space left on device
17:28:21 ERROR    - deploy-local failed: <IOError> [Errno 28] No space left on device

17:28:21 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'default', 'fetch', '--refresh-config'] on maps-test2004.codfw.wmnet returned [70]: 17:28:21 WARNING  - Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 253, in run
    app._load_config()
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 69, in _load_config
    overrides = self._get_config_overrides()
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 399, in _get_config_overrides
    cfg.write(yaml.dump(config, default_flow_style=False))
IOError: [Errno 28] No space left on device
17:28:21 ERROR    - deploy-local failed: <IOError> [Errno 28] No space left on device

Related Objects

Event Timeline

Yurik created this task.Oct 4 2016, 5:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 4 2016, 5:37 PM

The last line of the log tells it all, @Yurik:

IOError: [Errno 28] No space left on device
17:28:21 ERROR - deploy-local failed: <IOError> [Errno 28] No space left on device

Gehel added a subscriber: Gehel.Oct 5 2016, 8:22 PM

yep, maps-test2* are still not functional. This should be resolved once T147194 is done.

demon moved this task from Needs triage to Debt on the Scap board.Dec 8 2016, 1:56 AM
mmodell added a subscriber: mmodell.Dec 8 2016, 8:59 PM

is this even a scap issue?