I just tried to deploy kartotherian service using scap3. The canary target failed (the service kept restarting), so scap3 after hanging for a while offered to rollback. I accepted (hit enter), but the git repo on the canary target stayed the same version as in the deployment dir, so the service kept restarting.
yurik@maps-test2001:/srv/deployment/kartotherian/deploy$ git log commit 2b93211b683c277f0614e1b3f17f48d12c9b170d yurik@tin:/srv/deployment/kartotherian/deploy$ git log commit 2b93211b683c277f0614e1b3f17f48d12c9b170d
yurik@tin:/srv/deployment/kartotherian/deploy$ scap deploy -v
00:23:06 Started Deploy: kartotherian/deploy
00:23:06 Deploying Rev: 2b93211b683c277f0614e1b3f17f48d12c9b170d
00:23:06 {'server_groups': 'canary, default', 'statsd_host': 'statsd.eqiad.wmnet', 'master_rsync': 'deployment.eqiad.wmnet', 'service_name': 'kartotherian', 'hhvm_pid_file': '/run/hhvm/hhvm.pid', 'dsh_targets': 'targets', 'deploy_dir': '/srv/mediawiki', 'git_server': 'tin.eqiad.wmnet', 'udp2log_host': 'fluorine.eqiad.wmnet', 'git_repo_user': 'deploy-service', 'perform_checks': True, 'git_fat': False, 'bin_dir': '/usr/bin', 'ssh_auth_sock': '/run/keyholder/proxy.sock', 'apache_pid_file': '/var/run/apache2/apache2.pid', 'pybal_interface': 'lo:LVS', 'ssh_user': 'deploy-service', 'canary_dsh_targets': 'target-canary', 'canary_threshold': 10.0, 'logstash_host': 'logstash1001.eqiad.wmnet:9200', 'nrpe_dir': '/etc/nagios/nrpe.d', 'wmf_realm': 'production', 'lock_file': '/tmp/scap.kartotherian.lock', 'tcpircbot_host': 'neon.wikimedia.org', 'service_timeout': 120.0, 'git_deploy_dir': '/srv/deployment', 'patch_path': None, 'dsh_proxies': 'scap-proxies', 'git_scheme': 'http', 'git_repo': 'kartotherian/deploy', 'datacenter': 'eqiad', 'udp2log_port': '8420', 'canary_wait_time': 20, 'config_deploy': False, 'statsd_port': '8125', 'service_port': '6533', 'dsh_masters': 'scap-masters', 'dsh_api_canaries': 'mediawiki-api-canaries', 'git_submodules': True, 'dsh_app_canaries': 'mediawiki-appserver-canaries', 'git_upstream_submodules': False, 'stage_dir': '/srv/mediawiki-staging', 'tcpircbot_port': '9200', 'log_json': False}
00:23:06 Update DEPLOY_HEAD
00:23:06 Creating /srv/deployment/kartotherian/deploy/.git/DEPLOY_HEAD
00:23:06 Update server info
Entering 'src'
00:23:06
== CANARY ==
:* maps-test2001.codfw.wmnet
00:23:06 Running remote deploy cmd ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'canary', 'fetch']
kartotherian/deploy: fetch stage(s): 100% (ok: 1; fail: 0; left: 0)
00:23:15 Running remote deploy cmd ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'canary', 'config_deploy']
kartotherian/deploy: config_deploy stage(s): 100% (ok: 1; fail: 0; left: 0)
00:23:16 Running remote deploy cmd ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'canary', 'promote']
00:25:18 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'canary', 'promote'] on maps-test2001.codfw.wmnet returned [70]:
kartotherian/deploy: promote and restart_service stage(s): 100% (ok: 0; fail: 1; left: 0)
00:25:18 1 targets had deploy errors
Stage 'promote' failed on group 'canary'. Perform rollback? [y]:
00:25:36 Running remote deploy cmd ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'kartotherian/deploy', '-g', 'canary', 'rollback']
kartotherian/deploy: rollback stage(s): 100% (ok: 1; fail: 0; left: 0)
00:25:37 Finished Deploy: kartotherian/deploy (duration: 02m 30s)