Page MenuHomePhabricator
Create Task
Maniphest T99644

beta-scap-eqiad fails to sync due to "Permission denied (public key)"
Closed, ResolvedPublic
Actions

Description

https://integration.wikimedia.org/ci/job/beta-scap-eqiad/53388/console

00:01:05.266 02:36:43 Started cache_git_info
00:02:08.293 02:37:46 Finished cache_git_info (duration: 01m 03s)
00:02:08.293 02:37:46 Started sync-proxies
00:02:08.293 02:37:46 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n'] called with an empty host list.
00:02:08.293 02:37:46 Finished sync-proxies (duration: 00m 00s)
00:02:08.316 02:37:46 Started sync-apaches
00:02:08.316 sync-common:   0% (ok: 0; fail: 0; left: 4)                                     
00:02:10.928 02:37:49 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n'] on deployment-mediawiki01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-mediawiki01.eqiad.wmflabs,10.68.17.170' (ECDSA) to the list of known hosts.
00:02:10.928 Permission denied (publickey).
00:02:10.928 
00:02:10.929 sync-common:  25% (ok: 0; fail: 1; left: 3)                                     
00:02:10.929 02:37:49 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n'] on deployment-videoscaler01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-videoscaler01.eqiad.wmflabs,10.68.16.211' (ECDSA) to the list of known hosts.
00:02:10.929 Permission denied (publickey).
00:02:10.929 
00:02:10.930 sync-common:  50% (ok: 0; fail: 2; left: 2)                                     
00:02:12.922 02:37:51 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n'] on deployment-mediawiki02.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-mediawiki02.eqiad.wmflabs,10.68.16.127' (ECDSA) to the list of known hosts.
00:02:12.922 Permission denied (publickey).
00:02:12.922 
00:02:12.923 sync-common:  75% (ok: 0; fail: 3; left: 1)                                     
00:02:14.873 02:37:53 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n'] on deployment-jobrunner01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-jobrunner01.eqiad.wmflabs,10.68.17.96' (ECDSA) to the list of known hosts.
00:02:14.873 Permission denied (publickey).
00:02:14.873 
00:02:14.874 sync-common: 100% (ok: 0; fail: 4; left: 0)                                     
00:02:14.874 sync-common: 100% (ok: 0; fail: 4; left: 0)                                     
00:02:14.874 
00:02:14.874 02:37:53 4 apaches had sync errors
00:02:14.874 02:37:53 Finished sync-apaches (duration: 00m 06s)
00:02:14.875 02:37:53 Started scap-rebuild-cdbs
00:02:14.875 scap-rebuild-cdbs:   0% (ok: 0; fail: 0; left: 4)                               
00:02:14.952 02:37:53 sudo -u mwdeploy -n -- /srv/deployment/scap/scap/bin/scap-rebuild-cdbs on deployment-videoscaler01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-videoscaler01.eqiad.wmflabs,10.68.16.211' (ECDSA) to the list of known hosts.
00:02:14.952 Permission denied (publickey).
00:02:14.952 
00:02:14.953 scap-rebuild-cdbs:  25% (ok: 0; fail: 1; left: 3)                               
00:02:14.958 02:37:53 sudo -u mwdeploy -n -- /srv/deployment/scap/scap/bin/scap-rebuild-cdbs on deployment-jobrunner01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-jobrunner01.eqiad.wmflabs,10.68.17.96' (ECDSA) to the list of known hosts.
00:02:14.958 Permission denied (publickey).
00:02:14.958 
00:02:14.958 scap-rebuild-cdbs:  50% (ok: 0; fail: 2; left: 2)                               
00:02:14.959 02:37:53 sudo -u mwdeploy -n -- /srv/deployment/scap/scap/bin/scap-rebuild-cdbs on deployment-mediawiki02.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-mediawiki02.eqiad.wmflabs,10.68.16.127' (ECDSA) to the list of known hosts.
00:02:14.959 Permission denied (publickey).
00:02:14.959 
00:02:14.960 scap-rebuild-cdbs:  75% (ok: 0; fail: 3; left: 1)                               
00:02:14.986 02:37:53 sudo -u mwdeploy -n -- /srv/deployment/scap/scap/bin/scap-rebuild-cdbs on deployment-mediawiki01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-mediawiki01.eqiad.wmflabs,10.68.17.170' (ECDSA) to the list of known hosts.
00:02:14.986 Permission denied (publickey).
00:02:14.986 
00:02:14.986 scap-rebuild-cdbs: 100% (ok: 0; fail: 4; left: 0)                               
00:02:14.987 scap-rebuild-cdbs: 100% (ok: 0; fail: 4; left: 0)                               
00:02:14.987 
00:02:14.987 02:37:53 4 hosts had scap-rebuild-cdbs errors
00:02:14.987 02:37:53 Finished scap-rebuild-cdbs (duration: 00m 00s)
00:02:14.988 02:37:53 Started sync_wikiversions
00:02:14.998 02:37:53 Compiled /srv/mediawiki-staging/wikiversions-labs.json to /srv/mediawiki-staging/wikiversions-labs.cdb
00:02:14.998 sync_wikiversions:   0% (ok: 0; fail: 0; left: 4)                               
00:02:15.072 02:37:53 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-bastion.eqiad.wmflabs::common/wikiversions*.{json,cdb} /srv/mediawiki on deployment-videoscaler01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-videoscaler01.eqiad.wmflabs,10.68.16.211' (ECDSA) to the list of known hosts.
00:02:15.072 Permission denied (publickey).
00:02:15.072 
00:02:15.072 sync_wikiversions:  25% (ok: 0; fail: 1; left: 3)                               
00:02:15.073 02:37:53 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-bastion.eqiad.wmflabs::common/wikiversions*.{json,cdb} /srv/mediawiki on deployment-mediawiki01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-mediawiki01.eqiad.wmflabs,10.68.17.170' (ECDSA) to the list of known hosts.
00:02:15.073 Permission denied (publickey).
00:02:15.073 
00:02:15.073 sync_wikiversions:  50% (ok: 0; fail: 2; left: 2)                               
00:02:15.092 02:37:53 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-bastion.eqiad.wmflabs::common/wikiversions*.{json,cdb} /srv/mediawiki on deployment-mediawiki02.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-mediawiki02.eqiad.wmflabs,10.68.16.127' (ECDSA) to the list of known hosts.
00:02:15.092 Permission denied (publickey).
00:02:15.092 
00:02:15.092 sync_wikiversions:  75% (ok: 0; fail: 3; left: 1)                               
00:02:15.108 02:37:53 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-bastion.eqiad.wmflabs::common/wikiversions*.{json,cdb} /srv/mediawiki on deployment-jobrunner01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-jobrunner01.eqiad.wmflabs,10.68.17.96' (ECDSA) to the list of known hosts.
00:02:15.108 Permission denied (publickey).
00:02:15.108 
00:02:15.109 sync_wikiversions: 100% (ok: 0; fail: 4; left: 0)                               
00:02:15.109 sync_wikiversions: 100% (ok: 0; fail: 4; left: 0)                               
00:02:15.109 
00:02:15.109 02:37:53 Finished sync_wikiversions (duration: 00m 00s)
00:02:15.110 02:37:53 4 hosts had sync_wikiversions errors
00:02:15.110 02:37:53 Finished scap: beta-scap-eqiad (build #53388) (duration: 02m 11s)

Event Timeline

Krinkle created this task.May 19 2015, 2:40 AM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added a project: Beta-Cluster-Infrastructure.
Krinkle subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 19 2015, 2:40 AM
bd808 subscribed.May 19 2015, 2:56 AM

After deployment-bastion reboots, the keyholder ssh-agent that is used by scap needs to be primed manually (it's a security thing for prod):

$ ssh deployment-bastion.eqiad.wmflabs
$ sudo -s
$ sudo -u keyholder env SSH_AUTH_SOCK=/run/keyholder/agent.sock ssh-add /etc/keyholder.d/mwdeploy_rsa

The passphrase for the key can be found on deployment-salt in the file /var/lib/git/labs/private/files/ssh/tin/mwdeploy_rsa.passphrase

bd808 closed this task as Resolved.May 19 2015, 3:01 AM
bd808 claimed this task.

https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/53390/console

02:54:54 02:54:54 Started sync-apaches
02:54:54 sync-common:   0% (ok: 0; fail: 0; left: 4)                                     
02:55:41 sync-common:  25% (ok: 1; fail: 0; left: 3)                                     
02:55:47 sync-common:  50% (ok: 2; fail: 0; left: 2)                                     
02:55:48 sync-common:  75% (ok: 3; fail: 0; left: 1)                                     
02:55:52 sync-common: 100% (ok: 4; fail: 0; left: 0)                                     
02:55:52 sync-common: 100% (ok: 4; fail: 0; left: 0)                                     
02:55:52 
02:55:52 02:55:52 Finished sync-apaches (duration: 00m 58s)
02:55:52 02:55:52 Started scap-rebuild-cdbs
02:55:52 scap-rebuild-cdbs:   0% (ok: 0; fail: 0; left: 4)                               
02:57:09 scap-rebuild-cdbs:  25% (ok: 1; fail: 0; left: 3)                               
02:57:10 scap-rebuild-cdbs:  50% (ok: 2; fail: 0; left: 2)                               
02:57:15 scap-rebuild-cdbs:  75% (ok: 3; fail: 0; left: 1)                               
02:59:03 scap-rebuild-cdbs: 100% (ok: 4; fail: 0; left: 0)                               
02:59:03 scap-rebuild-cdbs: 100% (ok: 4; fail: 0; left: 0)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits