Page MenuHomePhabricator

Beta cluster l10nupdate key not set up properly
Closed, ResolvedPublic

Description

391 languages rebuilt out of 391
Completed at 2015-10-05 23:26:07+00:00. Copying LC files to /srv/mediawiki-staging
Updated 1 JSON file(s) in '/mnt/srv/mediawiki-staging/php-master/cache/l10n'.
Syncing to Apaches at 2015-10-05 23:26:16+00:00
Starting ssh-agent
Agent pid 10315
Identity added: /home/l10nupdate/.ssh/id_rsa (/home/l10nupdate/.ssh/id_rsa)
23:27:03 Started sync-proxies
23:27:03 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-master', '--include', 'php-master/cache', '--include', 'php-master/cache/l10n', '--include', 'php-master/cache/l10n/***'] called with an empty host list.
23:27:03 Finished sync-proxies (duration: 00m 00s)
23:27:03 Started sync-apaches
sync-common:   0% (ok: 0; fail: 0; left: 5)                                     
23:27:04 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-master', '--include', 'php-master/cache', '--include', 'php-master/cache/l10n', '--include', 'php-master/cache/l10n/***'] on deployment-tmh01.deployment-prep.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-tmh01.deployment-prep.eqiad.wmflabs,10.68.16.211' (ECDSA) to the list of known hosts.
Permission denied (publickey).

sync-common:  20% (ok: 0; fail: 1; left: 4)                                     
23:27:04 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-master', '--include', 'php-master/cache', '--include', 'php-master/cache/l10n', '--include', 'php-master/cache/l10n/***'] on deployment-jobrunner01.deployment-prep.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-jobrunner01.deployment-prep.eqiad.wmflabs,10.68.17.96' (ECDSA) to the list of known hosts.
Permission denied (publickey).

sync-common:  40% (ok: 0; fail: 2; left: 3)                                     
23:27:04 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-master', '--include', 'php-master/cache', '--include', 'php-master/cache/l10n', '--include', 'php-master/cache/l10n/***'] on deployment-mediawiki02.deployment-prep.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-mediawiki02.deployment-prep.eqiad.wmflabs,10.68.16.127' (ECDSA) to the list of known hosts.
Permission denied (publickey).

sync-common:  60% (ok: 0; fail: 3; left: 2)                                     
23:27:04 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-master', '--include', 'php-master/cache', '--include', 'php-master/cache/l10n', '--include', 'php-master/cache/l10n/***'] on deployment-mediawiki01.deployment-prep.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-mediawiki01.deployment-prep.eqiad.wmflabs,10.68.17.170' (ECDSA) to the list of known hosts.
Permission denied (publickey).

sync-common:  80% (ok: 0; fail: 4; left: 1)                                     
23:27:04 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-master', '--include', 'php-master/cache', '--include', 'php-master/cache/l10n', '--include', 'php-master/cache/l10n/***'] on deployment-mediawiki03.deployment-prep.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-mediawiki03.deployment-prep.eqiad.wmflabs,10.68.17.55' (ECDSA) to the list of known hosts.
Permission denied (publickey).

sync-common: 100% (ok: 0; fail: 5; left: 0)                                     
sync-common: 100% (ok: 0; fail: 5; left: 0)                                     

23:27:04 5 apaches had sync errors

Event Timeline

Krenair raised the priority of this task from to Needs Triage.
Krenair updated the task description. (Show Details)
Krenair subscribed.

You can run just this command to see the issue, without having to wait for l10nupdate: sudo-withagent l10nupdate sync-dir --no-shared-authsock -D ssh_user:l10nupdate "php-master/cache/l10n"

This would also do it: sudo-withagent l10nupdate ssh deployment-mediawiki03

krenair@deployment-mediawiki03:~$ cat /etc/ssh/userkeys/l10nupdate 
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzcA/wB0uoU+XgiYN/scGczrAGuN99O8L7m8TviqxgX9s+RexhPtn8FHss1GKi8oxVO1V+ssABVb2q0fGza4wqrHOlZadcFEGjQhZ4IIfUwKUo78mKhQsUyTd5RYMR0KlcjB4UyWSDX5tFHK6FE7/tySNTX7Tihau7KZ9R0Ax//KySCG0skKyI1BK4Ufb82S8wohrktBO6W7lag0O2urh9dKI0gM8EuP666DGnaNBFzycKLPqLaURCeCdB6IiogLHiR21dyeHIIAN0zD6SUyTGH2ZNlZkX05hcFUEWcsWE49+Ve/rdfu1wWTDnourH/Xm3IBkhVGqskB+yp3Jkz2D3Q== l10nupdate@fenari

Setting mediawiki::users::l10nupdate_public_key to the real contents of files/ssh/tin/l10nupdate/id_rsa.pub in labs/private would be a bad idea (the private key for that is public), I guess we need a new key created on the puppetmaster.

I experimented with that idea for a bit but found that target hosts still reject it due to it not matching labs LDAP.

mmodell moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.
mmodell subscribed.
bd808 renamed this task from l10nupdate key not set up properly to Beta cluster l10nupdate key not set up properly.Nov 28 2015, 11:26 PM
bd808 set Security to None.

The proposed changes for T119746: localisationupdate broken on wmf wikis by scap master-master sync changes should make this problem moot by removing the custom l10nupdate ssh key from use.

demon claimed this task.
demon subscribed.

This hasn't been an issue in ages.