the problem is a tonne of these:

02:10:28 Started sync-proxies
02:10:28 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-1.25wmf9', '--include', 'php-1.25wmf9/cache', '--include', 'php-1.25wmf9/cache/l10n', '--include', 'php-1.25wmf9/cache/l10n/***'] on mw1161.eqiad.wmnet returned [255]: Permission denied (publickey).

02:10:28 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-1.25wmf9', '--include', 'php-1.25wmf9/cache', '--include', 'php-1.25wmf9/cache/l10n', '--include', 'php-1.25wmf9/cache/l10n/***'] on mw1070.eqiad.wmnet returned [255]: Permission denied (publickey).

02:10:28 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-1.25wmf9', '--include', 'php-1.25wmf9/cache', '--include', 'php-1.25wmf9/cache/l10n', '--include', 'php-1.25wmf9/cache/l10n/***'] on mw1010.eqiad.wmnet returned [255]: Permission denied (publickey).

02:10:28 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-1.25wmf9', '--include', 'php-1.25wmf9/cache', '--include', 'php-1.25wmf9/cache/l10n', '--include', 'php-1.25wmf9/cache/l10n/***'] on mw1201.eqiad.wmnet returned [255]: Permission denied (publickey).

sync-proxies: 100% (ok: 0; fail: 4; left: 0)

[20:52:08] <bd808> Reedy: Bah. It's scap
[20:52:18] <Reedy> :(
[20:53:06] <bd808> Reedy: This is clobbering SSH_AUTH_SOCK -- https://github.com/wikimedia/mediawiki-tools-scap/blob/master/scap/cli.py#L171-L175
[20:53:49] <bd808> Reedy: So that needs a patch to only replace the auth sock if the shared one is present and readable I think
[20:56:56] <Reedy> bd808: os.path.isfile?
[20:57:12] <bd808> Reedy: The other way to fix it would be to change the permissions on the shared auth socket so that l10nupdate can read from it. In the long term that would be even better.
[20:58:53] <bd808> Reedy: os.path.exists. It could be a symlink in theory
[21:00:28] <bd808> Reedy: Or more pythonically, open the file for reading and only change the env is that succeeds
[21:00:33] <Reedy> if auth_sock is not None and os.path.exists(auth_sock):
[21:00:40] <Reedy> heh

Change 176750 had a related patch set uploaded (by Reedy):
Only use config ssh_auth_sock if set/readable/useable

https://gerrit.wikimedia.org/r/176750

Patch-For-Review

Urgency just became High+1 for us, it turns out there is no workaround, LocalisationUpdate is overriding newer, manually deployed extension messages.

Change 176750 merged by jenkins-bot:
Only use config ssh_auth_sock if set/readable/useable

https://gerrit.wikimedia.org/r/176750

In T76061#802030, @gerritbot wrote:

Change 176750 merged by jenkins-bot:
Only use config ssh_auth_sock if set/readable/useable
https://gerrit.wikimedia.org/r/176750

Annnnd, reverted.

In T76061#799528, @awight wrote:

Urgency just became High+1 for us, it turns out there is no workaround, LocalisationUpdate is overriding newer, manually deployed extension messages.

You did scap afterwards?

l10nupdate isn't actually syncing anything, so it should mean that nothing is actually changing on the mw servers

When @Reedy tried the patch he and @ori made in https://gerrit.wikimedia.org/r/#/c/176750/ it failed in prod:

17:58:47 sync-common failed: <error> [Errno 2] No such file or directory
17:58:47 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/***', 'mw1010.eqiad.wmnet', 'mw1070.eqiad.wmnet', 'mw1161.eqiad.wmnet', 'mw1201.eqiad.wmnet'] on mw1122 returned [70]: 17:58:47 Unhandled error:
Traceback (most recent call last):
  File "/srv/deployment/scap/scap/scap/cli.py", line 283, in run
    app._setup_environ()
  File "/srv/deployment/scap/scap/scap/cli.py", line 178, in _setup_environ
    sock.connect(auth_sock)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory

I think this mostly means we need to catch additional errors/exceptions in the try/except block but it could use additional debugging to figure that out.

I only pushed extension code, not a full scap. Are production app servers not reading from the messages files, is that why you ask? I gotta learn more :)

Okay, scap does in fact update the messages. Thanks! This means we have a workaround, I'm lowering the urgency again.

Okay, scap does in fact update the messages. Thanks! This means we have a workaround, I'm lowering the urgency again.

In T76061#802164, @awight wrote:

I only pushed extension code, not a full scap. Are production app servers not reading from the messages files, is that why you ask? I gotta learn more :)

Yeah, they don't. Loading the hundreds of json (was PHP) files, then any fallback chains is just slow.

l10nupdate/scap build cdb files with the full message set for a specific language, so it's effectively all in one place

see also: T85790

Where are we here? Is this still a problem, or has the underlying issue been sorted out?

Still happening as of 2015-02-16:

02:14:23 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-1.25wmf16', '--include', 'php-1.25wmf16/cache', '--include', 'php-1.25wmf16/cache/l10n', '--include', 'php-1.25wmf16/cache/l10n/***'] on mw1070.eqiad.wmnet returned [255]: Permission denied (publickey).

sync-proxies:  16% (ok: 0; fail: 1; left: 5)

l10nupdate is logging in SAL every morning that it worked, but looking at the /var/log/l10nupdatelog logs on tin show that the l10n cache is built but then fails to sync.

@mmodell: help here please.

There are two possible ways to fix this problem for the l10nupdate user invoking scap:

1. Make a way (probably via a new command line option) for scap to bypass using the share ssh-agent socket so that the special ssh key for the l10nupdate user can be used again as it was before scap was updated to use a shared agent.
2. Fix the file system permission on the shared ssh-agent to allow the l10nupdate user to read the socket.

Option 2 would be tricky to do without either giving the l10nupdate user more privileges than it currently has or making the socket permission look strange (eg 1770 l10nupdate:wikidev). I'm actually not sure if the socket can be changed to be owned by anyone other than the current keyholder user either. Making l10nupdate a member of wikidev would be a fairly large privilege escalation and should not be done in my opinion.

A possible third option that would be a bit of a combination of the other two but more involved would be to extend the keyholder service to be capable of running multiple instances simultaneously, running a proxy for the l10nupdate user's key and passing an alternate -D ssh_auth_sock:/run/keyholder/some-other-proxy.sock setting to the scap invocation.

In T76061#1045443, @bd808 wrote:

A possible third option that would be a bit of a combination of the other two but more involved would be to extend the keyholder service to be capable of running multiple instances simultaneously, running a proxy for the l10nupdate user's key and passing an alternate -D ssh_auth_sock:/run/keyholder/some-other-proxy.sock setting to the scap invocation.

Wait. Is this as easy as calling scap as scap -D ssh_auth_sock:$SSH_AUTH_SOCK?

In T76061#1045444, @bd808 wrote:

In T76061#1045443, @bd808 wrote:

A possible third option that would be a bit of a combination of the other two but more involved would be to extend the keyholder service to be capable of running multiple instances simultaneously, running a proxy for the l10nupdate user's key and passing an alternate -D ssh_auth_sock:/run/keyholder/some-other-proxy.sock setting to the scap invocation.

Wait. Is this as easy as calling scap as scap -D ssh_auth_sock:$SSH_AUTH_SOCK?

Not quite apparently. The l10nupdate-1 script invokes sync-dir via sudo-withagent. sudo-withagent invokes itself via sudo and then starts ssh-agent before running the passed command. $SSH_AUTH_SOCK is not set in the outer calling environment of l10nudpdate-1 and thus can't be set directly there via shell variable expansion. The shell variable would need to be expanded inside the recursive sudo-withagent invocation after $SSH_AUTH_SOCK has been set. This would require modifications to sudo-withagent to somehow mark a command line argument as something that needed to be eval'd at the correct time.

Change 191248 had a related patch set uploaded (by BryanDavis):
Add universal argument to ignore ssh_auth_sock

https://gerrit.wikimedia.org/r/191248

Patch-For-Review

Change 191251 had a related patch set uploaded (by BryanDavis):
l10nupdate: use --no-shared-authsock with sync-dir

https://gerrit.wikimedia.org/r/191251

Patch-For-Review

Change 191248 merged by jenkins-bot:
Add universal argument to ignore ssh_auth_sock

https://gerrit.wikimedia.org/r/191248

scap change +2'd but I don't have +2 on ops/puppet so that one isn't merged.

Change 191251 merged by Ottomata:
l10nupdate: use --no-shared-authsock with sync-dir

https://gerrit.wikimedia.org/r/191251

Both the scap and l10nupdate changes are deployed in production now (and we didn't break scap for normal deployers this time!). Lets see what happens on the next run in a few hours.

Still same error in the log for last run.

The sync-dir call is still failing with ssh key permissions issues:

Syncing to Apaches at 2015-02-23 02:02:22+00:00
Starting ssh-agent
Agent pid 6530
Identity added: /home/l10nupdate/.ssh/id_rsa (/home/l10nupdate/.ssh/id_rsa)
02:02:22 Started sync-proxies
sync-proxies:   0% (ok: 0; fail: 0; left: 6)                                    
02:02:22 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n',
  '--include', 'php-1.25wmf17', '--include', 'php-1.25wmf17/cache', 
  '--include', 'php-1.25wmf17/cache/l10n', '--include', 'php-1.25wmf17/cache/l10n/***'
  ] on mw1033.eqiad.wmnet returned [255]: Permission denied (publickey).

sync-proxies:  16% (ok: 0; fail: 1; left: 5)
[...snip...]
sync-proxies: 100% (ok: 0; fail: 6; left: 0)

I can however use the l10nupdate user's private key to authenticate from tin:

$ /usr/local/bin/sudo-withagent l10nupdate ssh mw1033.eqiad.wmnet
Starting ssh-agent
Agent pid 24302
Identity added: /home/l10nupdate/.ssh/id_rsa (/home/l10nupdate/.ssh/id_rsa)
Linux mw1033 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64
Ubuntu 14.04.1 LTS
mw1033 is role::mediawiki::appserver
The last Puppet run was at Mon Feb 23 20:52:13 UTC 2015 (5 minutes ago).
Ubuntu 14.04.1 LTS auto-installed on Mon Dec 1 11:20:22 UTC 2014.
Last login: Mon Feb 23 20:55:04 2015 from tin.eqiad.wmnet
l10nupdate@mw1033:~$ logout
Connection to mw1033.eqiad.wmnet closed.

I can also do this via sudo -u l10nupdate -- /usr/local/bin/sudo-withagent l10nupdate ssh -v mw1033.eqiad.wmnet which would seem to rule out the l10nupdate user not having access to it's own keys.

@mmodell could you find a time to interactively run the sync-dir as /usr/local/bin/l10nupdate-1 does it with a --verbose flag appended and see if we can get any more information on what is keeping this from working?

@bd808 I'll see what I can figure out

It would be super nice to have this finally fixed. :)

We have new i18n message that we would like updated this week when we add a "Wikibooks" site links section on Wikidata. I am not sure how to otherwise to properly pull in the updated messages.

In T76061#1060755, @aude wrote:

It would be super nice to have this finally fixed. :)
We have new i18n message that we would like updated this week when we add a "Wikibooks" site links section on Wikidata. I am not sure how to otherwise to properly pull in the updated messages.

The l10n cache files on tin are being updated every night, so at the worst right now a manual scap (or the normal train scap) pushes all the updates out to the wikis.

@bd808 thanks. we can do scap if/when needed... but hopefully we'll figure out why we have this issue with l10update

@bd808: So, yes, you can authenticate with the remote server via the key on tin, however, the remote server then has to connect back to tin to rsync the files, right? And on the remote machine there aren't any ssh keys in ~/.ssh/ ... how is the remote server supposed to authenticate, does it use agent forwarding for that?

Am I confused somehow about the way this works? I can't tell 100% whether it's pushing from tin or pulling from the remote machines, but it seems like it's connecting to the remote machines which in turn use rsync to connect back and grab the files.

In T76061#1061401, @mmodell wrote:

@bd808: So, yes, you can authenticate with the remote server via the key on tin, however, the remote server then has to connect back to tin to rsync the files, right? And on the remote machine there aren't any ssh keys in ~/.ssh/ ... how is the remote server supposed to authenticate, does it use agent forwarding for that?
Am I confused somehow about the way this works? I can't tell 100% whether it's pushing from tin or pulling from the remote machines, but it seems like it's connecting to the remote machines which in turn use rsync to connect back and grab the files.

The connection from tin to the target host uses ssh. On the target host side, rsync is run as the mwdeploy user to connect back to tin (or an rsync proxy). The rsync is done over the built-in rsync protocol via a connection to the running rsync daemon on tin (or an rsync proxy). We don't use the --rsh or -e options of rsync. See the tasks.sync_common method for more gory details.

Ok $SSH_AUTH_SOCK doesn't seem to be getting passed on to the sync targets. Don't we need to enable agent forwarding in ssh config or via command line?

hmmm I guess it must be something in scap then. I don't know how to get more debugging info from scap, adding --verbose didn't change anything at all.

In T76061#1061690, @mmodell wrote:

hmmm I guess it must be something in scap then. I don't know how to get more debugging info from scap, adding --verbose didn't change anything at all.

One thing that might help debugging the apparent ssh connection issues would be to add one or two -v flags to DEFAULT_RSYNC_ARGS in tasks.py. At some point I was planning on making the --verbose flag to scap and sync-* do that automatically but I never got a method for passing the flag on to tasks that I liked. The quick and dirty test would be just to live edit the file on tin (doesn't need to be synced to other scap hosts) and run sync-dir as the l10nupdate user.

apparently it's not getting to RSYNC because that didn't make any difference...

In T76061#1064902, @mmodell wrote:

apparently it's not getting to RSYNC because that didn't make any difference...

I'm not even sure why I thought that would help debug. I was thinking of some other random problem from the past apparently. What I really meant to suggest was adding verbose flags to the ssh command which is setup in the scap.ssh module by the SSH tuple at the top of the file.

scap and sync-* make the ssh connection as the mwdeply user. The l10nupdate ssh key is only good for itself and not this shared deploy user. To make this work we need to update scap to allow the caller to specify the user to run the ssh commands as. Something like sync-dir --ssh-user l10nupdate ....

Change 196143 had a related patch set uploaded (by BryanDavis):
l10nupdate: connect to remote hosts as l10nupdate user

https://gerrit.wikimedia.org/r/196143

In T76061#1111915, @bd808 wrote:

scap and sync-* make the ssh connection as the mwdeply user. The l10nupdate ssh key is only good for itself and not this shared deploy user. To make this work we need to update scap to allow the caller to specify the user to run the ssh commands as. Something like sync-dir --ssh-user l10nupdate ....

We already have the means to override config values using the -D key:value option, so https://gerrit.wikimedia.org/r/196143 uses that.

Change 196143 merged by Tim Starling:
l10nupdate: connect to remote hosts as l10nupdate user

https://gerrit.wikimedia.org/r/196143

How'd it go last night?