Page MenuHomePhabricator

Toolforge Stretch bastion occasionally closes SSH connection just before authentication succeeds
Closed, ResolvedPublic

Description

@Magnus ran into this issue yesterday, and I experienced it just now: occasionally, the Stretch bastion seems to close SSH connections just before authentication succeeds. At normal verbosity, it looks like this:

$ ssh toolforge                                            
Connection closed by 185.15.56.48 port 22

With -vvv, here are the last few lines of output:

…
debug1: Next authentication method: publickey                                                                                                                                                                                                                                                                                                              
debug1: Offering public key: /home/lucas/.ssh/[KEY] ED25519 SHA256:[HASH] explicit agent
debug3: send packet: type 50                                                                                                      
debug2: we sent a publickey packet, wait for reply                                                                                
debug3: receive packet: type 60                                                                                                                                                                                                         
debug1: Server accepts key: /home/lucas/.ssh/[KEY] ED25519 SHA256:[HASH] explicit agent                                                                                                 
debug3: sign_and_send_pubkey: ED25519 SHA256:[HASH]
debug3: sign_and_send_pubkey: signing using ssh-ed25519                                           
debug3: send packet: type 50
Connection closed by 185.15.56.48 port 22

The problem usually seems to resolve itself within minutes; here’s the corresponding output of a successful session immediately afterwards, at the same verbosity level:

…
debug1: Next authentication method: publickey                                                                                                                                                                                                                                             
debug1: Offering public key: /home/lucas/.ssh/[KEY] ED25519 SHA256:[HASH] explicit agent
debug3: send packet: type 50                                                                                                      
debug2: we sent a publickey packet, wait for reply                                                                                
debug3: receive packet: type 60                                                                                                                                                                                                         
debug1: Server accepts key: /home/lucas/.ssh/[KEY] ED25519 SHA256:[HASH] explicit agent                                                                                                 
debug3: sign_and_send_pubkey: ED25519 SHA256:[HASH]
debug3: sign_and_send_pubkey: signing using ssh-ed25519                                             
debug3: send packet: type 50                                                                                                                                               
debug3: receive packet: type 52                                                                                                      
debug1: Authentication succeeded (publickey).                                      
Authenticated to login-stretch.tools.wmflabs.org ([185.15.56.48]:22).                             
…

Still, it would be nice if this worked more realiably… any ideas what’s going on?

Event Timeline

Happened to me as well, yesterday (2019-03-08, 08:23UTC)

Is it possible this is related to the LDAP problems we've been seeing?

Ah, right, I forgot about those… sounds possible, yeah.

I think we can close this task, it hasn’t been happening for a while now (at least as far as I’m aware).