Page MenuHomePhabricator

Issues with phabricator-translations.git during exports
Closed, ResolvedPublic

Description

Got the following notice when running exports,

The authenticity of host 'git-ssh.wikimedia.org (2620:0:861:ed1a::3:16)' can't be established.
ECDSA key fingerprint is SHA256:b5OFtJYoeVZvIKmbyH/oxgRqvFscJaZmn7e9BXXyf9A.
Are you sure you want to continue connecting (yes/no)?

  [Symfony\Component\Process\Exception\ProcessTimedOutException]
  The process "/home/<xyz>/clupdate-git-repo 'ssh://vcs@git-ssh.wikimedia.org/source/phabricator-translations.git' '/<xyz>/phabricator' 'wmf/stable' 'b25cc91a86ede76249463f9c1224607e9f372d86'" exceeded the timeout
  of 300 seconds.


update [--variant VARIANT] [--] <project>

Details

Related Gerrit Patches:

Event Timeline

abi_ created this task.Jun 27 2019, 5:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2019, 5:08 PM
abi_ added a comment.Jul 1 2019, 11:01 AM

Noticed the same issue with the exports today as well.

I get:

l10n-bot repoupdate phabricator

                                                                                                                                                                                                                    
  [Symfony\Component\Process\Exception\ProcessFailedException]                                                                                                                                                      
  The command "/home/betawiki/config/bin/clupdate-git-repo 'ssh://vcs@git-ssh.wikimedia.org/source/phabricator-translations.git' '/resources/nike/phabricator' 'wmf/stable' 'b25cc91a86ede76249463f9c1224607e9f372  
  d86'" failed.                                                                                                                                                                                                     
  Exit Code: 1(General error)                                                                                                                                                                                       
  Working directory: /resources/nike                                                                                                                                                                                
  Output:                                                                                                                                                                                                           
  ================                                                                                                                                                                                                  
  Error Output:                                                                                                                                                                                                     
  ================                                                                                                                                                                                                  
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                                                                                                                                                       
  @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @                                                                                                                                                       
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                                                                                                                                                       
  IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!                                                                                                                                                             
  Someone could be eavesdropping on you right now (man-in-the-middle attack)!                                                                                                                                       
  It is also possible that a host key has just been changed.                                                                                                                                                        
  The fingerprint for the ECDSA key sent by the remote host is                                                                                                                                                      
  SHA256:b5OFtJYoeVZvIKmbyH/oxgRqvFscJaZmn7e9BXXyf9A.                                                                                                                                                               
  Please contact your system administrator.                                                                                                                                                                         
  Add correct host key in /home/nike/.ssh/known_hosts to get rid of this message.                                                                                                                                   
  Offending ECDSA key in /home/nike/.ssh/known_hosts:94                                                                                                                                                             
    remove with:                                                                                                                                                                                                    
    ssh-keygen -f "/home/nike/.ssh/known_hosts" -R git-ssh.wikimedia.org                                                                                                                                            
  ECDSA host key for git-ssh.wikimedia.org has changed and you have requested strict checking.                                                                                                                      
  Host key verification failed.                                                                                                                                                                                     
  fatal: Could not read from remote repository.                                                                                                                                                                     
  Please make sure you have the correct access rights                                                                                                                                                               
  and the repository exists.                                                                                                                                                                                        
                                                                                                                                                                                                                    

update [--variant VARIANT] [--] <project>

https://wikitech.wikimedia.org/w/index.php?title=Help:SSH_Fingerprints/git-ssh.wikimedia.org&diff=1827965&oldid=1768511 indicates this is a legitimate change.

There is StrictHostKeyChecking=no (which wouldn't fix my error) that could be used, but I don't like the security implications of that.

I think best would be if we could ship these known keys centrally and have l10n-bot use them, to leave an audit trail and to avoid each deployer having to fiddle with these manually.

One option is to add them to /etc/ssh/known_hosts (though man pages say it is /etc/ssh/ssh_known_hosts). This would need to go through puppet.

We could also pass through command line (in bin/persistent-ssh) -oGlobalKnownHostsFile=... or -oUserKnownHostsFile=... to override the location where ssh command looks for these, so that it wouldn't need to go through puppet. We would create such file in the translatewiki repo (bringing back the problem of trying to figure out where to place such a file). It also wouldn't affect svn/mercurial repos (I don't think we have any currently). I'd prefer this option so the relevant things are closer together.

persistent-ssh is only called in repoexport, so anyone doing manually l10n-bot repoupdate foo wouldn't use that wrapper. It would have to be moved down to repoupdate/repocommit scripts. Or fallback to using /etc/ssh/ssh_known_hosts.

abi_ added a subscriber: Pginer-WMF.Jul 8 2019, 9:34 AM

Hi @Pginer-WMF ,

This issue needs to be fixed as any translations made to the Phabricator project will not be exported. Hence we would like to move this task to the current quarter.

abi_ claimed this task.Jul 8 2019, 11:21 AM
abi_ added a comment.Jul 8 2019, 1:14 PM

I'll be updating persistent-ssh, and will add another flag - -oUserKnownHostsFile=... to specify the known hosts file. I'll then use persistent-ssh on the repoupdate and repocommit scripts as well, using export GIT_SSH="$DIRSCRIPT/persistent-ssh".

abi_ triaged this task as Normal priority.Jul 11 2019, 1:36 AM

As per discussion with Niklas, we'll be moving the GIT_SSH declaration from autoexport into repoupdate & repocommit

Change 522049 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[translatewiki@master] Add common known hosts file to be used for exports and imports

https://gerrit.wikimedia.org/r/522049

Change 522049 merged by jenkins-bot:
[translatewiki@master] Add common known hosts file to be used for exports and imports

https://gerrit.wikimedia.org/r/522049

abi_ added a comment.Jul 16 2019, 5:08 PM

Noticed some issues during the last export. Want to make sure that these were not related to changes made for this task, hence leaving this un-resolved.

abi_ closed this task as Resolved.Jul 20 2019, 4:27 AM

Tested this during the last 2 exports and it is working well. Marking this as resolved.

abi_ moved this task from Backlog to Done on the User-abi_ board.Jul 20 2019, 4:28 AM