Page MenuHomePhabricator
Create Task
Maniphest T226739

Issues with phabricator-translations.git during exports
Closed, ResolvedPublic
Actions

Description

Got the following notice when running exports,

The authenticity of host 'git-ssh.wikimedia.org (2620:0:861:ed1a::3:16)' can't be established.
ECDSA key fingerprint is SHA256:b5OFtJYoeVZvIKmbyH/oxgRqvFscJaZmn7e9BXXyf9A.
Are you sure you want to continue connecting (yes/no)?

  [Symfony\Component\Process\Exception\ProcessTimedOutException]
  The process "/home/<xyz>/clupdate-git-repo 'ssh://vcs@git-ssh.wikimedia.org/source/phabricator-translations.git' '/<xyz>/phabricator' 'wmf/stable' 'b25cc91a86ede76249463f9c1224607e9f372d86'" exceeded the timeout
  of 300 seconds.


update [--variant VARIANT] [--] <project>

Details

SubjectRepoBranchLines +/-
Add common known hosts file to be used for exports and importstranslatewikimaster+11 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

abi_ created this task.Jun 27 2019, 5:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2019, 5:08 PM
abi_ added a parent task: T208190: Translatewiki.net translation import and export failures.Jun 27 2019, 5:09 PM
abi_ added a comment.Jul 1 2019, 11:01 AM

Noticed the same issue with the exports today as well.

Nikerabbit added a comment.Jul 1 2019, 11:51 AM

I get:

l10n-bot repoupdate phabricator

                                                                                                                                                                                                                    
  [Symfony\Component\Process\Exception\ProcessFailedException]                                                                                                                                                      
  The command "/home/betawiki/config/bin/clupdate-git-repo 'ssh://vcs@git-ssh.wikimedia.org/source/phabricator-translations.git' '/resources/nike/phabricator' 'wmf/stable' 'b25cc91a86ede76249463f9c1224607e9f372  
  d86'" failed.                                                                                                                                                                                                     
  Exit Code: 1(General error)                                                                                                                                                                                       
  Working directory: /resources/nike                                                                                                                                                                                
  Output:                                                                                                                                                                                                           
  ================                                                                                                                                                                                                  
  Error Output:                                                                                                                                                                                                     
  ================                                                                                                                                                                                                  
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                                                                                                                                                       
  @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @                                                                                                                                                       
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                                                                                                                                                       
  IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!                                                                                                                                                             
  Someone could be eavesdropping on you right now (man-in-the-middle attack)!                                                                                                                                       
  It is also possible that a host key has just been changed.                                                                                                                                                        
  The fingerprint for the ECDSA key sent by the remote host is                                                                                                                                                      
  SHA256:b5OFtJYoeVZvIKmbyH/oxgRqvFscJaZmn7e9BXXyf9A.                                                                                                                                                               
  Please contact your system administrator.                                                                                                                                                                         
  Add correct host key in /home/nike/.ssh/known_hosts to get rid of this message.                                                                                                                                   
  Offending ECDSA key in /home/nike/.ssh/known_hosts:94                                                                                                                                                             
    remove with:                                                                                                                                                                                                    
    ssh-keygen -f "/home/nike/.ssh/known_hosts" -R git-ssh.wikimedia.org                                                                                                                                            
  ECDSA host key for git-ssh.wikimedia.org has changed and you have requested strict checking.                                                                                                                      
  Host key verification failed.                                                                                                                                                                                     
  fatal: Could not read from remote repository.                                                                                                                                                                     
  Please make sure you have the correct access rights                                                                                                                                                               
  and the repository exists.                                                                                                                                                                                        
                                                                                                                                                                                                                    

update [--variant VARIANT] [--] <project>
Nikerabbit added a comment.Jul 1 2019, 12:05 PM

https://wikitech.wikimedia.org/w/index.php?title=Help:SSH_Fingerprints/git-ssh.wikimedia.org&diff=1827965&oldid=1768511 indicates this is a legitimate change.

There is StrictHostKeyChecking=no (which wouldn't fix my error) that could be used, but I don't like the security implications of that.

I think best would be if we could ship these known keys centrally and have l10n-bot use them, to leave an audit trail and to avoid each deployer having to fiddle with these manually.

One option is to add them to /etc/ssh/known_hosts (though man pages say it is /etc/ssh/ssh_known_hosts). This would need to go through puppet.

We could also pass through command line (in bin/persistent-ssh) -oGlobalKnownHostsFile=... or -oUserKnownHostsFile=... to override the location where ssh command looks for these, so that it wouldn't need to go through puppet. We would create such file in the translatewiki repo (bringing back the problem of trying to figure out where to place such a file). It also wouldn't affect svn/mercurial repos (I don't think we have any currently). I'd prefer this option so the relevant things are closer together.

Nikerabbit added a comment.Jul 1 2019, 3:27 PM

persistent-ssh is only called in repoexport, so anyone doing manually l10n-bot repoupdate foo wouldn't use that wrapper. It would have to be moved down to repoupdate/repocommit scripts. Or fallback to using /etc/ssh/ssh_known_hosts.

abi_ added a subscriber: Pginer-WMF.Jul 8 2019, 9:34 AM

Hi @Pginer-WMF ,

This issue needs to be fixed as any translations made to the Phabricator project will not be exported. Hence we would like to move this task to the current quarter.

Pginer-WMF added a project: Language-Team (Language-2019-July-September).Jul 8 2019, 9:46 AM
abi_ claimed this task.Jul 8 2019, 11:21 AM
abi_ moved this task from Backlog to In Progress on the Language-Team (Language-2019-July-September) board.
abi_ added a comment.Jul 8 2019, 1:14 PM

I'll be updating persistent-ssh, and will add another flag - -oUserKnownHostsFile=... to specify the known hosts file. I'll then use persistent-ssh on the repoupdate and repocommit scripts as well, using export GIT_SSH="$DIRSCRIPT/persistent-ssh".

abi_ triaged this task as Medium priority.Jul 11 2019, 1:36 AM

As per discussion with Niklas, we'll be moving the GIT_SSH declaration from autoexport into repoupdate & repocommit

gerritbot added a comment.Jul 11 2019, 9:46 AM

Change 522049 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[translatewiki@master] Add common known hosts file to be used for exports and imports

https://gerrit.wikimedia.org/r/522049

gerritbot added a project: Patch-For-Review.Jul 11 2019, 9:46 AM
abi_ moved this task from In Progress to In Review on the Language-Team (Language-2019-July-September) board.Jul 13 2019, 2:52 AM
gerritbot added a comment.Jul 15 2019, 5:44 PM

Change 522049 merged by jenkins-bot:
[translatewiki@master] Add common known hosts file to be used for exports and imports

https://gerrit.wikimedia.org/r/522049

jenkins-bot mentioned this in rGTWN05cd93b69d3a: Add common known hosts file to be used for exports and imports.Jul 15 2019, 5:44 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 15 2019, 6:10 PM
abi_ moved this task from In Review to Recheck after deployment on the Language-Team (Language-2019-July-September) board.Jul 16 2019, 8:11 AM
abi_ added a comment.Jul 16 2019, 5:08 PM

Noticed some issues during the last export. Want to make sure that these were not related to changes made for this task, hence leaving this un-resolved.

abi_ closed this task as Resolved.Jul 20 2019, 4:27 AM
abi_ moved this task from Recheck after deployment to Done on the Language-Team (Language-2019-July-September) board.

Tested this during the last 2 exports and it is working well. Marking this as resolved.

abi_ moved this task from Backlog to Done on the User-abi_ board.Jul 20 2019, 4:28 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL