Page MenuHomePhabricator

Add ssh-client support to Kubernetes containers
Closed, ResolvedPublic

Description

So, our current use case is a bot that regularly parses a page, checks for updates and then pushes the update to GitHub with a pull request, authorizing with a deploy key via SSH (git pull/push, etc.) The alternative would be downgrading the project and dependencies to Python 3.5.3 and we already downgraded to 3.7.3 as the part of our shift to Kubernetes containers (which has been mostly okay) but downgrading to 3.5.3 would require us to use out-of-date dependencies which might lose support the next time we have to upgrade or fix something.
I think as Kubernetes on Toolforge begins to be used for things apart from webservices this will become more useful since ssh is quite mainstream for most use-cases.

Event Timeline

QEDK moved this task from Inbox to Tracking on the User-QEDK board.
taavi renamed this task from Add ssh support to Kubernetes containers to Add ssh-client support to Kubernetes containers.Aug 27 2021, 10:39 AM

@Bstorm Any thoughts on this? Including openssh-client on the base images seems somewhat reasonable to me.

Change 715215 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/docker-images/toollabs-images@master] stretch-sssd: Add openssh-client

https://gerrit.wikimedia.org/r/715215

The other option for GitHub is to use git+https with OAuth instead of git+ssh: https://docs.github.com/en/github/extending-github/git-automation-with-oauth-tokens. You can't limit an OAuth token to a single repository unless you use a machine user, but that's usually a good idea if you're writing automatically anyway.

For T289850 Gerrit also supports read/write git+https (docs). Both token and LDAP auth are enabled (T218750).

The other option for GitHub is to use git+https with OAuth instead of git+ssh: https://docs.github.com/en/github/extending-github/git-automation-with-oauth-tokens. You can't limit an OAuth token to a single repository unless you use a machine user, but that's usually a good idea if you're writing automatically anyway.

For T289850 Gerrit also supports read/write git+https (docs). Both token and LDAP auth are enabled (T218750).

The other way to do it would be using git credentials store and storing the credentials in-memory or file. In any case, the ideal way would be to make deploys happen using read-only SSH keys since that has no possibility of leaks and access is always restricted to one repository, that is the primary reason I wanted to use it in the first place. Eventually though, I just wrote the script separately in Python 3.5.3, split off the project into two and then ran it as a Toolforge cronjob.

The other option for GitHub is to use git+https with OAuth instead of git+ssh: https://docs.github.com/en/github/extending-github/git-automation-with-oauth-tokens. You can't limit an OAuth token to a single repository unless you use a machine user, but that's usually a good idea if you're writing automatically anyway.

For T289850 Gerrit also supports read/write git+https (docs). Both token and LDAP auth are enabled (T218750).

Thanks, but that's not what I am trying to do. I would like to subscribe to Gerrit stream events which requires an SSH connection. There's also an existing tool that subscribes to these events (Gerrit-newcomer-bot] that will presumably also need this capability in a kubernetes deployment, when it is migrated to use that. uses paramiko, a pure python implementation of SSH. I suppose I can attempt to do the same and use a library for that in PHP; I'm already using phpseclib in my tool which is supposed to do that.

If there is another option besides adding a package to the images (such as PHP libraries), that is preferred. The images are too big as is.

Change 715215 abandoned by Kosta Harlan:

[operations/docker-images/toollabs-images@master] bullseye-sssd: Add openssh-client

Reason:

https://gerrit.wikimedia.org/r/715215

Change 918610 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/docker-images/toollabs-images@master] {bullseye,buster}-sssd: add openssh-client

https://gerrit.wikimedia.org/r/918610

If there is another option besides adding a package to the images (such as PHP libraries), that is preferred. The images are too big as is.

My measurement of https://gerrit.wikimedia.org/r/918610 applied to the Bullseye base image showed openssh-client adding approximately 5MB to the at rest image size as reported by docker image ls.

Change 918610 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] {bullseye,buster}-sssd: add openssh-client

https://gerrit.wikimedia.org/r/918610

Mentioned in SAL (#wikimedia-cloud) [2023-05-15T17:17:44Z] <bd808> Rebuilding bullseye and buster docker containers to pick up openssh-client package addition (T258841)

bd808 changed the task status from Open to In Progress.May 15 2023, 5:18 PM
bd808 claimed this task.
$ webservice python3.9 shell
$ cd /tmp
$ git clone git@gitlab.wikimedia.org:toolforge-repos/python-toolforge.git
Cloning into 'python-toolforge'...
The authenticity of host 'gitlab.wikimedia.org (208.80.154.145)' can't be established.
ECDSA key fingerprint is SHA256:V1Q7vVIt/XR2yuue3kUa27/+cSwrCm8wLDfBROh8HMs.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'gitlab.wikimedia.org,208.80.154.145' (ECDSA) to the list of known hosts.
git@gitlab.wikimedia.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The ssh git transport will work now. I would not actively suggest that anyone use it, but y'all can evaluate opsec risks of placing ssh private keys in the Toolforge multi-tenant environment for yourselves. 🦶🔫🤕