Page MenuHomePhabricator

Run non-interactive commands on Toolforge kubernetes webservices
Open, MediumPublic

Description

It would be good to be able to run arbitrary non-interactive commands on kubernetes webservices.

Currently it you need to use the webservice --backend=kubernetes <type> shell command but this is always interactive.

For example to update the dependencies of a python webservice running on kubernetes you need to do something like this:

webservice --backend=kubernetes python shell

<wait for interactive shell on kubernetes controlled instance>

webservice-python-bootstrap

It would be nice to have something like:
webservice --backend=kubernetes python shell -- webservice-python-bootstrap

Event Timeline

Restricted Application edited projects, added Cloud-Services; removed Toolforge. · View Herald TranscriptJul 5 2017, 7:56 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
chasemp triaged this task as Medium priority.Jul 5 2017, 7:06 PM
chasemp added a subscriber: chasemp.

@Tarrow for now we are in a holding pattern on kubernetes features we can support but I think the idea is solid when we start gaining more traction.

Legoktm renamed this task from Run non-interactive commands on labs kubernetes webservices to Run non-interactive commands on Toolforge kubernetes webservices.May 31 2018, 5:45 PM
Legoktm updated the task description. (Show Details)

Proof of concept using kubectl directly against the 2020 Kubernetes cluster

$ /usr/bin/kubectl run interactive \
  --image=docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base:latest \
  --restart=Never --command=true --env=HOME=$HOME --labels='toolforge=tool' \
  --rm=true --stdin=true --tty=true \
  -- ls /etc
adduser.conf            gss            mke2fs.conf        resolv.conf
alternatives            gtk-3.0        modules-load.d     rmt
apt                     host.conf      motd               securetty
bash.bashrc             hostname       mtab               security
bash_completion.d       hosts          mysql              selinux
bindresvport.blacklist  ImageMagick-6  nanorc             shadow
binfmt.d                init.d         novaobserver.yaml  shadow-
ca-certificates         inputrc        nsswitch.conf      shells
ca-certificates.conf    issue          opt                skel
cron.daily              issue.net      os-release         ssl
dbus-1                  kernel         pam.conf           subgid
debconf.conf            ldap           pam.d              subuid
debian_version          ldap.conf      passwd             sysctl.d
default                 ldap.yaml      passwd-            systemd
deluser.conf            ld.so.cache    perl               terminfo
dhcp                    ld.so.conf     profile            timezone
dictionaries-common     ld.so.conf.d   profile.d          tmpfiles.d
dpkg                    libaudit.conf  python3            ucf.conf
emacs                   locale.alias   python3.7          update-motd.d
environment             locale.gen     rc0.d              vim
fonts                   localtime      rc1.d              wmcs-project
fstab                   login.defs     rc2.d              X11
gai.conf                logrotate.d    rc3.d              xattr.conf
group                   machine-id     rc4.d              xdg
group-                  mailcap        rc5.d
gshadow                 mailcap.order  rc6.d
gshadow-                mime.types     rcS.d
pod "interactive" deleted

Change 621776 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] Make webservice shell scriptable

https://gerrit.wikimedia.org/r/621776

This comment was removed by bd808.

Change 621776 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] Make webservice shell scriptable

https://gerrit.wikimedia.org/r/621776

This change works, but it is a bit less stable than the current method. When the interactive shell is left open for a long period of time (think hours, not minutes), kubectl can get disconnected from the running pod's container. When this happens, the pod is "leaked" in the tool's Kubernetes namespace and must be manually killed (kubectl delete pod ...). This could be confusing for folks who are not really aware of how webservice and kubectl interact yet as with a few leaks their namespace's quota for running new pods would be exhausted.

I'm not sure what balance to aim for is between making it possible to script webservice shell ... commands from the bastions (which the patch does) and making the webservice ... experience as intuitive as possible. @Tarrow, @Legoktm, @zhuyifei1999, @Bstorm: what are your thoughts?