Page MenuHomePhabricator

Make it less cumbersome to bootstrap and update python webservices
Closed, ResolvedPublic

Description

I have recently started working on my Python tools again, and it's really fun, because most things just work. There are two steps that I find annoying (probably because the rest of the process is so straightforward): bootstrapping a new tool, and updating it later on.

When I am bootstrapping a new tool, I follow the steps listed in https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#python_.28Python3_.2B_Kubernetes.29. Except I literally just copy and paste them one by one, in the exact order. The file layout and paths are all standardized, that it just works. But it's kind of annoying to repetitively do that, and some of the commands are long enough (e.g. webservice --backend=kubernetes python shell) that it's much harder to memorize. And you can't just paste them all in at once because of the special k8s shell thing. I can imagine a helper like $ new-python-webservice <git url> that follows those steps, creates the venvs, installs dependencies, and starts the webservice.

And if I updated my tool and changed a dependency? I need to login to the k8s shell, activate the venv, run pip, exit out, and restart. Again, a helper command that does all of this would be pretty nice.

I scoped this ticket about Python since that's what I've been working with and most familiar with. I'm not sure if other languages would also benefit from similar helpers.

Event Timeline

Change 435662 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/docker-images/toollabs-images@master] Add webservice-python-bootstrap command

https://gerrit.wikimedia.org/r/435662

This is ready for merge, but I don't have time to do the docker rebuild & upload right now. Will do ASAP, please @Legoktm ping me if I forget.

Change 435662 merged by Arturo Borrero Gonzalez:
[operations/docker-images/toollabs-images@master] Add webservice-python-bootstrap command

https://gerrit.wikimedia.org/r/435662

Mentioned in SAL (#wikimedia-cloud) [2018-05-31T11:31:49Z] <zhuyifei1999_> building & pushing python/web docker image T174769

Is it just me that getting a usable shell pod is ridiculously difficult? When I want to test this thing out:

1tools.zhuyifei1999-test@tools-bastion-02:~$ webservice --backend kubernetes python shell
2Pod is not ready in time
3tools.zhuyifei1999-test@tools-bastion-02:~$ webservice --backend kubernetes python shell
4If you don't see a command prompt, try pressing enter.
5tools.zhuyifei1999-test@interactive:~$
6tools.zhuyifei1999-test@interactive:~$ which webservice-python-bootstrap
7/usr/local/bin/webservice-python-bootstrap
8tools.zhuyifei1999-test@interactive:~$ statSession ended, resume using 'kubectl attach interactive -c interactive -i -t' command when the pod is running
9Pod stopped. Session cannot be resumed.
10
11tools.zhuyifei1999-test@tools-bastion-02:~$ webservice --backend kubernetes python shell
12Pod is not ready in time
13tools.zhuyifei1999-test@tools-bastion-02:~$ webservice --backend kubernetes python shell
14If you don't see a command prompt, try pressing enter.
15tools.zhuyifei1999-test@interactive:~$
16tools.zhuyifei1999-test@interactive:~$ cat `which webservice-python-bootstrap`
17#!/bin/bash
18
19set -euxo pipefail
20
21if test ! -d ~/www/python/venv; then
22 python3 -m venv ~/www/python/venv
23fi
24
25~/www/python/venv/bin/pip install --upgrade pip
26
27if test -f ~/www/python/src/requirements.txt; then
28 ~/www/python/venv/bin/pip install -r ~/www/python/src/requirements.txt
29fi
30
31tools.zhuyifei1999-test@interactive:~$ ls -l `which webservice-python-bootstrap`
32-rwxr-xr-x 1 root root 285 May 31 11:29 /usr/local/bin/webservice-python-bootstrap
33tools.zhuyifei1999-test@interactive:~$ webservice-python-bootstrap
34+ test '!' -d /data/project/zhuyifei1999-test//www/python/venv
35+ python3 -m venv /data/project/zhuyifei1999-test//www/python/venv
36Session ended, resume using 'kubectl attach interactive -c interactive -i -t' command when the pod is running
37Pod stopped. Session cannot be resumed.
38
39tools.zhuyifei1999-test@tools-bastion-02:~$ tree www
40[...]
4128 directories, 230 files
42tools.zhuyifei1999-test@tools-bastion-02:~$ ^C
43tools.zhuyifei1999-test@tools-bastion-02:~$ rm -r www
44tools.zhuyifei1999-test@tools-bastion-02:~$ webservice --backend kubernetes python shell
45Pod is not ready in time
46tools.zhuyifei1999-test@tools-bastion-02:~$ webservice --backend kubernetes python shell
47Pod is not ready in time
48tools.zhuyifei1999-test@tools-bastion-02:~$

Is it just me that getting a usable shell pod is ridiculously difficult? When I want to test this thing out:

I got a pod creation timeout the first time I tried as well, but a second attempt succeeded. I also noticed that the pod which timed out took just barely longer to launch than the webservice command was willing to wait and then took just over a minute to be destroyed by the scheduler. I looked at per-worker load for the k8s cluster and nothing jumped out as being highly unusual. It could be random bad luck, but it could also be that we are nearing some capacity limit for our current deployment. We have 337 namespaces (tools) with active pods right now which is the highest usage I've seen.

zhuyifei1999 assigned this task to Legoktm.
tools.zhuyifei1999-test@interactive:~$ rm -r /data/project/zhuyifei1999-test//www/
tools.zhuyifei1999-test@interactive:~$ mkdir -p /data/project/zhuyifei1999-test//www/python/src/
tools.zhuyifei1999-test@interactive:~$ echo wheel > /data/project/zhuyifei1999-test//www/python/src/requirements.txt
tools.zhuyifei1999-test@interactive:~$ webservice-python-bootstrap 
+ test '!' -d /data/project/zhuyifei1999-test//www/python/venv
+ python3 -m venv /data/project/zhuyifei1999-test//www/python/venv
+ /data/project/zhuyifei1999-test//www/python/venv/bin/pip install --upgrade pip
Downloading/unpacking pip from https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl#sha256=717cdffb2833be8409433a93746744b59505f42146e8d37de6c62b430e25d6d7
  Downloading pip-10.0.1-py2.py3-none-any.whl (1.3MB): 1.3MB downloaded
Installing collected packages: pip
  Found existing installation: pip 1.5.6
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...
+ test -f /data/project/zhuyifei1999-test//www/python/src/requirements.txt
+ /data/project/zhuyifei1999-test//www/python/venv/bin/pip install -r /data/project/zhuyifei1999-test//www/python/src/requirements.txt
Collecting wheel (from -r /data/project/zhuyifei1999-test//www/python/src/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl (41kB)
    100% |████████████████████████████████| 51kB 1.6MB/s 
Installing collected packages: wheel
Successfully installed wheel-0.31.1
tools.zhuyifei1999-test@interactive:~$

Let's track the timeout issue in a separate task if it gets really bad consistently.

I probably didn't say it that explicitly in the task description, but the second part of my gripe was that you have to log into the shell pod, wait for it to be ready, and then type in your command again. I'd like for something like webservice --backend=kubernetes python shell -- webservice-python-bootstrap to work, where I can pass the command I want to run in the shell from the start.

I probably didn't say it that explicitly in the task description, but the second part of my gripe was that you have to log into the shell pod, wait for it to be ready, and then type in your command again. I'd like for something like webservice --backend=kubernetes python shell -- webservice-python-bootstrap to work, where I can pass the command I want to run in the shell from the start.

This is already tracked as T169695: Run non-interactive commands on Toolforge kubernetes webservices.