Page MenuHomePhabricator

Deploy a tool using a buildpack-based image on Toolforge
Open, Needs TriagePublic


As a proof of concept, we should be able to build a buildpack-based image for a specific tool in Toolforge and run the webservice using it.

For now, we're focusing on Python 3.7 due to simplicity and my familiarity with it.

Event Timeline

Legoktm created this task.Oct 15 2020, 9:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 15 2020, 9:10 PM

Mentioned in SAL (#wikimedia-cloud) [2020-10-29T21:33:16Z] <legoktm> published image (T265681)

Mentioned in SAL (#wikimedia-cloud) [2020-10-29T22:20:50Z] <legoktm> switched test tool over to use buildpack image (T265681)

legoktm@toolsbeta-proxy-1:~$ curl -H "Host:" localhost
Hello to the new buildpack-based world!


I wrote a basic flask app:

# License: CC0-1.0
from flask import Flask

app = Flask(__name__)

def index():
    return 'Hello to the new buildpack-based world!'

Then built the buildpack-based image:

1root@tools-docker-imagebuilder-01:/home/legoktm/flask-example# pack build --builder --publish
2latest: Pulling from toolforge-buster0-builder
3Digest: sha256:70724160bd6c4ba38e750969d00aa69b4f032c3a5a20e140b25fbed5d054375f
4Status: Image is up to date for
50.9.1: Pulling from buildpacksio/lifecycle
6Digest: sha256:53bf0e18a734e0c4071aa39b950ed8841f82936e53fb2a0df56c6aa07f9c5023
7Status: Image is up to date for buildpacksio/lifecycle:0.9.1
9[detector] org.toolforge.buildpacks.python37 0.0.1
10[detector] org.toolforge.buildpacks.pip 0.0.1
11[detector] org.toolforge.buildpacks.uwsgi 0.0.1
13[analyzer] Previous image with name "" not found
14[analyzer] Restoring metadata for "org.toolforge.buildpacks.python37:apt" from cache
16[restorer] Restoring data for "org.toolforge.buildpacks.python37:apt" from cache
18[builder] ---> Python 3.7 Buildpack
19[builder] ----> Installing python3.7
20[builder] -----> Reusing cache
21[builder] -----> Updating apt caches
22[builder] Hit:1 buster/updates InRelease
23[builder] Get:2 buster InRelease [121 kB]
24[builder] Get:3 buster-updates InRelease [51.9 kB]
25[builder] rm: cannot remove '/var/cache/apt/archives/partial/*.deb': Permission denied
26[builder] Fetched 173 kB in 1s (183 kB/s)
27[builder] Reading package lists...
28[builder] -----> Fetching .debs for python3
29[builder] Reading package lists...
30[builder] Building dependency tree...
31[builder] The following additional packages will be installed:
32[builder] libmpdec2 libpython3-stdlib libpython3.7-minimal libpython3.7-stdlib
33[builder] libreadline7 libsqlite3-0 mime-support python3-minimal python3.7
34[builder] python3.7-minimal readline-common
35[builder] Suggested packages:
36[builder] python3-doc python3-tk python3-venv python3.7-venv python3.7-doc
37[builder] binfmt-support readline-doc
38[builder] Recommended packages:
39[builder] file
40[builder] The following NEW packages will be installed:
41[builder] libmpdec2 libpython3-stdlib libpython3.7-minimal libpython3.7-stdlib
42[builder] libreadline7 libsqlite3-0 mime-support python3 python3-minimal python3.7
43[builder] python3.7-minimal readline-common
44[builder] 0 upgraded, 12 newly installed, 0 to remove and 12 not upgraded.
45[builder] Need to get 0 B/5486 kB of archives.
46[builder] After this operation, 25.2 MB of additional disk space will be used.
47[builder] Download complete and in download only mode
48[builder] -----> Fetching .debs for python3-dev
49[builder] Reading package lists...
50[builder] Building dependency tree...
51[builder] The following additional packages will be installed:
52[builder] dh-python libexpat1-dev libmpdec2 libpython3-dev libpython3-stdlib
53[builder] libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib
54[builder] libreadline7 libsqlite3-0 mime-support python3 python3-distutils
55[builder] python3-lib2to3 python3-minimal python3.7 python3.7-dev python3.7-minimal
56[builder] readline-common
57[builder] Suggested packages:
58[builder] python3-doc python3-tk python3-venv python3.7-venv python3.7-doc
59[builder] binfmt-support readline-doc
60[builder] Recommended packages:
61[builder] file
62[builder] The following NEW packages will be installed:
63[builder] dh-python libexpat1-dev libmpdec2 libpython3-dev libpython3-stdlib
64[builder] libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib
65[builder] libreadline7 libsqlite3-0 mime-support python3 python3-dev python3-distutils
66[builder] python3-lib2to3 python3-minimal python3.7 python3.7-dev python3.7-minimal
67[builder] readline-common
68[builder] 0 upgraded, 21 newly installed, 0 to remove and 12 not upgraded.
69[builder] Need to get 0 B/56.4 MB of archives.
70[builder] After this operation, 117 MB of additional disk space will be used.
71[builder] Download complete and in download only mode
72[builder] -----> Installing dh-python_3.20190308_all.deb
73[builder] -----> Installing libexpat1-dev_2.2.6-2+deb10u1_amd64.deb
74[builder] -----> Installing libmpdec2_2.4.2-2_amd64.deb
75[builder] -----> Installing libpython3-dev_3.7.3-1_amd64.deb
76[builder] -----> Installing libpython3-stdlib_3.7.3-1_amd64.deb
77[builder] -----> Installing libpython3.7-dev_3.7.3-2+deb10u2_amd64.deb
78[builder] -----> Installing libpython3.7-minimal_3.7.3-2+deb10u2_amd64.deb
79[builder] -----> Installing libpython3.7-stdlib_3.7.3-2+deb10u2_amd64.deb
80[builder] -----> Installing libpython3.7_3.7.3-2+deb10u2_amd64.deb
81[builder] -----> Installing libreadline7_7.0-5_amd64.deb
82[builder] -----> Installing libsqlite3-0_3.27.2-3_amd64.deb
83[builder] -----> Installing mime-support_3.62_all.deb
84[builder] -----> Installing python3-dev_3.7.3-1_amd64.deb
85[builder] -----> Installing python3-distutils_3.7.3-1_all.deb
86[builder] -----> Installing python3-lib2to3_3.7.3-1_all.deb
87[builder] -----> Installing python3-minimal_3.7.3-1_amd64.deb
88[builder] -----> Installing python3.7-dev_3.7.3-2+deb10u2_amd64.deb
89[builder] -----> Installing python3.7-minimal_3.7.3-2+deb10u2_amd64.deb
90[builder] -----> Installing python3.7_3.7.3-2+deb10u2_amd64.deb
91[builder] -----> Installing python3_3.7.3-1_amd64.deb
92[builder] -----> Installing readline-common_7.0-5_all.deb
93[builder] -----> Writing profile script
94[builder] -----> Rewrite package-config files
95[builder] ----> Installing pip
96[builder] % Total % Received % Xferd Average Speed Time Time Time Current
97[builder] Dload Upload Total Spent Left Speed
98[builder] 100 1841k 100 1841k 0 0 59.9M 0 --:--:-- --:--:-- --:--:-- 59.9M
99[builder] OK
100[builder] Collecting pip
101[builder] Downloading pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
102[builder] Collecting setuptools
103[builder] Downloading setuptools-50.3.2-py3-none-any.whl (785 kB)
104[builder] Collecting wheel
105[builder] Downloading wheel-0.35.1-py2.py3-none-any.whl (33 kB)
106[builder] Installing collected packages: pip, setuptools, wheel
107[builder] Successfully installed pip-20.2.4 setuptools-50.3.2 wheel-0.35.1
108[builder] ----> Installing virtualenv
109[builder] Collecting virtualenv
110[builder] Downloading virtualenv-20.1.0-py2.py3-none-any.whl (4.9 MB)
111[builder] Collecting six<2,>=1.9.0
112[builder] Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
113[builder] Collecting appdirs<2,>=1.4.3
114[builder] Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
115[builder] Collecting filelock<4,>=3.0.0
116[builder] Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
117[builder] Collecting importlib-metadata<3,>=0.12; python_version < "3.8"
118[builder] Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
119[builder] Collecting distlib<1,>=0.3.1
120[builder] Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
121[builder] Collecting zipp>=0.5
122[builder] Downloading zipp-3.4.0-py3-none-any.whl (5.2 kB)
123[builder] Installing collected packages: six, appdirs, filelock, zipp, importlib-metadata, distlib, virtualenv
124[builder] Successfully installed appdirs-1.4.4 distlib-0.3.1 filelock-3.0.12 importlib-metadata-2.0.0 six-1.15.0 virtualenv-20.1.0 zipp-3.4.0
125[builder] ---> pip Buildpack
126[builder] ----> Creating virtualenv
127[builder] created virtual environment in 340ms
128[builder] creator CPython3Posix(dest=/layers/org.toolforge.buildpacks.pip/pip/venv, clear=False, global=False)
129[builder] seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/tfb/.local/share/virtualenv)
130[builder] added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
131[builder] activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
132[builder] Requirement already up-to-date: pip in /layers/org.toolforge.buildpacks.pip/pip/venv/lib/python3.7/site-packages (20.2.4)
133[builder] ----> Installing from requirements.txt
134[builder] Collecting flask
135[builder] Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
136[builder] Collecting click>=5.1
137[builder] Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
138[builder] Collecting Werkzeug>=0.15
139[builder] Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
140[builder] Collecting itsdangerous>=0.24
141[builder] Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
142[builder] Collecting Jinja2>=2.10.1
143[builder] Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
144[builder] Collecting MarkupSafe>=0.23
145[builder] Downloading MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (27 kB)
146[builder] Installing collected packages: click, Werkzeug, itsdangerous, MarkupSafe, Jinja2, flask
147[builder] Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 Werkzeug-1.0.1 click-7.1.2 flask-1.1.2 itsdangerous-1.1.0
148[builder] ---> uwsgi Buildpack
149[builder] ----> Installing uwsgi
150[builder] created virtual environment in 213ms
151[builder] creator CPython3Posix(dest=/layers/org.toolforge.buildpacks.uwsgi/uwsgi/venv, clear=False, global=False)
152[builder] seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/tfb/.local/share/virtualenv)
153[builder] added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
154[builder] activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
155[builder] Requirement already up-to-date: pip in /layers/org.toolforge.buildpacks.uwsgi/uwsgi/venv/lib/python3.7/site-packages (20.2.4)
156[builder] Collecting uwsgi==2.0.18
157[builder] Downloading uwsgi-2.0.18.tar.gz (801 kB)
158[builder] Building wheels for collected packages: uwsgi
159[builder] Building wheel for uwsgi ( started
160[builder] Building wheel for uwsgi ( finished with status 'done'
161[builder] Created wheel for uwsgi: filename=uWSGI-2.0.18-cp37-cp37m-linux_x86_64.whl size=505965 sha256=5a0207564aa468563f3b11f877fdcb4fb752a0c16916096ba775c29a185d4551
162[builder] Stored in directory: /tmp/pip-ephem-wheel-cache-rt56u5og/wheels/e7/d0/f2/b93905969980eaf9fe18183b0c43470d63b13fcd3a1f6c8efe
163[builder] Successfully built uwsgi
164[builder] Installing collected packages: uwsgi
165[builder] Successfully installed uwsgi-2.0.18
166[builder] ----> Configuring uwsgi web launcher
168[exporter] Adding layer 'org.toolforge.buildpacks.python37:python37'
169[exporter] Adding layer 'org.toolforge.buildpacks.pip:pip'
170[exporter] Adding layer 'org.toolforge.buildpacks.uwsgi:uwsgi'
171[exporter] Adding 1/1 app layer(s)
172[exporter] Adding layer 'launcher'
173[exporter] Adding layer 'config'
174[exporter] Adding layer 'process-types'
175[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
176[exporter] Adding label ''
177[exporter] Adding label 'io.buildpacks.project.metadata'
178[exporter] Setting default process type 'web'
179[exporter] *** Images (sha256:079863d923d7615b84e9f5652b4ed4ec74d20bbc89e1249dcef063029a216435):
181[exporter] Reusing cache layer 'org.toolforge.buildpacks.python37:apt'
182[exporter] Adding cache layer 'org.toolforge.buildpacks.python37:python37'
183[exporter] Adding cache layer 'org.toolforge.buildpacks.pip:pip'
184[exporter] Adding cache layer 'org.toolforge.buildpacks.uwsgi:uwsgi'
185Successfully built image

And then I deployed it in toolsbeta:

2apiVersion: apps/v1
3kind: Deployment
5 generation: 1
6 labels:
7 name: test
8 toolforge: tool
9 "web"
10 "tfb"
11 name: apt-browser
12 namespace: tool-test
14 replicas: 1
15 selector:
16 matchLabels:
17 name: test
18 toolforge: tool
19 "web"
20 "tfb"
21 template:
22 metadata:
23 labels:
24 name: test
25 toolforge: tool
26 "web"
27 "tfb"
28 spec:
29 containers:
30 - image:
31 imagePullPolicy: Always
32 name: webservice
33 ports:
34 - containerPort: 8000
35 name: http
36 protocol: TCP
37 resources: {}
39apiVersion: v1
40kind: Service
42 name: test
43 namespace: tool-test
44 labels:
45 name: test
46 toolforge: tool
47 "web"
48 "tfb"
50 ports:
51 - name: http
52 protocol: TCP
53 port: 8000
54 targetPort: 8000
55 selector:
56 name: test
59kind: Ingress
61 name: test-subdomain
62 namespace: tool-test
63 labels:
64 name: test
65 toolforge: tool
66 "web"
67 "tfb"
69 rules:
70 - host:
71 http:
72 paths:
73 - backend:
74 serviceName: test
75 servicePort: 8000

Bstorm added a subscriber: Bstorm.Thu, Oct 29, 10:30 PM

Now we just need to get <CD thing> to make those k8s objects. That's awesome.

Well, that and the buildpack run...but that's the manual version mapped out.