Page MenuHomePhabricator

[] Deploy a tool using a buildpack-based image on Toolforge
Closed, ResolvedPublic


As a proof of concept, we should be able to build a buildpack-based image for a specific tool in Toolforge and run the webservice using it.

For now, we're focusing on Python 3.7 due to simplicity and my familiarity with it.

Related Objects


Event Timeline

Mentioned in SAL (#wikimedia-cloud) [2020-10-29T21:33:16Z] <legoktm> published image (T265681)

Mentioned in SAL (#wikimedia-cloud) [2020-10-29T22:20:50Z] <legoktm> switched test tool over to use buildpack image (T265681)

legoktm@toolsbeta-proxy-1:~$ curl -H "Host:" localhost
Hello to the new buildpack-based world!


I wrote a basic flask app:

# License: CC0-1.0
from flask import Flask

app = Flask(__name__)

def index():
    return 'Hello to the new buildpack-based world!'

Then built the buildpack-based image:

1root@tools-docker-imagebuilder-01:/home/legoktm/flask-example# pack build --builder --publish
2latest: Pulling from toolforge-buster0-builder
3Digest: sha256:70724160bd6c4ba38e750969d00aa69b4f032c3a5a20e140b25fbed5d054375f
4Status: Image is up to date for
50.9.1: Pulling from buildpacksio/lifecycle
6Digest: sha256:53bf0e18a734e0c4071aa39b950ed8841f82936e53fb2a0df56c6aa07f9c5023
7Status: Image is up to date for buildpacksio/lifecycle:0.9.1
9[detector] org.toolforge.buildpacks.python37 0.0.1
10[detector] org.toolforge.buildpacks.pip 0.0.1
11[detector] org.toolforge.buildpacks.uwsgi 0.0.1
13[analyzer] Previous image with name "" not found
14[analyzer] Restoring metadata for "org.toolforge.buildpacks.python37:apt" from cache
16[restorer] Restoring data for "org.toolforge.buildpacks.python37:apt" from cache
18[builder] ---> Python 3.7 Buildpack
19[builder] ----> Installing python3.7
20[builder] -----> Reusing cache
21[builder] -----> Updating apt caches
22[builder] Hit:1 buster/updates InRelease
23[builder] Get:2 buster InRelease [121 kB]
24[builder] Get:3 buster-updates InRelease [51.9 kB]
25[builder] rm: cannot remove '/var/cache/apt/archives/partial/*.deb': Permission denied
26[builder] Fetched 173 kB in 1s (183 kB/s)
27[builder] Reading package lists...
28[builder] -----> Fetching .debs for python3
29[builder] Reading package lists...
30[builder] Building dependency tree...
31[builder] The following additional packages will be installed:
32[builder] libmpdec2 libpython3-stdlib libpython3.7-minimal libpython3.7-stdlib
33[builder] libreadline7 libsqlite3-0 mime-support python3-minimal python3.7
34[builder] python3.7-minimal readline-common
35[builder] Suggested packages:
36[builder] python3-doc python3-tk python3-venv python3.7-venv python3.7-doc
37[builder] binfmt-support readline-doc
38[builder] Recommended packages:
39[builder] file
40[builder] The following NEW packages will be installed:
41[builder] libmpdec2 libpython3-stdlib libpython3.7-minimal libpython3.7-stdlib
42[builder] libreadline7 libsqlite3-0 mime-support python3 python3-minimal python3.7
43[builder] python3.7-minimal readline-common
44[builder] 0 upgraded, 12 newly installed, 0 to remove and 12 not upgraded.
45[builder] Need to get 0 B/5486 kB of archives.
46[builder] After this operation, 25.2 MB of additional disk space will be used.
47[builder] Download complete and in download only mode
48[builder] -----> Fetching .debs for python3-dev
49[builder] Reading package lists...
50[builder] Building dependency tree...
51[builder] The following additional packages will be installed:
52[builder] dh-python libexpat1-dev libmpdec2 libpython3-dev libpython3-stdlib
53[builder] libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib
54[builder] libreadline7 libsqlite3-0 mime-support python3 python3-distutils
55[builder] python3-lib2to3 python3-minimal python3.7 python3.7-dev python3.7-minimal
56[builder] readline-common
57[builder] Suggested packages:
58[builder] python3-doc python3-tk python3-venv python3.7-venv python3.7-doc
59[builder] binfmt-support readline-doc
60[builder] Recommended packages:
61[builder] file
62[builder] The following NEW packages will be installed:
63[builder] dh-python libexpat1-dev libmpdec2 libpython3-dev libpython3-stdlib
64[builder] libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib
65[builder] libreadline7 libsqlite3-0 mime-support python3 python3-dev python3-distutils
66[builder] python3-lib2to3 python3-minimal python3.7 python3.7-dev python3.7-minimal
67[builder] readline-common
68[builder] 0 upgraded, 21 newly installed, 0 to remove and 12 not upgraded.
69[builder] Need to get 0 B/56.4 MB of archives.
70[builder] After this operation, 117 MB of additional disk space will be used.
71[builder] Download complete and in download only mode
72[builder] -----> Installing dh-python_3.20190308_all.deb
73[builder] -----> Installing libexpat1-dev_2.2.6-2+deb10u1_amd64.deb
74[builder] -----> Installing libmpdec2_2.4.2-2_amd64.deb
75[builder] -----> Installing libpython3-dev_3.7.3-1_amd64.deb
76[builder] -----> Installing libpython3-stdlib_3.7.3-1_amd64.deb
77[builder] -----> Installing libpython3.7-dev_3.7.3-2+deb10u2_amd64.deb
78[builder] -----> Installing libpython3.7-minimal_3.7.3-2+deb10u2_amd64.deb
79[builder] -----> Installing libpython3.7-stdlib_3.7.3-2+deb10u2_amd64.deb
80[builder] -----> Installing libpython3.7_3.7.3-2+deb10u2_amd64.deb
81[builder] -----> Installing libreadline7_7.0-5_amd64.deb
82[builder] -----> Installing libsqlite3-0_3.27.2-3_amd64.deb
83[builder] -----> Installing mime-support_3.62_all.deb
84[builder] -----> Installing python3-dev_3.7.3-1_amd64.deb
85[builder] -----> Installing python3-distutils_3.7.3-1_all.deb
86[builder] -----> Installing python3-lib2to3_3.7.3-1_all.deb
87[builder] -----> Installing python3-minimal_3.7.3-1_amd64.deb
88[builder] -----> Installing python3.7-dev_3.7.3-2+deb10u2_amd64.deb
89[builder] -----> Installing python3.7-minimal_3.7.3-2+deb10u2_amd64.deb
90[builder] -----> Installing python3.7_3.7.3-2+deb10u2_amd64.deb
91[builder] -----> Installing python3_3.7.3-1_amd64.deb
92[builder] -----> Installing readline-common_7.0-5_all.deb
93[builder] -----> Writing profile script
94[builder] -----> Rewrite package-config files
95[builder] ----> Installing pip
96[builder] % Total % Received % Xferd Average Speed Time Time Time Current
97[builder] Dload Upload Total Spent Left Speed
98[builder] 100 1841k 100 1841k 0 0 59.9M 0 --:--:-- --:--:-- --:--:-- 59.9M
99[builder] OK
100[builder] Collecting pip
101[builder] Downloading pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
102[builder] Collecting setuptools
103[builder] Downloading setuptools-50.3.2-py3-none-any.whl (785 kB)
104[builder] Collecting wheel
105[builder] Downloading wheel-0.35.1-py2.py3-none-any.whl (33 kB)
106[builder] Installing collected packages: pip, setuptools, wheel
107[builder] Successfully installed pip-20.2.4 setuptools-50.3.2 wheel-0.35.1
108[builder] ----> Installing virtualenv
109[builder] Collecting virtualenv
110[builder] Downloading virtualenv-20.1.0-py2.py3-none-any.whl (4.9 MB)
111[builder] Collecting six<2,>=1.9.0
112[builder] Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
113[builder] Collecting appdirs<2,>=1.4.3
114[builder] Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
115[builder] Collecting filelock<4,>=3.0.0
116[builder] Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
117[builder] Collecting importlib-metadata<3,>=0.12; python_version < "3.8"
118[builder] Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
119[builder] Collecting distlib<1,>=0.3.1
120[builder] Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
121[builder] Collecting zipp>=0.5
122[builder] Downloading zipp-3.4.0-py3-none-any.whl (5.2 kB)
123[builder] Installing collected packages: six, appdirs, filelock, zipp, importlib-metadata, distlib, virtualenv
124[builder] Successfully installed appdirs-1.4.4 distlib-0.3.1 filelock-3.0.12 importlib-metadata-2.0.0 six-1.15.0 virtualenv-20.1.0 zipp-3.4.0
125[builder] ---> pip Buildpack
126[builder] ----> Creating virtualenv
127[builder] created virtual environment in 340ms
128[builder] creator CPython3Posix(dest=/layers/org.toolforge.buildpacks.pip/pip/venv, clear=False, global=False)
129[builder] seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/tfb/.local/share/virtualenv)
130[builder] added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
131[builder] activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
132[builder] Requirement already up-to-date: pip in /layers/org.toolforge.buildpacks.pip/pip/venv/lib/python3.7/site-packages (20.2.4)
133[builder] ----> Installing from requirements.txt
134[builder] Collecting flask
135[builder] Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
136[builder] Collecting click>=5.1
137[builder] Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
138[builder] Collecting Werkzeug>=0.15
139[builder] Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
140[builder] Collecting itsdangerous>=0.24
141[builder] Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
142[builder] Collecting Jinja2>=2.10.1
143[builder] Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
144[builder] Collecting MarkupSafe>=0.23
145[builder] Downloading MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (27 kB)
146[builder] Installing collected packages: click, Werkzeug, itsdangerous, MarkupSafe, Jinja2, flask
147[builder] Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 Werkzeug-1.0.1 click-7.1.2 flask-1.1.2 itsdangerous-1.1.0
148[builder] ---> uwsgi Buildpack
149[builder] ----> Installing uwsgi
150[builder] created virtual environment in 213ms
151[builder] creator CPython3Posix(dest=/layers/org.toolforge.buildpacks.uwsgi/uwsgi/venv, clear=False, global=False)
152[builder] seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/tfb/.local/share/virtualenv)
153[builder] added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
154[builder] activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
155[builder] Requirement already up-to-date: pip in /layers/org.toolforge.buildpacks.uwsgi/uwsgi/venv/lib/python3.7/site-packages (20.2.4)
156[builder] Collecting uwsgi==2.0.18
157[builder] Downloading uwsgi-2.0.18.tar.gz (801 kB)
158[builder] Building wheels for collected packages: uwsgi
159[builder] Building wheel for uwsgi ( started
160[builder] Building wheel for uwsgi ( finished with status 'done'
161[builder] Created wheel for uwsgi: filename=uWSGI-2.0.18-cp37-cp37m-linux_x86_64.whl size=505965 sha256=5a0207564aa468563f3b11f877fdcb4fb752a0c16916096ba775c29a185d4551
162[builder] Stored in directory: /tmp/pip-ephem-wheel-cache-rt56u5og/wheels/e7/d0/f2/b93905969980eaf9fe18183b0c43470d63b13fcd3a1f6c8efe
163[builder] Successfully built uwsgi
164[builder] Installing collected packages: uwsgi
165[builder] Successfully installed uwsgi-2.0.18
166[builder] ----> Configuring uwsgi web launcher
168[exporter] Adding layer 'org.toolforge.buildpacks.python37:python37'
169[exporter] Adding layer 'org.toolforge.buildpacks.pip:pip'
170[exporter] Adding layer 'org.toolforge.buildpacks.uwsgi:uwsgi'
171[exporter] Adding 1/1 app layer(s)
172[exporter] Adding layer 'launcher'
173[exporter] Adding layer 'config'
174[exporter] Adding layer 'process-types'
175[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
176[exporter] Adding label ''
177[exporter] Adding label 'io.buildpacks.project.metadata'
178[exporter] Setting default process type 'web'
179[exporter] *** Images (sha256:079863d923d7615b84e9f5652b4ed4ec74d20bbc89e1249dcef063029a216435):
181[exporter] Reusing cache layer 'org.toolforge.buildpacks.python37:apt'
182[exporter] Adding cache layer 'org.toolforge.buildpacks.python37:python37'
183[exporter] Adding cache layer 'org.toolforge.buildpacks.pip:pip'
184[exporter] Adding cache layer 'org.toolforge.buildpacks.uwsgi:uwsgi'
185Successfully built image

And then I deployed it in toolsbeta:

2apiVersion: apps/v1
3kind: Deployment
5 generation: 1
6 labels:
7 name: test
8 toolforge: tool
9 "web"
10 "tfb"
11 name: apt-browser
12 namespace: tool-test
14 replicas: 1
15 selector:
16 matchLabels:
17 name: test
18 toolforge: tool
19 "web"
20 "tfb"
21 template:
22 metadata:
23 labels:
24 name: test
25 toolforge: tool
26 "web"
27 "tfb"
28 spec:
29 containers:
30 - image:
31 imagePullPolicy: Always
32 name: webservice
33 ports:
34 - containerPort: 8000
35 name: http
36 protocol: TCP
37 resources: {}
39apiVersion: v1
40kind: Service
42 name: test
43 namespace: tool-test
44 labels:
45 name: test
46 toolforge: tool
47 "web"
48 "tfb"
50 ports:
51 - name: http
52 protocol: TCP
53 port: 8000
54 targetPort: 8000
55 selector:
56 name: test
59kind: Ingress
61 name: test-subdomain
62 namespace: tool-test
63 labels:
64 name: test
65 toolforge: tool
66 "web"
67 "tfb"
69 rules:
70 - host:
71 http:
72 paths:
73 - backend:
74 serviceName: test
75 servicePort: 8000

Now we just need to get <CD thing> to make those k8s objects. That's awesome.

Well, that and the buildpack run...but that's the manual version mapped out.

Andrew triaged this task as Medium priority.Jan 12 2021, 5:09 PM
Andrew moved this task from Soon! to Inbox on the cloud-services-team (Kanban) board.

This doesn't work on tools, only toolsbeta, pending T265557 things

Test with:

andrew@toolsbeta-proxy-3:~$ curl -H "Host:" https://localhost -k
Hello to the new buildpack-based world!

I went ahead and put the example through the mill deploying it in toolsbeta from my GitHub repo though the Tekton pipeline instead of from NFS with pack or whatever. The particular builder's resulting image ends up pretty large (218 MB), which that's nowhere near the size of a standard Toolforge image (which is like 600MB easy). By comparison, using a paketo builder, I set up a much more complex ruby image that was only 43MB.

toolsbeta.test@toolsbeta-sgebastion-04:~$ curl -H "Host:" -k
Hello to the new buildpack-based world!

That said, paketo doesn't come with a python build pack at the moment. I'd have to use Heroku or Google's to get that.

I was ablet o build (with buildpacks), deploy (manually) and run a webservice on toolsbeta manually.

toolsbeta.test@toolsbeta-sgebastion-05:~$ toolforge build-list
run_name        status  start_time      end_time        source_url      repo_url        image_name      image_tag       builder_image
test-buildpacks-pipelinerun-9hv4d       ok      2022-04-14T09:54:52Z    2022-04-14T09:56:15Z       python  snap
# Edited the deployment and remove replicasets/pods to force refresh
dcaro@toolsbeta-proxy-3:~$ curl -v -H "Host:" --insecure -o /dev/null
< HTTP/2 200
< server: nginx/1.14.2
< date: Thu, 14 Apr 2022 10:05:56 GMT
< content-type: text/html; charset=utf-8
< content-length: 5371
< strict-transport-security: max-age=31622400
< x-clacks-overhead: GNU Terry Pratchett
< permissions-policy: interest-cohort=()
< content-security-policy-report-only: default-src 'self' 'unsafe-eval' 'unsafe-inline' blob: data: filesystem: mediastream: * * * * * * * * * * * * * * wss://; report-uri;
{ [5371 bytes data]
100  5371  100  5371    0     0   187k      0 --:--:-- --:--:-- --:--:--  194k
* Connection #0 to host left intact

So almost there, missing the webservice cli part, and deploying in tools instead of tolosbeta for the POC.

dcaro renamed this task from Deploy a tool using a buildpack-based image on Toolforge to [] Deploy a tool using a buildpack-based image on Toolforge.Aug 26 2022, 8:31 AM
dcaro raised the priority of this task from Medium to High.Mar 6 2023, 2:48 PM