Page MenuHomePhabricator

Create Python 3.7 buildpack builder for Toolforge
Closed, ResolvedPublic


This buildpack will install Python 3.7, pip/virtualenv, any dependencies in requirements.txt and then start the webserver with uwsgi. More details on how this is working technically is at

Event Timeline

Legoktm created this task.Oct 15 2020, 9:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 15 2020, 9:26 PM

Change 634349 had a related patch set uploaded (by Legoktm; owner: Kunal Mehta):
[operations/docker-images/toollabs-images@master] Add buildpack images ("stacks")

Change 633270 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[cloud/toolforge/buildpacks@master] Initial commit of Python 3.7 builder

Legoktm claimed this task.Oct 15 2020, 9:56 PM

Change 634349 merged by jenkins-bot:
[operations/docker-images/toollabs-images@master] Add buildpack images ("stacks")

We also need to build and publish the builder image using pack. Logically I think it would make sense to do this on the current image builder instance since it can already publish to the docker registry. I filed T266270: Install pack on Toolforge servers for figuring out how to get pack on there.

Change 633270 merged by jenkins-bot:
[cloud/toolforge/buildpacks@master] Initial commit of Python 3.7 builder

Mentioned in SAL (#wikimedia-cloud) [2020-10-29T04:03:31Z] <legoktm> published image (T265686)

Legoktm closed this task as Resolved.Thu, Oct 29, 4:09 AM

It works (locally)!

1user@cloud-dev:~/projects/tool-apt-browser$ pack build test-tool-apt-browser --builder
2latest: Pulling from toolforge-buster0-builder
3e084d51813c0: Pull complete
485d3b276e462: Pull complete
5fc40f3723762: Pull complete
680e5793f88ab: Pull complete
7e38e11bd9ba3: Pull complete
8190c0f6053da: Pull complete
94f9c9b92ac60: Pull complete
10369c70aa0e57: Pull complete
11cdd68c421b73: Pull complete
12c7e8de017a79: Pull complete
1349aaa01735ad: Pull complete
144ca6d97c7b0c: Pull complete
15b6897515cbf8: Pull complete
16a1850d638a19: Pull complete
1789732bc75041: Pull complete
18Digest: sha256:70724160bd6c4ba38e750969d00aa69b4f032c3a5a20e140b25fbed5d054375f
19Status: Downloaded newer image for
20latest: Pulling from toolforge-buster0-run
21e084d51813c0: Already exists
2285d3b276e462: Already exists
23fc40f3723762: Already exists
2480e5793f88ab: Already exists
25e38e11bd9ba3: Already exists
26Digest: sha256:4bedb11c0cbb41b35256800d96293a8817209d1e6a34bf6ecb7b2f26c25b818b
27Status: Downloaded newer image for
280.9.1: Pulling from buildpacksio/lifecycle
294000adbbc3eb: Pull complete
30474f7dcb012d: Pull complete
31Digest: sha256:53bf0e18a734e0c4071aa39b950ed8841f82936e53fb2a0df56c6aa07f9c5023
32Status: Downloaded newer image for buildpacksio/lifecycle:0.9.1
34[detector] org.toolforge.buildpacks.python37 0.0.1
35[detector] org.toolforge.buildpacks.pip 0.0.1
36[detector] org.toolforge.buildpacks.uwsgi 0.0.1
38[analyzer] Previous image with name "test-tool-apt-browser" not found
41[builder] ---> Python 3.7 Buildpack
42[builder] ----> Installing python3.7
43[builder] -----> Detected Aptfile or Stack changes, flushing cache
44[builder] -----> Adding custom repositories
45[builder] -----> Updating apt caches
46[builder] Get:1 buster/updates InRelease [65.4 kB]
47[builder] Get:2 buster/updates/main amd64 Packages [304 kB]
48[builder] Get:3 buster InRelease [121 kB]
49[builder] Get:4 buster-updates InRelease [51.9 kB]
50[builder] Get:5 buster/main amd64 Packages [10.7 MB]
51[builder] Get:6 buster-updates/main amd64 Packages [8728 B]
52[builder] rm: cannot remove '/var/cache/apt/archives/partial/*.deb': Permission denied
53[builder] Fetched 11.3 MB in 4s (2837 kB/s)
54[builder] Reading package lists...
55[builder] -----> Fetching .debs for python3
56[builder] Reading package lists...
57[builder] Building dependency tree...
58[builder] The following additional packages will be installed:
59[builder] libmpdec2 libpython3-stdlib libpython3.7-minimal libpython3.7-stdlib
60[builder] libreadline7 libsqlite3-0 mime-support python3-minimal python3.7
61[builder] python3.7-minimal readline-common
62[builder] Suggested packages:
63[builder] python3-doc python3-tk python3-venv python3.7-venv python3.7-doc
64[builder] binfmt-support readline-doc
65[builder] Recommended packages:
66[builder] file
67[builder] The following NEW packages will be installed:
68[builder] libmpdec2 libpython3-stdlib libpython3.7-minimal libpython3.7-stdlib
69[builder] libreadline7 libsqlite3-0 mime-support python3 python3-minimal python3.7
70[builder] python3.7-minimal readline-common
71[builder] 0 upgraded, 12 newly installed, 0 to remove and 12 not upgraded.
72[builder] Need to get 5486 kB of archives.
73[builder] After this operation, 25.2 MB of additional disk space will be used.
74[builder] Get:1 buster/main amd64 libpython3.7-minimal amd64 3.7.3-2+deb10u2 [589 kB]
75[builder] Get:2 buster/main amd64 python3.7-minimal amd64 3.7.3-2+deb10u2 [1731 kB]
76[builder] Get:3 buster/main amd64 python3-minimal amd64 3.7.3-1 [36.6 kB]
77[builder] Get:4 buster/main amd64 mime-support all 3.62 [37.2 kB]
78[builder] Get:5 buster/main amd64 libmpdec2 amd64 2.4.2-2 [87.2 kB]
79[builder] Get:6 buster/main amd64 readline-common all 7.0-5 [70.6 kB]
80[builder] Get:7 buster/main amd64 libreadline7 amd64 7.0-5 [151 kB]
81[builder] Get:8 buster/main amd64 libsqlite3-0 amd64 3.27.2-3 [641 kB]
82[builder] Get:9 buster/main amd64 libpython3.7-stdlib amd64 3.7.3-2+deb10u2 [1732 kB]
83[builder] Get:10 buster/main amd64 python3.7 amd64 3.7.3-2+deb10u2 [330 kB]
84[builder] Get:11 buster/main amd64 libpython3-stdlib amd64 3.7.3-1 [20.0 kB]
85[builder] Get:12 buster/main amd64 python3 amd64 3.7.3-1 [61.5 kB]
86[builder] Fetched 5486 kB in 2s (2577 kB/s)
87[builder] Download complete and in download only mode
88[builder] -----> Fetching .debs for python3-dev
89[builder] Reading package lists...
90[builder] Building dependency tree...
91[builder] The following additional packages will be installed:
92[builder] dh-python libexpat1-dev libmpdec2 libpython3-dev libpython3-stdlib
93[builder] libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib
94[builder] libreadline7 libsqlite3-0 mime-support python3 python3-distutils
95[builder] python3-lib2to3 python3-minimal python3.7 python3.7-dev python3.7-minimal
96[builder] readline-common
97[builder] Suggested packages:
98[builder] python3-doc python3-tk python3-venv python3.7-venv python3.7-doc
99[builder] binfmt-support readline-doc
100[builder] Recommended packages:
101[builder] file
102[builder] The following NEW packages will be installed:
103[builder] dh-python libexpat1-dev libmpdec2 libpython3-dev libpython3-stdlib
104[builder] libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib
105[builder] libreadline7 libsqlite3-0 mime-support python3 python3-dev python3-distutils
106[builder] python3-lib2to3 python3-minimal python3.7 python3.7-dev python3.7-minimal
107[builder] readline-common
108[builder] 0 upgraded, 21 newly installed, 0 to remove and 12 not upgraded.
109[builder] Need to get 50.9 MB/56.4 MB of archives.
110[builder] After this operation, 117 MB of additional disk space will be used.
111[builder] Get:1 buster/main amd64 python3-lib2to3 all 3.7.3-1 [76.7 kB]
112[builder] Get:2 buster/main amd64 python3-distutils all 3.7.3-1 [142 kB]
113[builder] Get:3 buster/main amd64 dh-python all 3.20190308 [99.3 kB]
114[builder] Get:4 buster/main amd64 libexpat1-dev amd64 2.2.6-2+deb10u1 [153 kB]
115[builder] Get:5 buster/main amd64 libpython3.7 amd64 3.7.3-2+deb10u2 [1498 kB]
116[builder] Get:6 buster/main amd64 libpython3.7-dev amd64 3.7.3-2+deb10u2 [48.4 MB]
117[builder] Get:7 buster/main amd64 libpython3-dev amd64 3.7.3-1 [20.1 kB]
118[builder] Get:8 buster/main amd64 python3.7-dev amd64 3.7.3-2+deb10u2 [510 kB]
119[builder] Get:9 buster/main amd64 python3-dev amd64 3.7.3-1 [1264 B]
120[builder] Fetched 50.9 MB in 15s (3355 kB/s)
121[builder] Download complete and in download only mode
122[builder] -----> Installing dh-python_3.20190308_all.deb
123[builder] -----> Installing libexpat1-dev_2.2.6-2+deb10u1_amd64.deb
124[builder] -----> Installing libmpdec2_2.4.2-2_amd64.deb
125[builder] -----> Installing libpython3-dev_3.7.3-1_amd64.deb
126[builder] -----> Installing libpython3-stdlib_3.7.3-1_amd64.deb
127[builder] -----> Installing libpython3.7-dev_3.7.3-2+deb10u2_amd64.deb
128[builder] -----> Installing libpython3.7-minimal_3.7.3-2+deb10u2_amd64.deb
129[builder] -----> Installing libpython3.7-stdlib_3.7.3-2+deb10u2_amd64.deb
130[builder] -----> Installing libpython3.7_3.7.3-2+deb10u2_amd64.deb
131[builder] -----> Installing libreadline7_7.0-5_amd64.deb
132[builder] -----> Installing libsqlite3-0_3.27.2-3_amd64.deb
133[builder] -----> Installing mime-support_3.62_all.deb
134[builder] -----> Installing python3-dev_3.7.3-1_amd64.deb
135[builder] -----> Installing python3-distutils_3.7.3-1_all.deb
136[builder] -----> Installing python3-lib2to3_3.7.3-1_all.deb
137[builder] -----> Installing python3-minimal_3.7.3-1_amd64.deb
138[builder] -----> Installing python3.7-dev_3.7.3-2+deb10u2_amd64.deb
139[builder] -----> Installing python3.7-minimal_3.7.3-2+deb10u2_amd64.deb
140[builder] -----> Installing python3.7_3.7.3-2+deb10u2_amd64.deb
141[builder] -----> Installing python3_3.7.3-1_amd64.deb
142[builder] -----> Installing readline-common_7.0-5_all.deb
143[builder] -----> Writing profile script
144[builder] -----> Rewrite package-config files
145[builder] ----> Installing pip
146[builder] % Total % Received % Xferd Average Speed Time Time Time Current
147[builder] Dload Upload Total Spent Left Speed
148[builder] 100 1841k 100 1841k 0 0 2596k 0 --:--:-- --:--:-- --:--:-- 2593k
149[builder] OK
150[builder] Collecting pip
151[builder] Downloading pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
152[builder] Collecting setuptools
153[builder] Downloading setuptools-50.3.2-py3-none-any.whl (785 kB)
154[builder] Collecting wheel
155[builder] Downloading wheel-0.35.1-py2.py3-none-any.whl (33 kB)
156[builder] Installing collected packages: pip, setuptools, wheel
157[builder] Successfully installed pip-20.2.4 setuptools-50.3.2 wheel-0.35.1
158[builder] ----> Installing virtualenv
159[builder] Collecting virtualenv
160[builder] Downloading virtualenv-20.1.0-py2.py3-none-any.whl (4.9 MB)
161[builder] Collecting filelock<4,>=3.0.0
162[builder] Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
163[builder] Collecting six<2,>=1.9.0
164[builder] Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
165[builder] Collecting importlib-metadata<3,>=0.12; python_version < "3.8"
166[builder] Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
167[builder] Collecting distlib<1,>=0.3.1
168[builder] Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
169[builder] Collecting appdirs<2,>=1.4.3
170[builder] Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
171[builder] Collecting zipp>=0.5
172[builder] Downloading zipp-3.4.0-py3-none-any.whl (5.2 kB)
173[builder] Installing collected packages: filelock, six, zipp, importlib-metadata, distlib, appdirs, virtualenv
174[builder] Successfully installed appdirs-1.4.4 distlib-0.3.1 filelock-3.0.12 importlib-metadata-2.0.0 six-1.15.0 virtualenv-20.1.0 zipp-3.4.0
175[builder] ---> pip Buildpack
176[builder] ----> Creating virtualenv
177[builder] created virtual environment in 586ms
178[builder] creator CPython3Posix(dest=/layers/org.toolforge.buildpacks.pip/pip/venv, clear=False, global=False)
179[builder] seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/tfb/.local/share/virtualenv)
180[builder] added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
181[builder] activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
182[builder] Requirement already up-to-date: pip in /layers/org.toolforge.buildpacks.pip/pip/venv/lib/python3.7/site-packages (20.2.4)
183[builder] ----> Installing from requirements.txt
184[builder] Collecting Flask
185[builder] Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
186[builder] Collecting flask-caching
187[builder] Downloading Flask_Caching-1.9.0-py2.py3-none-any.whl (33 kB)
188[builder] Collecting flask-dataapi
189[builder] Downloading flask_dataapi-0.2.1-py3-none-any.whl (15 kB)
190[builder] Collecting python-debian
191[builder] Downloading python_debian-0.1.38-py3-none-any.whl (70 kB)
192[builder] Collecting requests
193[builder] Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
194[builder] Collecting toolforge
195[builder] Downloading toolforge-4.3.2-py3-none-any.whl (15 kB)
196[builder] Collecting click>=5.1
197[builder] Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
198[builder] Collecting Jinja2>=2.10.1
199[builder] Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
200[builder] Collecting itsdangerous>=0.24
201[builder] Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
202[builder] Collecting Werkzeug>=0.15
203[builder] Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
204[builder] Collecting chardet
205[builder] Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
206[builder] Collecting six
207[builder] Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
208[builder] Collecting idna<3,>=2.5
209[builder] Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
210[builder] Collecting certifi>=2017.4.17
211[builder] Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
212[builder] Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
213[builder] Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
214[builder] Collecting pymysql
215[builder] Downloading PyMySQL-0.10.1-py2.py3-none-any.whl (47 kB)
216[builder] Collecting MarkupSafe>=0.23
217[builder] Downloading MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (27 kB)
218[builder] Installing collected packages: click, MarkupSafe, Jinja2, itsdangerous, Werkzeug, Flask, flask-caching, flask-dataapi, chardet, six, python-debian, idna, certifi, urllib3, requests, pymysql, toolforge
219[builder] Successfully installed Flask-1.1.2 Jinja2-2.11.2 MarkupSafe-1.1.1 Werkzeug-1.0.1 certifi-2020.6.20 chardet-3.0.4 click-7.1.2 flask-caching-1.9.0 flask-dataapi-0.2.1 idna-2.10 itsdangerous-1.1.0 pymysql-0.10.1 python-debian-0.1.38 requests-2.24.0 six-1.15.0 toolforge-4.3.2 urllib3-1.25.11
220[builder] ---> uwsgi Buildpack
221[builder] ----> Installing uwsgi
222[builder] created virtual environment in 117ms
223[builder] creator CPython3Posix(dest=/layers/org.toolforge.buildpacks.uwsgi/uwsgi/venv, clear=False, global=False)
224[builder] seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/tfb/.local/share/virtualenv)
225[builder] added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
226[builder] activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
227[builder] Requirement already up-to-date: pip in /layers/org.toolforge.buildpacks.uwsgi/uwsgi/venv/lib/python3.7/site-packages (20.2.4)
228[builder] Collecting uwsgi==2.0.18
229[builder] Downloading uwsgi-2.0.18.tar.gz (801 kB)
230[builder] Building wheels for collected packages: uwsgi
231[builder] Building wheel for uwsgi ( started
232[builder] Building wheel for uwsgi ( finished with status 'done'
233[builder] Created wheel for uwsgi: filename=uWSGI-2.0.18-cp37-cp37m-linux_x86_64.whl size=505966 sha256=bd4bb041517da3e3c99b4164114832b1e6e56262555be07d2039562cb54e1ba0
234[builder] Stored in directory: /tmp/pip-ephem-wheel-cache-9bausrtf/wheels/e7/d0/f2/b93905969980eaf9fe18183b0c43470d63b13fcd3a1f6c8efe
235[builder] Successfully built uwsgi
236[builder] Installing collected packages: uwsgi
237[builder] Successfully installed uwsgi-2.0.18
238[builder] ----> Configuring uwsgi web launcher
240[exporter] Adding layer 'org.toolforge.buildpacks.python37:python37'
241[exporter] Adding layer 'org.toolforge.buildpacks.pip:pip'
242[exporter] Adding layer 'org.toolforge.buildpacks.uwsgi:uwsgi'
243[exporter] Adding 1/1 app layer(s)
244[exporter] Adding layer 'launcher'
245[exporter] Adding layer 'config'
246[exporter] Adding layer 'process-types'
247[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
248[exporter] Adding label ''
249[exporter] Adding label 'io.buildpacks.project.metadata'
250[exporter] Setting default process type 'web'
251[exporter] *** Images (99c22f5f2c65):
252[exporter] test-tool-apt-browser
253[exporter] Adding cache layer 'org.toolforge.buildpacks.python37:apt'
254[exporter] Adding cache layer 'org.toolforge.buildpacks.python37:python37'
255[exporter] Adding cache layer 'org.toolforge.buildpacks.pip:pip'
256[exporter] Adding cache layer 'org.toolforge.buildpacks.uwsgi:uwsgi'
257Successfully built image test-tool-apt-browser
258user@cloud-dev:~/projects/tool-apt-browser$ pack inspect-image test-tool-apt-browser
259Inspecting image: test-tool-apt-browser
262(not present)
266Stack: org.toolforge.buildpacks.stack.buster0
268Base Image:
269 Reference: 83bcc447231179edacbcea219e0bec8bab16f2abd6d7b142ef8c4b3e4e29ad56
270 Top Layer: sha256:9e5007eefe05d1e984897e9dc7beed6cf202715b4b3a362195660166b2ed6324
272Run Images:
277 org.toolforge.buildpacks.python37 0.0.1
278 org.toolforge.buildpacks.pip 0.0.1
279 org.toolforge.buildpacks.uwsgi 0.0.1
283 web (default) bash source ./export && /layers/org.toolforge.buildpacks.uwsgi/uwsgi/venv/bin/uwsgi --http-socket :8000 --callable app --manage-script-name --workers 4 --mount "/" --die-on-term --strict --master --venv /layers/org.toolforge.buildpacks.pip/pip/venv
285user@cloud-dev:~/projects/tool-apt-browser$ docker run test-tool-apt-browser
286*** Starting uWSGI 2.0.18 (64bit) on [Thu Oct 29 04:07:44 2020] ***
287compiled with version: 8.3.0 on 29 October 2020 04:06:38
288os: Linux-4.19.152-1.pvops.qubes.x86_64 #1 SMP Sun Oct 18 12:03:19 UTC 2020
289nodename: 0d78ba7a67d2
290machine: x86_64
291clock source: unix
292detected number of CPU cores: 2
293current working directory: /workspace
294detected binary path: /layers/org.toolforge.buildpacks.uwsgi/uwsgi/venv/bin/uwsgi
295!!! no internal routing support, rebuild with pcre support !!!
296your memory page size is 4096 bytes
297detected max file descriptor number: 1048576
298lock engine: pthread robust mutexes
299thunder lock: disabled (you can enable it with --thunder-lock)
300uwsgi socket 0 bound to TCP address :8000 fd 3
301Python version: 3.7.3 (default, Jul 25 2020, 13:03:44) [GCC 8.3.0]
302PEP 405 virtualenv detected: /layers/org.toolforge.buildpacks.pip/pip/venv
303Set PythonHome to /layers/org.toolforge.buildpacks.pip/pip/venv
304*** Python threads support is disabled. You can enable it with --enable-threads ***
305Python main interpreter initialized at 0x5cb7d1b16f70
306your server socket listen backlog is limited to 100 connections
307your mercy for graceful operations on workers is 60 seconds
308mapped 364520 bytes (355 KB) for 4 cores
309*** Operational MODE: preforking ***
310mounting on /
311/layers/org.toolforge.buildpacks.pip/pip/venv/lib/python3.7/site-packages/flask_caching/ UserWarning: Flask-Caching: CACHE_TYPE is set to null, caching is effectively disabled.
312 "Flask-Caching: CACHE_TYPE is set to null, "
313WSGI app 0 (mountpoint='/') ready in 1 seconds on interpreter 0x5cb7d1b16f70 pid: 1 (default app)
314*** uWSGI is running in multiple interpreter mode ***
315spawned uWSGI master process (pid: 1)
316spawned uWSGI worker 1 (pid: 11, cores: 1)
317spawned uWSGI worker 2 (pid: 12, cores: 1)
318spawned uWSGI worker 3 (pid: 13, cores: 1)
319spawned uWSGI worker 4 (pid: 14, cores: 1)
320^CSIGINT/SIGQUIT received...killing workers...
321worker 1 buried after 1 seconds
322worker 2 buried after 1 seconds
323worker 3 buried after 1 seconds
324worker 4 buried after 1 seconds
325goodbye to uWSGI.