Seeing that there were a lot of missing or outdated Python Debian packages in Buster compared to what the latest Thumbor master needs, I decided to install the minimum via Debian and all the Python dependencies via pip. Next I will look into whether I can turn all those pip calls into local wheel files, which would be a cleaner way to package this for production.

I think that doing all the Debian packaging and backports required would be very complicated and time-consuming. I don't see the point if we're going to generate a machine image anyway.

I got pretty far with Thumbor itself, but ran into a small amount of remaining test errors that I filed upstream: https://github.com/thumbor/thumbor/issues/1326

Progress! Turns out I should have been using pytest instead of nosetest to run the tests. I've filed a small PR to make the python binary used in the Makefile configurable: https://github.com/thumbor/thumbor/pull/1328

This simple Dockerfile now works:

FROM debian:buster

RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y python3 libssl-dev libcurl4-openssl-dev redis memcached imagemagick build-essential cython3 webp ffmpeg gifsicle exiftool libjpeg-turbo-progs libcairo2 python3-distutils curl git coreutils libexiv2-dev libboost-python-dev
WORKDIR /root
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
RUN python3 get-pip.py
RUN git clone -b makefile-python-configurable https://github.com/gi11es/thumbor
WORKDIR /root/thumbor
RUN pip3 install setuptools
RUN make PYTHON=python3 setup test

@jijiki would it be acceptable if the Python dependencies for Thumbor 5 run in Kubernetes are brought via wheel files? In my Docker WIP I'm pulling them via pip.

@Gilles We do not have any other python applications running on Kubernetes, so we are exploring new ground here. I will discuss it a bit more with serviceops to see how we will handle this.

PS. YAY 🎉 !

@Gilles Please go ahead bringing deps via wheel files, we do not have any alternative solution for this :)

It's now complete: https://github.com/gi11es/thumbor-docker

Next I will experiment a little with Blubber to see if it could be used to generate this Dockerfile/docker image.

My 2c. From what I learned from docker books and such, containers and k8s are not recommended for two types of applications: 1- stateful applications (given the ephemeral nature of containers) or 2- High I/O applications (given that it introduce its own network). I use docker for jitsi (the video calling application) and I regret it, it's terrible. I think thumbor might see a performance degradation due to this.

Thumbor is neither stateful nor high I/O at Wikimedia. There's ATS and Varnish in front of it and it reads and write to Swift, and can re-run past requests safely.

oh it's not stateful but I think it's high I/O compared to other applications (maybe not as high as jitsi but higher than other apps in k8s). let's do benchmark and see.

In T267327#6659813, @Ladsgroup wrote:

oh it's not stateful but I think it's high I/O compared to other applications (maybe not as high as jitsi but higher than other apps in k8s). let's do benchmark and see.

I assume you mean network I/O. It's true that thumbor has quite a bit of incoming traffic. Averaging over the last 7 days ~100MBs [1]. Naturally, given the nature of the service is to return thumbnails, return traffic is close to 1/25th of that (avg 3.5MBs). That's about 70% of what currently kubernetes manages[2] (some 150MB/s in and out) for the total of services living on it, without any issues or complaints about performance up to now. So traffic wise we are probably going to be a-ok.

But I think you mean that it's latency/jitter wise that we are going to see a performance degradation, presumably due to the in-kernel DNATs when addressing the kubernetes service, as well as the double load balancing that will be happening (LVS+kubernetes). It's true that this will add a few milliseconds of latency. However, given the fact that thumbor's 75th latency percentile lingers around ~1.5 seconds[3], I think we will not notice it.

[1] https://grafana.wikimedia.org/d/XhFPDdMGz/cluster-overview?viewPanel=84&orgId=1&var-site=eqiad&var-cluster=thumbor&var-instance=All&var-datasource=thanos&from=now-7d&to=now
[2] https://grafana.wikimedia.org/d/XhFPDdMGz/cluster-overview?viewPanel=84&orgId=1&from=now-7d&to=now&var-site=eqiad&var-cluster=kubernetes&var-instance=All&var-datasource=thanos
[3] https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor?viewPanel=35&orgId=1

Moving back for re-triage as it's been dormant 6 months in a columnn for things "this quarter".

Change 771416 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/debs/python-thumbor-wikimedia@master] WIP: build docker images using blubber and pip dependencies

https://gerrit.wikimedia.org/r/771416

Change 771416 abandoned by Hnowlan:

[operations/debs/python-thumbor-wikimedia@master] WIP: build docker images using blubber and pip dependencies

Reason:

https://gerrit.wikimedia.org/r/771416