Page MenuHomePhabricator

Run latest Thumbor on Docker with Buster + Python 3
Open, MediumPublic

Description

Progress saved here: https://github.com/gi11es/thumbor-docker

Current status:

  • Dockerfile successfully running full test suite for Thumbor + Wikimedia Thumbor plugins on Buster/Python 3

Next:

  • Convert Dockerfile to Blubber
  • Run Blubberized Docker image in CI, running the full test suite

Related Objects

StatusSubtypeAssignedTask
OpenNone
Resolvedakosiaris
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedReedy
ResolvedReedy
ResolvedBawolff
ResolvedAnomie
ResolvedBawolff
ResolvedBawolff
ResolvedLegoktm
ResolvedLucas_Werkmeister_WMDE
ResolvedBawolff
Resolvedsbassett
Resolvedsbassett
ResolvedJdforrester-WMF
Resolvedsbassett
Resolvedsbassett
ResolvedReedy
ResolvedReedy
ResolvedJdforrester-WMF
ResolvedReedy
ResolvedReedy
ResolvedReedy
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedReedy
ResolvedReedy
ResolvedReedy
ResolvedJdforrester-WMF
Resolvedhashar
Resolvedhashar
ResolvedJdforrester-WMF
Resolvedhashar
DeclinedMoritzMuehlenhoff
Invalidthcipriani
Resolvedmmodell
Resolvedhashar
ResolvedJoe
ResolvedJMeybohm
ResolvedJMeybohm
DuplicateDzahn
DeclinedDzahn
ResolvedJdforrester-WMF
OpenNone
OpenNone
Opencmassaro
DeclinedNone

Event Timeline

Gilles triaged this task as Medium priority.Nov 5 2020, 2:27 PM

Seeing that there were a lot of missing or outdated Python Debian packages in Buster compared to what the latest Thumbor master needs, I decided to install the minimum via Debian and all the Python dependencies via pip. Next I will look into whether I can turn all those pip calls into local wheel files, which would be a cleaner way to package this for production.

I think that doing all the Debian packaging and backports required would be very complicated and time-consuming. I don't see the point if we're going to generate a machine image anyway.

I got pretty far with Thumbor itself, but ran into a small amount of remaining test errors that I filed upstream: https://github.com/thumbor/thumbor/issues/1326

Progress! Turns out I should have been using pytest instead of nosetest to run the tests. I've filed a small PR to make the python binary used in the Makefile configurable: https://github.com/thumbor/thumbor/pull/1328

This simple Dockerfile now works:

FROM debian:buster

RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y python3 libssl-dev libcurl4-openssl-dev redis memcached imagemagick build-essential cython3 webp ffmpeg gifsicle exiftool libjpeg-turbo-progs libcairo2 python3-distutils curl git coreutils libexiv2-dev libboost-python-dev
WORKDIR /root
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
RUN python3 get-pip.py
RUN git clone -b makefile-python-configurable https://github.com/gi11es/thumbor
WORKDIR /root/thumbor
RUN pip3 install setuptools
RUN make PYTHON=python3 setup test

@jijiki would it be acceptable if the Python dependencies for Thumbor 5 run in Kubernetes are brought via wheel files? In my Docker WIP I'm pulling them via pip.

@Gilles We do not have any other python applications running on Kubernetes, so we are exploring new ground here. I will discuss it a bit more with serviceops to see how we will handle this.

PS. YAY 🎉 !

@Gilles Please go ahead bringing deps via wheel files, we do not have any alternative solution for this :)

It's now complete: https://github.com/gi11es/thumbor-docker

Next I will experiment a little with Blubber to see if it could be used to generate this Dockerfile/docker image.

My 2c. From what I learned from docker books and such, containers and k8s are not recommended for two types of applications: 1- stateful applications (given the ephemeral nature of containers) or 2- High I/O applications (given that it introduce its own network). I use docker for jitsi (the video calling application) and I regret it, it's terrible. I think thumbor might see a performance degradation due to this.

Thumbor is neither stateful nor high I/O at Wikimedia. There's ATS and Varnish in front of it and it reads and write to Swift, and can re-run past requests safely.

oh it's not stateful but I think it's high I/O compared to other applications (maybe not as high as jitsi but higher than other apps in k8s). let's do benchmark and see.

oh it's not stateful but I think it's high I/O compared to other applications (maybe not as high as jitsi but higher than other apps in k8s). let's do benchmark and see.

I assume you mean network I/O. It's true that thumbor has quite a bit of incoming traffic. Averaging over the last 7 days ~100MBs [1]. Naturally, given the nature of the service is to return thumbnails, return traffic is close to 1/25th of that (avg 3.5MBs). That's about 70% of what currently kubernetes manages[2] (some 150MB/s in and out) for the total of services living on it, without any issues or complaints about performance up to now. So traffic wise we are probably going to be a-ok.

But I think you mean that it's latency/jitter wise that we are going to see a performance degradation, presumably due to the in-kernel DNATs when addressing the kubernetes service, as well as the double load balancing that will be happening (LVS+kubernetes). It's true that this will add a few milliseconds of latency. However, given the fact that thumbor's 75th latency percentile lingers around ~1.5 seconds[3], I think we will not notice it.

[1] https://grafana.wikimedia.org/d/XhFPDdMGz/cluster-overview?viewPanel=84&orgId=1&var-site=eqiad&var-cluster=thumbor&var-instance=All&var-datasource=thanos&from=now-7d&to=now
[2] https://grafana.wikimedia.org/d/XhFPDdMGz/cluster-overview?viewPanel=84&orgId=1&from=now-7d&to=now&var-site=eqiad&var-cluster=kubernetes&var-instance=All&var-datasource=thanos
[3] https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor?viewPanel=35&orgId=1

Moving back for re-triage as it's been dormant 6 months in a columnn for things "this quarter".