Page MenuHomePhabricator

Build Thumbor packages for buster
Closed, ResolvedPublic0 Estimated Story Points


Build Thumbor dependancies for Buster

  • thumbor
  • python-thumbor-community-core
  • python-manhole
  • python-thumbor-wikimedia

(All build fine on a WMCS Buster machine, needs to be redone on our real packaging host)

Packages in buster-wikimedia main:

  • python-manhole: 1.3.0-1+deb10 (python-manhole, python3-manhole, manhole-cli)
  • thumbor_6.5.1-7+wmf1+buster1
  • python-thumbor-community-core_0.4.0-1+deb10u1
  • python-thumbor-wikimedia_2.5-1+deb10u1

Event Timeline

jijiki triaged this task as Medium priority.Apr 22 2019, 8:25 PM
jijiki created this task.

Thumbor was very straightforward, it only required a tiny patch to relax a python version check in (bringing it in line with thumbor master). My build is available on buster.thumbor.eqiad.wmflabs, in my home directory. It installs and runs fine.

I couldn't find a repo for the python-thumbor-community-core package Debian sources. I assume that the build we have is based on what I put together here originally:

Likewise, I fixed it up (required backporting an upstream bugfix) and built it on buster.thumbor.eqiad.wmflabs, in my home directory.

Same story got python-manhole, took it from It build as-is. To be found with the other ones on buster.thumbor.eqiad.wmflabs, in my home directory.

Change 505727 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/software/thumbor-plugins@master] Buster compatibility

As reference:

root@install1002:/srv/wikimedia# reprepro lsbycomponent thumbor
thumbor |        6.3.2+git20170607-1 |  jessie-wikimedia | backports | amd64, source
thumbor | 6.3.2+git20170607-1+deb9u1 | stretch-wikimedia |      main | amd64, source

root@install1002:/srv/wikimedia# reprepro lsbycomponent python-thumbor-community-core
python-thumbor-community-core |    0.4.0-0wmf1 |  jessie-wikimedia | backports | amd64, i386, source
python-thumbor-community-core | 0.4.0-1+deb9u1 | stretch-wikimedia |      main | amd64, i386, source

root@install1002:/srv/wikimedia# reprepro lsbycomponent python-manhole
python-manhole | 1.3.0-1 |  jessie-wikimedia | backports | amd64, i386
python-manhole | 1.3.0-1 | stretch-wikimedia |      main | amd64, i386

@elukey are the sources for these debian packages in version control somewhere? I ended up packaging the current sid version of Thumbor, which is newer (6.5.1).

@elukey are the sources for these debian packages in version control somewhere? I ended up packaging the current sid version of Thumbor, which is newer (6.5.1).

For simple rebuilds (as for the Thumbor packages) we simply pull the existing Debian source via "apt-get source thumbor=$VERSION" and rebuild it. boron is restricted, but AFAIK we had a role(builder) host in labs in the past.

That makes sense. I'll try that for the thumbor package, I think it's best to decouple the thumbor version upgrade from the Buster upgrade.

Me or @jijiki can pick the deb sources (with patches if needed) and rebuild them for buster-wikimedia on boron, and then upload them. Basically re-doing what has been done in labs in a "prod" environment, that's it :)

I am happy to do it, but: I am afraid it will have to wait for next week, as this is a short week for me:)

I need to find a working solution on labs first. Currently either I use thumbor-6.3.2+git20170607 and it has a bunch of test failures (package won't build). I haven't looked into a potential source for those, but there seems to be a lot. thumbor-6.5.1, on the other hand, builds fine. But I have a very mysterious hanging test in python-thumbor-wikimedia.

I need to find the root cause for one or the other, basically.

I'm started to get an idea of what's happening with 6.5.1 getting stuck on some of our thumbor plugin tests.

One of our tests, test_multi_max_paths, seems to leave a lock behind in BaseHandler.url_locks (via acquire_url_lock/release_url_lock).

I see that this is actually some upstream code I wrote a long time ago:

The multi handler, that this test leaving the lock behind covers, is a feature that we didn't end up using. It's something I wrote very early on in the migration to Thumbor, it was an optimisation for prerendering. We could delete that code entirely, TBH. Which means we can safely skip the multi_handler test (as a result, the other test doesn't get stuck anymore).

I'll still check if I can find the root cause of the issue and fix it if it's something obvious.

@Gilles ping me when you think we are ready to build packages for buster, we would do it soon either way

Change 505837 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Remove unused "multi" thumbor handler

Change 505727 merged by Gilles:
[operations/software/thumbor-plugins@master] Buster compatibility

Change 506115 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/software/thumbor-plugins@master] Version bump

Change 506115 merged by Gilles:
[operations/software/thumbor-plugins@master] Version bump

Change 506116 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.5

@jijiki this is all good to go, successfully built on buster.thumbor.eqiad.wmflabs python-thumbor-community-core and thumbor need tiny patches (you'll find them in there). And python-thumbor-wikimedia needs the upgrade/patch above. Manhole build as-is.

In the end I had to upgrade Thumbor to the latest currently on Debian unstable (6.5.1) because the version we're currently running seemed to be having a lot of issues on Buster out of the box. It didn't seem worth the time to figure those out when upgrading made the issues go away.

Change 506116 merged by Effie Mouzeli:
[operations/debs/python-thumbor-wikimedia@master] Upgrade to 2.5

jijiki updated the task description. (Show Details)

@Gilles All packages have been rebuilt and added to buster-wikimedia main repo. Please reopen if we have any issues.

Change 505837 merged by Effie Mouzeli:
[operations/puppet@production] Remove unused "multi" thumbor handler