(not sure how familiar you are with all of this, so apologies beforehand if I'm being too verbose/repetitive)

We don't have to upload them to jessie-backports — we can and that'd be great, but note that it's a little more complicated than just uploading them: to do so, we'd have to ask each of those maintainers first, per backports policy, and then -to be good citizens- we'd have to keep those backports up-to-date with what's in testing. It's not too much work, but it's not trivial either.

What we can do too for our use case would be to include those in our own jessie-wikimedia apt repository, so these can be used by our infrastructure. Since thumbor is not in Debian (yet? :) anyway, having all those backports in upstream jessie-backports might not be of much use to anyone anyway.

As for python-statsd, this was uploaded to Debian unstable by @hashar and sponsored by myself. @fgiunchedi did a couple more uploads more recently, I see, but it still has an RC bug and hasn't been in testing for 179 days. We, Wikimedian Debian maintainers :) should definitely fix this (or orphan the package).

Finally — thanks so much for doing all this work! I would found it totally reasonable if you had said "we want these deployed, opsens/Debianites figure it out", so seeing you going above and beyond to prepare all this is incredibly appreciated. Kudos.

Most of them are maintained by Marcelo Jorge Vieira who packaged Thumbor (or attempted to) a while ago:

https://packages.debian.org/stretch/python-derpconf
https://packages.debian.org/stretch/python-nose-yanc
https://packages.debian.org/stretch/python-pexif
https://packages.debian.org/stretch/python-preggy
https://packages.debian.org/stretch/python-pyvows
https://packages.debian.org/stretch/python-tornado-pyvows

The ones that have other maintainers are:

https://packages.debian.org/stretch/python-scipy
https://packages.debian.org/sid/python-statsd (as you know)

I think there is value to going as far as packaging Thumbor for Debian, as it will spread usage of the version we're using ourselves.

Regarding the statsd package, the RC bug is fixed by an upstream fix, which is in my "backport" (which is in fact a "new upstream release": https://github.com/gi11es/thumbor-debian-backports/blob/master/python-statsd/python-statsd_3.2.1-2_source.changes ). Ironically the fix came a couple of days after it was last packaged by @fgiunchedi

If all these backports I've prepared make it into debian, all that's left is packaging Thumbor itself. Conveniently, Marcelo's work from several major versions ago is still in their git repo and can serve as a starting point: https://github.com/thumbor/thumbor/tree/master/debian

And in terms of remaining dependencies that Thumbor needs to be packaged and aren't currently, they're all for tests. Most of them look like simple Python libraries that should be easy to package. A notable exception is Sentry, but I thought I could have the Thumbor debian package skip that test on build, considering how difficult Sentry would be to package on its own. Basically a lot of Thumbor's test dependencies are testing optional features of Thumbor. My take on this is that I can try to package as much of those dependencies as possible as long as they're easy, and skip tests like Sentry integration, considering that from Debian's perspective, there's no Sentry package yet anyway.

I prepared all those backports precisely because after discussing with @fgiunchedi it looked like Thumbor's dependency list was reasonable and it was worth attempting to package it *and* get it into debian.

TODO: fix libthumbor's package description (copypasta)

+1 to what @faidon said, thanks @Gilles indeed for all the effort you've put in this! I'll start with python-statsd as the lowest hanging fruit.

re: test dependencies, if the thumbor package builds/installs/runs that's a good first step even skipping tests on build due to deps, let us know how we can help with the missing test dependencies

So far I'm not running into major issues packaging any of the dependencies with their own tests running during the build. The install dependencies are already all covered and what's listed in this task's description is what I have left to cover all the test dependencies.

re: python-statsd I've uploaded the 3.1.2-2 to unstable and will backport once it has migrated to testing (cc @hashar)

pillow 3.2.0-2~bpo8+1 uploaded to jessie-backports, should appear in the next few days

The only remaining dependency, the upstream update of scikit-image is proving difficult. The package is massive, its packaging is complicated. It has massive subdependencies like pandas too (which I've taken care of, but still). Right now I'm only contemplating two test failures, so I could just skip them hoping that there's nothing to it and hope that's all it takes, but it might be too much trouble for what it's worth.

Quite simply, the newer version of scikit-image (and its dependencies that required packaging work: dask, cloudpickle, partd, locket, spur, pandas, toolz) is only used by Thumbor tests for a single method:

https://github.com/thumbor/thumbor/blob/7c18eead4a1480d3f95011c9a36661ea1ef4e3e4/tests/base.py#L23

Now, using visual comparison (DSSIM) is critical for some Thumbor tests. But the massive size of scikit-image and its dependencies seems overkill for this.

Therefore, before I spend more energy trying to complete the difficult package update for scikit-image, I thought I might explore using an alternative for Thumbor's tests:

https://github.com/jterrace/pyssim

The dependency list for that module looks way more reasonable. If I make it work, there's a good chance it will be merged upstream, as the Thumbor guys are quite open to this sort of improvement.

@fgiunchedi pointed out that pyssim has no license: https://github.com/jterrace/pyssim/issues/14

I've tracked down the original author of the code this is based on and emailed him to ask if he could specify one.

At my request, the original author set a license for his code and picked MIT. I passed that information to the maintainer of pyssim, who did the same: https://github.com/jterrace/pyssim/commit/dfac1217664908a457245f01a81c2873fa8ca62f

I then sent the PR to Thumbor to replace scikit-image with pyssim: https://github.com/thumbor/thumbor/pull/732#issuecomment-222374650

PR merged: https://github.com/thumbor/thumbor/commit/be582f5620281efbee635e043537f0a7bd1a696c

I've uploaded python-statsd and pexif to jessie-backports, they should appear in the next few days

yanc uploaded too, when that is approved we can also go ahead with preggy -> pyvows -> derpconf

I've started working on https://wiki.debian.org/Python/Thumbor as Marcelo requested and I'm almost done filing ITPs. I'll add a column for jessie and jessie-backports as well, to highlight what needs to be backported.

Note to self: rename the python-remotecv package to remotecv as it's a standalone program, not a python library. Same for python-pyres.

Upstream issued a new major version at my request, the current Debian package I've prepared needs to be updated to Thumbor 6.1.0

Pushed to our internal repo by @fgiunchedi