Service-based thumbnailing re-architecture in production with Thumbor
Open, NormalPublic

Description

The goal of this task is to remove our dependency on MediaWiki for generating thumbnails in Wikimedia production.

This task does not include moving MediaWiki to a full sha1-based URL scheme for thumbnails and originals. That can be achieved separately (T66214, T19577).

It does not include getting rid of Swift for thumbnail storage, which will be tackled in a follow-up project.

It does not include getting rid of Mediawiki parsing originals metadata, as this is not a task Thumbor was designed to do. Another service would be needed to remove that functionality from Mediawiki.

Strategy for deployment in Wikimedia production:

  • Thumbor is stateless and acts as drop-in for current MediaWiki PHP image scalers
  • Thumbor will remain behind Varnish
  • Thumbor will be integrated at the Swift level (rewrite.py)

Related Objects

StatusAssignedTask
OpenGilles
ResolvedGilles
Resolveddpatrick
InvalidGilles
ResolvedGilles
DuplicateGilles
DuplicateGilles
DuplicateGilles
ResolvedGilles
Resolvedfgiunchedi
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
InvalidGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
OpenNone
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
DeclinedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
Resolvedfgiunchedi
ResolvedCmjohnson
OpenGilles
StalledGilles
Openfgiunchedi
There are a very large number of changes, so older changes are hidden. Show Older Changes
Gilles updated the task description. (Show Details)Jul 26 2016, 8:18 AM
Gilles updated the task description. (Show Details)Jul 26 2016, 8:20 AM
Gilles updated the task description. (Show Details)
Gilles renamed this task from Service-based thumbnailing re-architecture in production to Service-based thumbnailing re-architecture in production with Thumbor.

Change 309029 had a related patch set uploaded (by Filippo Giunchedi):
swift: disable thumbor shadow traffic

https://gerrit.wikimedia.org/r/309029

Change 309029 merged by Filippo Giunchedi:
swift: disable thumbor shadow traffic

https://gerrit.wikimedia.org/r/309029

Gilles moved this task from Backlog to Doing on the Thumbor board.Oct 13 2016, 5:54 AM
Gilles moved this task from Doing to Backlog on the Thumbor board.Nov 24 2016, 4:00 PM
Gilles moved this task from Backlog to Doing on the Performance-Team board.Dec 6 2016, 12:29 AM
Gilles moved this task from Backlog to Doing on the Thumbor board.Dec 6 2016, 2:25 PM
Gilles moved this task from Doing to Next-up on the Performance-Team board.Dec 22 2016, 10:42 AM

Change 358599 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: use thumbor.svc for codfw too

https://gerrit.wikimedia.org/r/358599

Change 358599 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: use thumbor.svc for codfw too

https://gerrit.wikimedia.org/r/358599

ori added a subscriber: ori.Jun 13 2017, 6:26 PM

I know you are not 100% done, but still: congratulations, Sirs! It's really impressive to see a long arc of design and implementation work and operational planning come to fruition. Kudos!

Gilles moved this task from Next-up to Doing on the Performance-Team board.Jun 14 2017, 7:36 PM
Gilles updated the task description. (Show Details)Jun 19 2017, 1:39 PM

Change 360307 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: have thumbor.svc alert critical

https://gerrit.wikimedia.org/r/360307

Change 360307 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: have thumbor.svc alert critical

https://gerrit.wikimedia.org/r/360307

Change 360309 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Increase Swift timeout for Thumbor

https://gerrit.wikimedia.org/r/360309

Change 360309 merged by Filippo Giunchedi:
[operations/puppet@production] Increase Swift timeout for Thumbor

https://gerrit.wikimedia.org/r/360309

Change 360350 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thumbor: use jessie-backports as target release for python-thumbor-wikimedia

https://gerrit.wikimedia.org/r/360350

Change 360350 merged by Filippo Giunchedi:
[operations/puppet@production] thumbor: use jessie-backports as target release for python-thumbor-wikimedia

https://gerrit.wikimedia.org/r/360350