Service-based thumbnailing re-architecture in production with Thumbor
Closed, ResolvedPublic

Description

The goal of this task is to remove our dependency on MediaWiki for generating thumbnails in Wikimedia production.

This task does not include moving MediaWiki to a full sha1-based URL scheme for thumbnails and originals. That can be achieved separately (T66214, T19577).

It does not include getting rid of Swift for thumbnail storage, which will be tackled in a follow-up project.

It does not include getting rid of Mediawiki parsing originals metadata, as this is not a task Thumbor was designed to do. Another service would be needed to remove that functionality from Mediawiki.

Strategy for deployment in Wikimedia production:

  • Thumbor is stateless and acts as drop-in for current MediaWiki PHP image scalers
  • Thumbor will remain behind Varnish
  • Thumbor will be integrated at the Swift level (rewrite.py)

Related Objects

StatusAssignedTask
ResolvedGilles
ResolvedGilles
Resolveddpatrick
InvalidGilles
ResolvedGilles
DuplicateGilles
DuplicateGilles
DuplicateGilles
ResolvedGilles
Resolvedfgiunchedi
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
InvalidGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
DeclinedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
Resolvedfgiunchedi
ResolvedCmjohnson
StalledGilles
ResolvedGilles
Openfgiunchedi
Resolvedfgiunchedi
There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 309029 had a related patch set uploaded (by Filippo Giunchedi):
swift: disable thumbor shadow traffic

https://gerrit.wikimedia.org/r/309029

Change 309029 merged by Filippo Giunchedi:
swift: disable thumbor shadow traffic

https://gerrit.wikimedia.org/r/309029

Gilles moved this task from Backlog to Doing on the Thumbor board.Oct 13 2016, 5:54 AM
Gilles moved this task from Doing to Backlog on the Thumbor board.Nov 24 2016, 4:00 PM
Gilles moved this task from Backlog to Doing on the Performance-Team board.Dec 6 2016, 12:29 AM
Gilles moved this task from Backlog to Doing on the Thumbor board.Dec 6 2016, 2:25 PM
Gilles moved this task from Doing to Next-up on the Performance-Team board.Dec 22 2016, 10:42 AM

Change 358599 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: use thumbor.svc for codfw too

https://gerrit.wikimedia.org/r/358599

Change 358599 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: use thumbor.svc for codfw too

https://gerrit.wikimedia.org/r/358599

ori added a subscriber: ori.Jun 13 2017, 6:26 PM

I know you are not 100% done, but still: congratulations, Sirs! It's really impressive to see a long arc of design and implementation work and operational planning come to fruition. Kudos!

Gilles moved this task from Next-up to Doing on the Performance-Team board.Jun 14 2017, 7:36 PM
Gilles updated the task description. (Show Details)Jun 19 2017, 1:39 PM

Change 360307 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: have thumbor.svc alert critical

https://gerrit.wikimedia.org/r/360307

Change 360307 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: have thumbor.svc alert critical

https://gerrit.wikimedia.org/r/360307

Change 360309 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Increase Swift timeout for Thumbor

https://gerrit.wikimedia.org/r/360309

Change 360309 merged by Filippo Giunchedi:
[operations/puppet@production] Increase Swift timeout for Thumbor

https://gerrit.wikimedia.org/r/360309

Change 360350 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thumbor: use jessie-backports as target release for python-thumbor-wikimedia

https://gerrit.wikimedia.org/r/360350

Change 360350 merged by Filippo Giunchedi:
[operations/puppet@production] thumbor: use jessie-backports as target release for python-thumbor-wikimedia

https://gerrit.wikimedia.org/r/360350

Change 367373 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thumbor: bump MemoryLimit to 15%

https://gerrit.wikimedia.org/r/367373

Change 367373 merged by Filippo Giunchedi:
[operations/puppet@production] thumbor: bump MemoryLimit to 15%

https://gerrit.wikimedia.org/r/367373

Prtksxna removed a subscriber: Prtksxna.Jul 26 2017, 9:59 AM
Gilles closed this task as Resolved.Aug 7 2017, 8:16 PM

Thumbor has been serving all thumbnail traffic for over a month now. I think this task can be closed as the project is completed and successful. Subsequent bugfixes and improvements are tracked with the Thumbor project.

Change 377264 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thumbor: use memorysize_mb fact for unit MemoryLimit

https://gerrit.wikimedia.org/r/377264

Change 377264 merged by Filippo Giunchedi:
[operations/puppet@production] thumbor: use memorysize_mb fact for unit MemoryLimit

https://gerrit.wikimedia.org/r/377264