Page MenuHomePhabricator

Service-based thumbnailing re-architecture in production with Thumbor
Closed, ResolvedPublic

Description

The goal of this task is to remove our dependency on MediaWiki for generating thumbnails in Wikimedia production.

This task does not include moving MediaWiki to a full sha1-based URL scheme for thumbnails and originals. That can be achieved separately (T66214, T19577).

It does not include getting rid of Swift for thumbnail storage, which will be tackled in a follow-up project.

It does not include getting rid of Mediawiki parsing originals metadata, as this is not a task Thumbor was designed to do. Another service would be needed to remove that functionality from Mediawiki.

Strategy for deployment in Wikimedia production:

  • Thumbor is stateless and acts as drop-in for current MediaWiki PHP image scalers
  • Thumbor will remain behind Varnish
  • Thumbor will be integrated at the Swift level (rewrite.py)

Details

Related Gerrit Patches:
operations/puppet : productionthumbor: use memorysize_mb fact for unit MemoryLimit
operations/puppet : productionthumbor: bump MemoryLimit to 15%
operations/puppet : productionthumbor: use jessie-backports as target release for python-thumbor-wikimedia
operations/puppet : productionIncrease Swift timeout for Thumbor
operations/puppet : productionhieradata: have thumbor.svc alert critical
operations/puppet : productionhieradata: use thumbor.svc for codfw too
operations/puppet : productionswift: disable thumbor shadow traffic

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusAssignedTask
ResolvedGilles
ResolvedGilles
Resolved dpatrick
InvalidGilles
ResolvedGilles
DuplicateGilles
DuplicateGilles
DuplicateGilles
ResolvedGilles
Resolvedfgiunchedi
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
InvalidGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
DeclinedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
Resolvedfgiunchedi
ResolvedCmjohnson
DeclinedGilles
ResolvedGilles
Openfgiunchedi
Resolvedfgiunchedi

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 309029 had a related patch set uploaded (by Filippo Giunchedi):
swift: disable thumbor shadow traffic

https://gerrit.wikimedia.org/r/309029

Change 309029 merged by Filippo Giunchedi:
swift: disable thumbor shadow traffic

https://gerrit.wikimedia.org/r/309029

Gilles moved this task from Backlog to Doing on the Thumbor board.Oct 13 2016, 5:54 AM
Gilles moved this task from Doing to Backlog on the Thumbor board.Nov 24 2016, 4:00 PM
Gilles moved this task from Backlog to Doing on the Thumbor board.Dec 6 2016, 2:25 PM

Change 358599 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: use thumbor.svc for codfw too

https://gerrit.wikimedia.org/r/358599

Change 358599 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: use thumbor.svc for codfw too

https://gerrit.wikimedia.org/r/358599

ori added a subscriber: ori.Jun 13 2017, 6:26 PM

I know you are not 100% done, but still: congratulations, Sirs! It's really impressive to see a long arc of design and implementation work and operational planning come to fruition. Kudos!

Gilles updated the task description. (Show Details)Jun 19 2017, 1:39 PM

Change 360307 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: have thumbor.svc alert critical

https://gerrit.wikimedia.org/r/360307

Change 360307 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: have thumbor.svc alert critical

https://gerrit.wikimedia.org/r/360307

Change 360309 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Increase Swift timeout for Thumbor

https://gerrit.wikimedia.org/r/360309

Change 360309 merged by Filippo Giunchedi:
[operations/puppet@production] Increase Swift timeout for Thumbor

https://gerrit.wikimedia.org/r/360309

Change 360350 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thumbor: use jessie-backports as target release for python-thumbor-wikimedia

https://gerrit.wikimedia.org/r/360350

Change 360350 merged by Filippo Giunchedi:
[operations/puppet@production] thumbor: use jessie-backports as target release for python-thumbor-wikimedia

https://gerrit.wikimedia.org/r/360350

Change 367373 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thumbor: bump MemoryLimit to 15%

https://gerrit.wikimedia.org/r/367373

Change 367373 merged by Filippo Giunchedi:
[operations/puppet@production] thumbor: bump MemoryLimit to 15%

https://gerrit.wikimedia.org/r/367373

Prtksxna removed a subscriber: Prtksxna.Jul 26 2017, 9:59 AM
Gilles closed this task as Resolved.Aug 7 2017, 8:16 PM

Thumbor has been serving all thumbnail traffic for over a month now. I think this task can be closed as the project is completed and successful. Subsequent bugfixes and improvements are tracked with the Thumbor project.

Change 377264 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thumbor: use memorysize_mb fact for unit MemoryLimit

https://gerrit.wikimedia.org/r/377264

Change 377264 merged by Filippo Giunchedi:
[operations/puppet@production] thumbor: use memorysize_mb fact for unit MemoryLimit

https://gerrit.wikimedia.org/r/377264