Page MenuHomePhabricator

Purging in Thumbor
Closed, ResolvedPublic

Description

Preferably in the form of a plugin, otherwise as a new feature/pull request on their project.

Event Timeline

• Gilles claimed this task.
• Gilles raised the priority of this task from to Medium.
• Gilles updated the task description. (Show Details)
• Gilles added subscribers: ori, jeremyb, • Aklapper and 9 others.

At first, we could avoid the purging issue in thumbor by simply storing things very shortly in it. I.e. the originals would stay for a minute, to cater for upload thumbnail prerendering bursts. Thumbnails wouldn't be stored at all. Purging requests should delete the original from swift immediately and schedule a varnish purge 1+ minutes later. This way by the time a new request would come to varnish later than a minute after the purge, it would be funneled to thumbor, which in turn would try to re-download the original and fail.

Then, if we want to use thumbor as a hot swap/safety net in case of disaster scenarios where varnish fails partially or fully, we need to store thumbnails on thumbor machines for a period of time. This will require writing a new FS-based storage engine for those, as the current one's directory tree scheme is completely impractical for content hash-based deletions of multiple thumbnails at once:

vagrant@mediawiki-vagrant:/tmp/thumbor/result_storage/v2/un/sa/unsafe/110x147/filters:sharpen(0.04,0,false)/127.0.0.1:8080/images/9/9a$

A new entry point for deletion commands would have to be added to thumbor.

• Gilles renamed this task from Figure out purging strategy for thumbor to Implement purging in Thumbor.Sep 7 2015, 7:53 PM
• Gilles updated the task description. (Show Details)
• Gilles set Security to None.
• Gilles renamed this task from Implement purging in Thumbor to Purging in Thumbor.Sep 7 2015, 8:10 PM

Ideally purging would work for the original (stored in memcached) and all thumbnail sizes at once (stored on the FS).

Change 251316 had a related patch set uploaded (by Gilles):
Add ability to purge originals cached by thumbor

https://gerrit.wikimedia.org/r/251316

Change 251413 had a related patch set uploaded (by Gilles):
Purger for thumbor

https://gerrit.wikimedia.org/r/251413

Change 251432 had a related patch set uploaded (by Gilles):
Add libcurl-dev to Python Jenkins

https://gerrit.wikimedia.org/r/251432

Change 251432 merged by Ori.livneh:
Add libcurl-dev to Python Jenkins

https://gerrit.wikimedia.org/r/251432

Change 251479 had a related patch set uploaded (by Hashar):
contint: explicitly set ssl/tls libcurl dev package

https://gerrit.wikimedia.org/r/251479

Change 251479 merged by Muehlenhoff:
contint: explicitly set ssl/tls libcurl dev package

https://gerrit.wikimedia.org/r/251479

Change 251413 merged by jenkins-bot:
Purger for thumbor

https://gerrit.wikimedia.org/r/251413

Change 252578 had a related patch set uploaded (by Gilles):
Purge thumbnails

https://gerrit.wikimedia.org/r/252578

Change 251316 merged by jenkins-bot:
Add ability to purge originals cached by thumbor

https://gerrit.wikimedia.org/r/251316