Page MenuHomePhabricator

Limit the number or expensive thumbnails that can be processed by the scalers in parallel
Closed, ResolvedPublic

Description

Certain files are very expensive to thumbnail (e.g. large TIFF files); receiving several such requests can bring the scalers down. This was a recurring issue with certain GWToolset uploads. To prevent such issues, there should be a way to mark files as expensive and throttle the scaling of such files.

This is tracked by the Multimedia team as https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/623 . The current plan is:

  • have the media handler decide whether scaling the file is expensive, and pass that flag on
  • for expensive files, the pool counter should bucket by some hash of the file name instead of the file name itself, so that there are only <range of hash function> slots available for expensive thumbnailing operations
  • the hash has to be deterministic to avoid stampeding, and the range has to be configurable

This is not an ideal solution, since hash collisions will make it hard to estimate the exact effect of the throttling - some of the jobs will be blocked even when there are less jobs than available slots. It will make it easy to limit the maximum possible load though, which is still an improvement over the current situation.

(The alternative would be to just have a big "expensive thumbnails" bucket without any kind of hashing - given that expensive thumbnail requests are slow and likely to be stampeded, that would be even worse.)


Version: 1.24rc
Severity: normal

Details

Reference
bz65691

Event Timeline

bzimport raised the priority of this task from to Normal.
bzimport set Reference to bz65691.
Tgr created this task.May 23 2014, 6:43 PM

Change 135702 had a related patch set uploaded by Gergő Tisza:
Add global limit to PoolCounter

https://gerrit.wikimedia.org/r/135702

Change 135703 had a related patch set uploaded by Gergő Tisza:
Use separate PoolCounter config for expensive thumbnails

https://gerrit.wikimedia.org/r/135703

Change 135976 had a related patch set uploaded by Gergő Tisza:
Set expensive flag for large TIFF files

https://gerrit.wikimedia.org/r/135976

Change 135976 merged by jenkins-bot:
Set expensive flag for large TIFF files

https://gerrit.wikimedia.org/r/135976

Change 135701 merged by jenkins-bot:
Allow media handlers to mark files as expensive

https://gerrit.wikimedia.org/r/135701

Change 135702 merged by jenkins-bot:
Add global limit to PoolCounter

https://gerrit.wikimedia.org/r/135702

Change 135704 merged by jenkins-bot:
Limit the number of expensive thumbnails processed at the same time

https://gerrit.wikimedia.org/r/135704

Change 135703 merged by jenkins-bot:
Use separate PoolCounter config for expensive thumbnails

https://gerrit.wikimedia.org/r/135703

Gilles moved this task from Untriaged to Done on the Multimedia board.
Gilles raised the priority of this task from Normal to Unbreak Now!.
Gilles lowered the priority of this task from Unbreak Now! to Normal.Dec 4 2014, 11:21 AM
bd808 moved this task from Tag to Archive on the Performance board.Mar 11 2015, 7:29 PM