Page MenuHomePhabricator

LocalFile::prerenderThumbnail should have a page limit
Closed, ResolvedPublic

Description

Right now when a large pdf / multipage tiff file is uploaded, LocalFile::prerenderThumbnail generates a large number of jobs, equal to the number of pages multiplied by the size of
wgUploadThumbnailRenderMap.

This means that in production, where that array has size 4, a 1000 pages pdf generates 4000 prerender jobs, all of which will have to extract a single page from this huge pdf, a quite expensive operation that regularly overloads our thumbnailing infrastructure.

I also don't see a huge advantage in generating so many thumbnails. I would propose to modify the mechanism so that we pregenerate the thumbnails only for the first N pages of any document exceeding M pages, with N and M being configuration parameters.

Event Timeline

Change 798887 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@master] LocalFile::prerenderThumbnails: Limit the number of thumbnail jobs triggered

https://gerrit.wikimedia.org/r/798887

Change 798887 merged by jenkins-bot:

[mediawiki/core@master] LocalFile::prerenderThumbnails: Limit the number of thumbnail jobs triggered

https://gerrit.wikimedia.org/r/798887

Joe triaged this task as High priority.May 25 2022, 9:16 AM

I would say this probably should be backported, given this is a fix for an ongoing production issue.

Change 799389 had a related patch set uploaded (by Krinkle; author: Jforrester):

[mediawiki/core@REL1_38] LocalFile::prerenderThumbnails: Limit the number of thumbnail jobs triggered

https://gerrit.wikimedia.org/r/799389

Change 799390 had a related patch set uploaded (by Krinkle; author: Jforrester):

[mediawiki/core@REL1_37] LocalFile::prerenderThumbnails: Limit the number of thumbnail jobs triggered

https://gerrit.wikimedia.org/r/799390

Change 799391 had a related patch set uploaded (by Krinkle; author: Jforrester):

[mediawiki/core@REL1_35] LocalFile::prerenderThumbnails: Limit the number of thumbnail jobs triggered

https://gerrit.wikimedia.org/r/799391

This issue was introduced in MW 1.37 as part of change 698367 (ref T284416) which added support for pre-rendering the thumbnail for multi-page files, which was previously not functional (not even the first page, presumably) .

Change 799389 merged by jenkins-bot:

[mediawiki/core@REL1_38] LocalFile::prerenderThumbnails: Limit the number of thumbnail jobs triggered

https://gerrit.wikimedia.org/r/799389

Change 799390 merged by jenkins-bot:

[mediawiki/core@REL1_37] LocalFile::prerenderThumbnails: Limit the number of thumbnail jobs triggered

https://gerrit.wikimedia.org/r/799390