| Etonkovidova | |
| Sep 23 2020, 11:35 PM |
| F32361489: Screen Shot 2020-09-23 at 4.14.08 PM.png | |
| Sep 23 2020, 11:35 PM |
The issue is exactly as T258833: [wmf.1] MediaSearch page 429 error responses on initial search request but for Other tab. I re-checked the searches for Images/Audio/Video/Categories - the issue is rarely (for complex search queries -three words and more) present there.
Steps to Reproduce:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | egardner | T271965 [Epic] Performance optimizations and improvements for Special:MediaSearch | |||
| Resolved | BUG REPORT | matthiasmullie | T263705 [wmf.10] MediaSearch Other tab - 429 error is present |
Assuming this is about sdaw-mediasearch, hence adding that project tag so other people who don't know or don't care about team tags can also find this task
I just did https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseMediaInfo/+/631170, which will make us request more common thumbnail sizes for images.
This would not affect these in the "other" tab (those are a pretty common size already), but might make make the problem small enough that users don't get throttled as agressively anymore.
Let's wait to address this and see whether aforementioned patch fixes this sufficiently already.
This is still valid: many of the document files simply don't have a warm thumbnail in the requested (common) width.
Moving back to ready for dev so we can look into this further.
Let's first see if there's anything we can do on the frontend to limit the amount of thumbnail requests in this tab (throttle requests when scrolling very fast? request smaller sized batches & require manual click to load more? IDK...)
Once there's nothing more we can do there, we'll probably need to look into raising thresholds for generating these thumbs.
We actually moved in the opposite direction with T264805. In an attempt to minimize the amount of pending images a user sees, we're loading them a little before they even enter the viewport.
I think that the threshold for generating thumbnails on the server has been something of a bottleneck for the MediaSearch page from day 1. I would be happy to take a look at how things are set up but I don't really know where to begin; is the limit coming from application code, from configuration, from some underlying tool like ImageMagick, or do we need to allocate more (server/bandwidth/whatever) resources?
I've just tested the steps to reproduce, and even after a lot of scrolling, I didn't run into 429s. Yesterday was the same. Several hundreds of thumbnails (that obviously had to be generated on the fly, given their slow response time) loaded just fine.
I don't know whether it's still much of an issue.
However, I think we can still improve things a little bit:
1/ We can use one of the pre-generated sizes (https://www.mediawiki.org/wiki/Requests_for_comment/Standardized_thumbnails_sizes#Currently_used_numbers)
They'll be larger and take longer to transfer, but they'll be ready to go. This should eliminate rate limiting (because they don't need to be generated) and speed up things overall.
(only doing this for "other", because some of those file types are quite expensive & slow to generate images, and we don't really have issues with the other tabs)
2/ by debouncing the intersection callbacks for fast scrolling: we don't need to load all those images - only the ones that are still on-screen once the scrolling slows down/stops.
This would reduce the amount of requests (= load on the server) when scrolling really fast, and improve speed (= fewer queued requests).
(that's only really an edge-case tweak, though - I expect most people won't scroll through results at a pace they can't see the thumbnails anyway...)
Change 663809 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/WikibaseMediaInfo@master] Debounce thumbnail loading
Change 663810 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/WikibaseMediaInfo@master] For other tab, request images at a pregenerated size
Change 663809 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Debounce thumbnail loading
Change 663810 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] For other tab, request images at a pregenerated size
Checked in commons wmf.32 - no 429 errors, the thumbnails are promptly displayed when scrolling.