MediaViewer loads most of its scripts at the onload event, after the page has finished loading. This makes sense when the user does not interact with it, as loading potentially unneeded scrips should not delay loading of the page. But when the user is specifically interested in MediaViewer (they visited a #mediaviewer URL, or clicked on an image while the page was loading), it makes no sense to have them wait until the loading of the page and all images on it finishes before starting to load and set up MediaViewer. This is especially problematic on category and gallery pages which can take a long time to fully load.
When there is a MediaViewer hash, or the early click handler catches a click, MediaViewer and the requested image should load immediately. This is non-trivial as it might require setting up MediaViewer before the page has fully loaded (and thus the list of thumbnails is available), although the current way of using thumbnails is ugly and should be refactored anyway (see also T59298). As a partial solution, loading MediaViewer files immediately and setting MediaViewer up on domready would already be an improvement.