Ping. Still seeing these 404s on https://commons.wikimedia.org/ . ...

I can't reproduce this. Wikimedia app servers haven't produced /static/1.27.0-$version/ urls since February 26 (config diff). Since then, we only use /w/ for static urls. (E.g. /w/resources.)

When opening pages on https://commons.wikimedia.org/ I see the following urls requests in the background:

• https://commons.wikimedia.org/w/resources/lib/jquery.ui/themes/smoothness/images/ui-bg_glass_75_e6e6e6_1x400.png?944bb
• https://commons.wikimedia.org/w/resources/lib/jquery.ui/themes/smoothness/images/ui-icons_888888_256x240.png?302ae

The last wmf branch that used static symlinks was 1.27.0-wmf.15. We are currently at 1.27.0-wmf.23.

The (then unused) logic for branch-name static urls was removed in 3f2943e482. The remaining symlinks were removed in 29b8e505b22.

The references for jQuery UI urls are not cached in HTML, but in stylesheets served from load.php.

My guess is that these are perhaps cached client-side in your localStorage, and the module has not changed since February, and we are versioning the unminified styles (and as such, don't see the change in expanded urls when version hashing the for module).

While we waited the usual 30 days before removing the symlinks for old branches, they were still cached client side (which has no expiry).

This is not a new problem and not caused by the migration to w/static.php. In the past when removing those symlinks they will broken cached urls as well. If anything, the migration to static.php solved this problem going forward as we now produce branch-agonistic urls that relate to file content only (and as long as a module doesn't change, its files don't change and remain accessible). Once an image changes, the module version changes, and produce a new url with the old one only staying around for as long as the startup module is cached (~5 minutes).

Ok. I can see that they're gone in a fresh browser install or incognito view. But I've definitely closed the whole browser app since the change, and no amount of force-reload gets rid of these 404s. I guess the links exist in something cached browser-side which a force-reload of the page does not actually reload?

In T134368#2277677, @BBlack wrote:

Ok. I can see that they're gone in a fresh browser install or incognito view. But I've definitely closed the whole browser app since the change, and no amount of force-reload gets rid of these 404s. I guess the links exist in something cached browser-side which a force-reload of the page does not actually reload?

Indeed. In the most optimal scenario these are not HTTP requests, they are gets to a client-side key/value store, which are somewhat comparable to reading a cookie. Reloads do not affect them.

See https://www.mediawiki.org/wiki/ResourceLoader/Features#Store for more information about this feature.

So, how do we get rid of those stored broken URIs? It seems like a lot of clients will be stuck with them for a very long time...

The proper solution is to roll out pure module-content based versioning to file modules (which covers jquery-ui). More specifically, to apply hashing to module content post-minification, rather than before.

This requires careful auditing of the involved hot code paths. Several of these are not fast enough to trigger for all modules in the startup response (which is where version hashed are computed). Enabling this means we'd build (transform+concat) and minify over 1000 modules in https://en.wikipedia.org/w/load.php?debug=true&modules=startup&only=scripts. I believe this is feasible with the right optimisations and object caching in place, but we're not there yet.

The original ResourceLoader design had no content-based versioning (only timestamp-based versioning with the original files as input, no run-time configuration or transformations like CSSMin embedding and url expanding salted into the hash).

Last year I added content-based versioning, but this uncovered various performance issues in the Less compilation stack. So it's only enabled for "simple" modules that generate code (e.g. ResourceLoader LanguageDataModule). The main corpus of file modules don't have this enabled yet.

...

Short-term workaround: Increment $wgResourceLoaderStorageVersion in wmf-config.

Mentioned in SAL [2016-06-08T22:01:32Z] <krinkle@tin> Synchronized wmf-config/CommonSettings.php: Bump wgResourceLoaderStorageVersion (T134368) (duration: 00m 28s)