Whoops, I didn't mean to give it such an alarming title. I should proofread better, sorry about that.

Images on the beta cluster haven't been happy for a while :/ not sure if the above reported issue is a symptom of some/all:

• T344605: deployment-prep needs a Thumbor instance
• T340908: Unable to upload files on Beta Commons
• T359768: upload.wikimedia.beta.wmflabs.org: cannot find server

or a fun new exciting bug :)

After enabling ERROR logging for all channels on the beta cluster (T228838#10137684), a new error appeared that seems relevant:
https://beta-logs.wmcloud.org/goto/80947fcce4f9e3452442d057205dc31b

GET http://deployment-imagescaler03.deployment-prep.eqiad.wmflabs:8800/wikipedia/commons/thumb/temp/... HTTP/1.1 - NULL cURL error 6: Could not resolve host: deployment-imagescaler03.deployment-prep.eqiad.wmflabs (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://deployment-imagescaler03.deployment-prep.eqiad.wmflabs:8800/wikipedia/commons/thumb/temp/...

deployment-imagescaler03 has been replaced with deployment-imagescaler04, but thumbor things still seem to be unhappy. https://meta.wikimedia.beta.wmflabs.org/wiki/File:Deployment-mediawiki14_load_2025-07-02.png is a new test upload I did today.

If nothing else it looks like the deployed version of thumbor is very stale. Prod looks to have 2025-06-03-122529-production in the helmfile config.

docker::gvisor::ensure: absent
profile::docker::runner::service_defs:
  thumbor:
    bind_mounts:
      /etc/thumbor.d: /etc/thumbor.d
    config: {}
    host_network: true
    image_name: operations-software-thumbor-plugins
    namespace: wikimedia
    port: 8080
    version: 2023-10-06-153556-production

I added a default gadget on betacommons, https://commons.wikimedia.beta.wmcloud.org/wiki/MediaWiki:Gadget-LoadFull.js that adds links to the sidebar to load full-size versions of images just once, and to enable this by default.

When looking at a bunch of thumbnails of really big files this won't be funny at all, but at least it makes betacommons somewhat more usable.