Legacy thumbor instances were decommissioned this morning - I didn't realise beta was using it. I'll try to understand how this was configured and redirect to the current prod instances.

deployment-prep's swift instance has an outdated and no longer extant imagescaler host configured for thumbor https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/cloud/eqiad1/deployment-prep/common.yaml#537

I'm not sure how this worked previously

There has definitely been an increase in the last few days based on some rather unscientific log grepping:

hnowlan@deployment-ms-fe04:~$ for i in /var/log/swift/proxy-access.log*gz; do echo $i; zcat $i | head -n1 | awk '{ print $1 $2 }';  zgrep wikipedia-commons-local-thumb $i | grep 503 | wc -l; done
/var/log/swift/proxy-access.log.1.gz
Aug22
181
/var/log/swift/proxy-access.log.2.gz
Aug21
108
/var/log/swift/proxy-access.log.3.gz
Aug20
10
/var/log/swift/proxy-access.log.4.gz
Aug19
10

Decommissioning was done on the 21st.

however, this swift instance is using the misconfigured thumbor mentioned above.

Adding a hack to route requests for the old imagescaler hostname to a netcat session shows requests incoming - thumbnailing has been broken in beta since T294148#8535241. A replacement Thumbor instance is needed

I want to test an issue but can't do it properly because beta cluster has no thumbnailer. I could perhaps test locally, but I'd rather test on beta cluster.

Edit: Well I managed to test it anyway, but really, not having a Thumbor instance on beta cluster makes testing difficult at times.

Recently some files were uploaded to betacommons. It had been quiet for a while.

Without thumbnails it's a considerable pain to find the copyright violations and spam among them. On the mobile site of betacommons there is an overlap of the text "Beta Wikimedia Commons" and the searchbox, but whether this is the result of a bug or a logo image failing to load I can't say.

Adding some recent uploaders from betacommons who might have been confused by the lack of thumbnails.

In T344605#9305651, @AlexisJazz wrote:

Recently some files were uploaded to betacommons. It had been quiet for a while.

It had been quiet for a while because of T340908: Unable to upload files on Beta Commons.

We could just enable the core thumbnailer, that shouldn't take much effort.

In T344605#9306930, @Tgr wrote:

In T344605#9305651, @AlexisJazz wrote:

Recently some files were uploaded to betacommons. It had been quiet for a while.

It had been quiet for a while because of T340908: Unable to upload files on Beta Commons.

That task was filed in July 2023 but the last upload before that was in December 2022. But maybe it was broken the whole time.

We could just enable the core thumbnailer, that shouldn't take much effort.

Is the core thumbnailer still available on WMF wikis? I'm asking because T147995#9273775. I set up a MediaWiki instance (apt-get) and actually have no idea if it uses thumbor.
https://wikitech.wikimedia.org/wiki/Thumbor says "As of February 2018, all thumbnail traffic for private wikis is served by Thumbor." I'm unsure what "private wiki" means in this context.

Anyhow, having any kind of thumbnail will make testing and deleting unwanted files on beta cluster easier. Though it should also be noted that sometimes issues need to be tested that specially relate to thumbnailing, so I'd still hope thumbor will come back at some point. But if it's possible to enable the core thumbnailer that'll be a good stopgap.

In T344605#9307142, @AlexisJazz wrote:

That task was filed in July 2023 but the last upload before that was in December 2022. But maybe it was broken the whole time.

It was.

Is the core thumbnailer still available on WMF wikis? I'm asking because T147995#9273775.

Depends on your definition of available. For the purposes of that task, I think the answer is yes, at least on some wikis.

I set up a MediaWiki instance (apt-get) and actually have no idea if it uses thumbor.

It doesn't, Thumbor is not part of MediaWiki and needs to be set up separately.

https://wikitech.wikimedia.org/wiki/Thumbor says "As of February 2018, all thumbnail traffic for private wikis is served by Thumbor." I'm unsure what "private wiki" means in this context.

Wikis where the content (including image thumbnails) is only visible to logged-in users.

Change 971623 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] Do not try to use Thumbor on beta

https://gerrit.wikimedia.org/r/971623

Change 971623 merged by jenkins-bot:

[operations/mediawiki-config@master] Do not try to use Thumbor on beta

https://gerrit.wikimedia.org/r/971623

Well that didn't work. The error actually comes from a Swift plugin hosted in Puppet, definitely not touching that. Short of disabling nice URLs for thumbnails, I don't think much can be done here without a working beta Thumbor.

(Core thumbnailing also fails with Unknown option --no-external-files, but there's no point in debugging that.)

Change 972263 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] Revert "Do not try to use Thumbor on beta"

https://gerrit.wikimedia.org/r/972263

Change 972263 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Do not try to use Thumbor on beta"

https://gerrit.wikimedia.org/r/972263

Mentioned in SAL (#wikimedia-operations) [2023-12-05T21:02:44Z] <jforrester@deploy2002> Started scap: Backport for [[gerrit:972263|Revert "Do not try to use Thumbor on beta" (T344605)]], [[gerrit:980009|nlwikivoyage: Drop Listings extension (T352696)]], [[gerrit:980047|Drop Listings extension from Wikivoyages where unused (T352719)]]

Mentioned in SAL (#wikimedia-operations) [2023-12-05T21:04:11Z] <jforrester@deploy2002> tgr and jforrester: Backport for [[gerrit:972263|Revert "Do not try to use Thumbor on beta" (T344605)]], [[gerrit:980009|nlwikivoyage: Drop Listings extension (T352696)]], [[gerrit:980047|Drop Listings extension from Wikivoyages where unused (T352719)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-12-05T21:11:29Z] <jforrester@deploy2002> Finished scap: Backport for [[gerrit:972263|Revert "Do not try to use Thumbor on beta" (T344605)]], [[gerrit:980009|nlwikivoyage: Drop Listings extension (T352696)]], [[gerrit:980047|Drop Listings extension from Wikivoyages where unused (T352719)]] (duration: 08m 45s)