I can't figure out where this message comes from. Do you know?

Apache, I'd assume.

https://auth.wikimedia.beta.wmflabs.org/enwiki/wiki/Special:UserLogin and https://auth.wikimedia.beta.wmflabs.org/enwiki/w/index.php load fine. I assume this is about when you manually view the domain root? (I've updated the task based on that assumption).

This error message comes from wmf-config, because it isn't a valid /:dbname/* url.

https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/4da5cecfa43f633dc3cb935d631d840be0f37a46/multiversion/MWMultiVersion.php#409

Unlike the other self::error() conditions, though, this one is trivially reachable and so may be deserving a custom response, or redirect, or separate docroot with a static index.html file.

The "File not found." is not coming from anywhere in operations/mediawiki-config though. And it doesn't look like an Apache error either, unless we have customized them really weirdly.

It is the same kind of "File not found." served at these URLs:

• https://en.wikipedia.org/foo.php (with .php)
• https://en.wikipedia.beta.wmflabs.org/foo.php

It comes from Apache (or php-fpm), for URLs that match a rewrite rule that PHP is meant to serve.

It does not happen for https://en.wikipedia.org/foo (without .php) in production, because that isn't directed to PHP.

I'm guessing that our auth-specific ^/([^/]*)/(.*)$ rewrite rule (Puppet source) might be matching this? It seems like it wouldn't because it contains two non-optional slashes. On the other hand, both of the match groups accept zero or more, so // and //foo would match which is unexpected. It might be that Apache is being "helpful" in tolerating additional slashes.

Alternatively, it might be that the vhost is lacking an include for the default config that declares ErrorDocument 404 /w/404.php. This semes unlikely, since afaik that is unconditionally included and not restricted by VirtualHost.

I don't think the error message is a problem per se - we just need to understand exactly what's going on so we can be confident that there is no way to access files that shouldn't be accessed (e.g. PrivateSettings). That's very likely the case since those are protected by Apache permissions, and especially since as @Krinkle pointed out the presence of this error is not very different from normal wikis. But an accurate description of what's happening would be nice.

After reading the config files, old commits and Apache docs for most of today, I think I figured out why the weird "File not found." error happens:

• /foo does not exist, so it's handled by ErrorDocument 404 /w/404.php
• ErrorDocument "internally redirects" to /w/404.php
• The ^/([^/]*)/(.*)$ rewrite rule rewrites /w/404.php to /404.php
• <FilesMatch "\.php$"> matches, sending a request for /404.php to php-fpm
• php-fpm backend returns that ugly error message

I found this by blaming this unusual line referring to /w/404.php: https://gerrit.wikimedia.org/g/operations/puppet/+/29611a362ba427b1843b735eb6cf5aa373679c2f/modules/mediawiki/templates/apache/sites/wwwportals.conf.erb#67 through which I found T158782, which was in fact the same problem on the portals site. It's from 2017, I apparently filed that task myself, and Timo was involved in fixing it.

We should be able to fix this next week in a similar manner. I'd also like to avoid those ugly 404 errors for normal requests ending with .php, maybe we can do that too. And also add a nice landing page.

Change #1100530 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/puppet@production] MediaWiki: Ensure nice 404 instead of php-fpm 404 on auth domain

https://gerrit.wikimedia.org/r/1100530

Change #1100532 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/puppet@production] MediaWiki: Redirect auth domain root to wikimedia.org portal

https://gerrit.wikimedia.org/r/1100532

Change #1100534 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/puppet@production] MediaWiki: Only proxy existing .php files, otherwise return nice 404

https://gerrit.wikimedia.org/r/1100534

I've cherry-picked these patches to the beta cluster (following these instructions) and they all seem to work correctly. You can test right now.

I scheduled them for the Puppet request window tomorrow to be officially blessed: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20241205T1700 (that seems like a lot of patches, so we might not complete them all).

The patches were not blessed today because the engineer handling the Puppet request window was not comfortable being the sole reviewer on them. Someone more familiar with the beta cluster has to review them. Not sure who that would be.

Reviewed by @Tgr and @Krinkle, and scheduled for tomorrow again.

Change #1100530 merged by RLazarus:

[operations/puppet@production] MediaWiki: Ensure nice 404 instead of php-fpm 404 on auth domain

https://gerrit.wikimedia.org/r/1100530

Change #1100532 merged by RLazarus:

[operations/puppet@production] MediaWiki: Redirect auth domain root to wikimedia.org portal

https://gerrit.wikimedia.org/r/1100532