Narrowing down the problem: I see the actual client IP in the apache httpd logs for my requests. So it seems that the problem is somewhere between what gets passed to mediawiki and how mediawiki does treat such data.

MediaWiki looks at $_SERVER['REMOTE_ADDR'], the relevant code is https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master/includes/WebRequest.php#1240 getRawIP and getIP

Change 747101 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] mediawiki: add the ability to inject php files for debugging

https://gerrit.wikimedia.org/r/747101

Change 747101 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: add the ability to inject php files for debugging

https://gerrit.wikimedia.org/r/747101

I added a debug script that just dumps $_SERVER, and indeed REMOTE_ADDR is 127.0.0.1, while on mwdebug1001 it's set to the IP address of the host, while X_FORWARDED_FOR is always set to the IP of the client:

mwdebug1001:

["REMOTE_ADDR"]=>
string(27) "2620:0:861:103:10:64:32:123"
["HTTP_X_FORWARDED_FOR"]=>
string(49) "<redacted>, 10.20.0.54"

k8s-experimental:

["REMOTE_ADDR"]=>
string(9) "127.0.0.1"
["HTTP_X_FORWARDED_FOR"]=>
string(49) "<redacted>, 10.20.0.54"

in fact, i don't think we should rely on REMOTE_ADDR at all, or if we do, we need to install and configure mod_remoteip. I should add that how mediawiki behaves is *very surprising* at least in the getRawIP function.

Change 747462 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] mediawiki: add ability to inject apache configurations early

https://gerrit.wikimedia.org/r/747462

Change 747462 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: add ability to inject apache configurations early

https://gerrit.wikimedia.org/r/747462

Sadly the story is more complex; in fact, only requests coming from the edge contain X-Client-Ip by default, so we need to inject it into any request at the tls termination layer.

Basically:

• Add the header x-client-ip: <connecting ip> in envoy's request downstream to apache httpd
• Add mod_remoteip with RemoteIPHeader X-Client-Ip to apache's configuration as early as possible
• Also add 127.0.0.1/32 as RemoteIPInternalProxy so that envoy can be trusted with its x-client-ip header.

Change 747838 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] mediawiki: inject x-client-ip from envoy

https://gerrit.wikimedia.org/r/747838

Change 747838 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: inject x-client-ip from envoy

https://gerrit.wikimedia.org/r/747838

With my last changes, I'm now able to correctly see the page, and REMOTE_ADDR is not set to localhost in either of the following situations:

• Request from the edge
• Request from another service which sets x-client-ip
• Request from another service which doesn't set x-client-ip

I'll report in the following table the values for the relevant $_SERVER variables on k8s and on mwdebug1001:

Change 754897 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/docker-images/production-images@master] mediawiki-httpd: add and configure mod_remoteip

https://gerrit.wikimedia.org/r/754897

Change 754897 merged by Giuseppe Lavagetto:

[operations/docker-images/production-images@master] mediawiki-httpd: add and configure mod_remoteip

https://gerrit.wikimedia.org/r/754897