Page MenuHomePhabricator
Create Task
Maniphest T288848

Make HTTP calls work within mediawiki on kubernetes
Closed, ResolvedPublic
Actions

Description

@dancy was looking into why /favicon.ico requests aren't working on k8s. They are meant to be rewritten to /w/favicon.php, and indeed are, but there the trail goes cold. The http request that the MW PHP code makes from there is failing.

We don't yet have Logstash telemetry on errors here, but I suspected this might mean HTTP calls between wikis aren't working more generally, and indeed that appears to be the case.

The most prominent example, afaik, of HTTP calls within MW is File description pages which make API calls to Commons. For example:

https://test.wikipedia.org/wiki/File:Example.png

When viewed over XWD/k8s, the file description is missing. This is potentially a cache poisoining issue as well since file descriptions are kept in memcached, and with a fairly higih TTL.

Details

SubjectRepoBranchLines +/-
Enable $wgLocalHTTPProxy on Kubernetes, regardless of groupoperations/mediawiki-configmaster+1 -1
Support $wgLocalHTTPProxy in MultiHttpClientmediawiki/coremaster+141 -13
http: Don't set X-Forwarded-Proto when using a reverse proxymediawiki/coremaster+0 -4
Enable $wgLocalHTTPProxy on group0 wikisoperations/mediawiki-configmaster+1 -0
Add framework for setting $wgLocalHTTPProxyoperations/mediawiki-configmaster+12 -0
Allow using a reverse proxy for local HTTP requestsmediawiki/corewmf/1.38.0-wmf.4+57 -4
Allow using a reverse proxy for local HTTP requestsmediawiki/coremaster+57 -4
Make the apple dictionary bridge work in kubernetesoperations/mediawiki-configmaster+13 -2
services_proxy: Add mwapi envoyproxy for MediaWiki-internal requestsoperations/puppetproduction+8 -0
http: Add $wgLocalHTTPProxy to set a proxy for local requestsmediawiki/coremaster+25 -9
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT255792 Quibble runs core:unit tests twice!
 OpenNoneT328919 Upgrade to PHPUnit 10
 OpenNoneT338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+
 OpenNoneT328921 Drop PHP 7.4 support from MediaWiki
 StalledNoneT334726 Use return type `never` in Wikibase
 OpenNoneT328922 Drop PHP 8.0 support from MediaWiki
 StalledNoneT319055 Upgrade to psr/container 2.x
 StalledKrinkleT319432 Migrate WMF production from PHP 7.4 to PHP 8.1
 OpenNoneT291916 Tracking task for Bullseye migrations in production
 StalledNoneT356293 Migrate MW appservers' base images to bullseye
 OpenNoneT290536 Serve production traffic via Kubernetes
 ResolvedJoeT283056 Create a mwdebug deployment for mediawiki on kubernetes
 ResolvedJoeT284417 Add the puppet CA to the MediaWiki deployment
 ResolvedNoneT284418 Add conditional to mediawiki-config for stuff running on kubernetes
 ResolvedjijikiT284420 Add all redis and memcached backends to mw on k8s automatically
 ResolvedjijikiT284421 Enable TLS termination on the mwdebug deployment. fix the service definition in the chart
 ResolveddancyT284581 Ownership of the /tmp/mw-cache directories should be www-data in the mediawiki-multiversion image
 ResolveddancyT284582 The restricted/mediawiki-multiversion image should include the production version of private/PrivateSettings.php
 ResolvedJoeT285298 Make all httpbb tests pass on the mwdebug deployment.
 ResolvedJoeT285309 Install wiki-specific php extensions in the mediawiki production image
 ResolvedjeenaT285325 Check out www-portals repo in the mediawiki-webserver and in the mediawiki-multiversion images
 ResolvedJoeT285232 The restricted/mediawiki-webserver image should include skins and resources
 ResolvedJoeT286491 Access mwdebug kubernetes deployment via the 'X-Wikimedia-Debug' header
 ResolveddancyT287512 GitInfo is missing from mwdebug-kubernetes deployment
 ResolvedJoeT287570 Ensure the code is deployed to mediawiki on k8s when it is deployed to production
 Resolved• dpifkeT288164 Ensure WikimediaDebug "log" and "profile" features work with k8s-mwdebug
 ResolveddancyT287495 Create a variant of mediawiki-multiversion which installs php-tideways-xhprof
 ResolvedJoeT288851 Make logging work for mediawiki in k8s
 ResolvedClement_GoubertT326794 Ingest php-slowlog in logstash
 ResolvedcolewhiteT328318 Index orchestrator object fields from ECS 1.11.0 in OpenSearch
 ResolvedjijikiT290485 mediawiki-debug image does not produce profiling info
Resolved• dpifkeT288165 Create separate ArcLamp pipeline for k8s-mwdebug
 ResolvedLegoktmT288848 Make HTTP calls work within mediawiki on kubernetes

Event Timeline

Krinkle created this task.Aug 13 2021, 6:16 PM
dancy mentioned this in T285232: The restricted/mediawiki-webserver image should include skins and resources.Aug 13 2021, 6:28 PM
• dpifke subscribed.Aug 13 2021, 7:14 PM
JMeybohm added subscribers: Legoktm, JMeybohm.Aug 17 2021, 9:24 AM

I'd assume that MW makes HTTP calls to the public endpoints of MW. Those will be blocked in k8s as we generally prohibit egress traffic. I'm not sure this is the right solution here, but all other services talking to mw-api do so via a service-proxy listener (see: https://wikitech.wikimedia.org/wiki/Envoy#Example_(calling_mw-api) ). /cc @Legoktm

jijiki moved this task from Backlog to In Progress on the MW-on-K8s board.Aug 17 2021, 10:06 AM
Legoktm added a comment.Aug 18 2021, 4:29 PM
In T288848#7287882, @JMeybohm wrote:

I'd assume that MW makes HTTP calls to the public endpoints of MW. Those will be blocked in k8s as we generally prohibit egress traffic. I'm not sure this is the right solution here, but all other services talking to mw-api do so via a service-proxy listener (see: https://wikitech.wikimedia.org/wiki/Envoy#Example_(calling_mw-api) ). /cc @Legoktm

Yeah, MW mostly sends requests to the public endpoints because it's convenient for developers, but we really need to stop that, for all the reasons we introduced envoy (TLS overhead, unnecessarily hitting ATS/varnish, etc.)

https://codesearch.wmcloud.org/operations/?q=%2Fw%2F&i=nope&files=&excludeFiles=&repos=Wikimedia%20MediaWiki%20config is a naive codesearch that identifies most of these places (note that some of those URLs are used client-side and OK).

Each extension or core code will need to support configuring and sending a Host header for the domain we want to hit. I'll file subtasks for the cases I see.

TK-999 subscribed.Aug 18 2021, 7:39 PM

For the record, to resolve the same issue during our effort to upgrade Fandom's MW-on-k8s deployment, we ended up creating an HttpRequestFactory service override to dynamically use the current k8s service as proxy for request URLs known to belong to wikis on our platform. Some "automatic" solution like that could help avoid the whack-a-mole of updating every extension and core call site that makes internal HTTP requests. The logic for determining whether an URL should be automatically proxied or not would probably be similar to the existing core MWHttpRequest::isLocalURL method.

Legoktm added a comment.Aug 19 2021, 12:47 AM
In T288848#7292923, @TK-999 wrote:

For the record, to resolve the same issue during our effort to upgrade Fandom's MW-on-k8s deployment, we ended up creating an HttpRequestFactory service override to dynamically use the current k8s service as proxy for request URLs known to belong to wikis on our platform. Some "automatic" solution like that could help avoid the whack-a-mole of updating every extension and core call site that makes internal HTTP requests. The logic for determining whether an URL should be automatically proxied or not would probably be similar to the existing core MWHttpRequest::isLocalURL method.

Thanks, this is super helpful. I think this is mostly the direction we should go in, developers and configuration can still use the normal wiki domains, e.g. https://meta.wikimedia.org/w/api.php, but internally MediaWiki will route the request over an envoy proxy.

For this to work, we'll need something like $wgLocalHttpProxy, which allows setting a different proxy (from $wgHttpProxy) for when MWHttpRequest::isLocalURL returns true.

One other consideration is whether we need to specifically route index.php and api.php requests to the correct cluster, or can we send them all to api? For reference, the main internal requests I see are either to action=raw (SpamBlacklist/TitleBlacklist) or action=render (InstantCommons) which are both API-like behavior.

Legoktm mentioned this in T289224: Split search.wikimedia.org out of ops/mediawiki-config into separate service.Aug 19 2021, 1:24 AM
gerritbot added a comment.Aug 19 2021, 4:08 PM

Change 713896 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@master] http: Add $wgLocalHTTPProxy to set a proxy for local requests

https://gerrit.wikimedia.org/r/713896

gerritbot added a project: Patch-For-Review.Aug 19 2021, 4:08 PM
Legoktm claimed this task.Aug 19 2021, 4:11 PM
In T288848#7293721, @Legoktm wrote:

One other consideration is whether we need to specifically route index.php and api.php requests to the correct cluster, or can we send them all to api? For reference, the main internal requests I see are either to action=raw (SpamBlacklist/TitleBlacklist) or action=render (InstantCommons) which are both API-like behavior.

I think the simplest thing to do is send it all to the API cluster for now.

gerritbot added a comment.Aug 19 2021, 4:37 PM

Change 713896 merged by jenkins-bot:

[mediawiki/core@master] http: Add $wgLocalHTTPProxy to set a proxy for local requests

https://gerrit.wikimedia.org/r/713896

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.20; 2021-08-23).Aug 19 2021, 5:00 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 19 2021, 5:10 PM
Legoktm mentioned this in T289518: Code Stewardship Review for search.wikimedia.org.Aug 23 2021, 5:51 PM
gerritbot added a comment.Aug 23 2021, 9:36 PM

Change 714420 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] services_proxy: Add mwapi envoyproxy for MediaWiki-internal requests

https://gerrit.wikimedia.org/r/714420

gerritbot added a project: Patch-For-Review.Aug 23 2021, 9:36 PM
Legoktm added a comment.Aug 23 2021, 10:20 PM

My deployment plan is:

  • Turn on envoy proxy nowish, test various requests with curl manually
  • Enable proxy in mwdebug k8s deployment too.
  • After the train rolls to group0 tomorrow, set $wgLocalHTTPProxy to point to the new proxy
    • Test InstantCommons and GlobalUserPage still work (just find an uncached image/user on an obscure wiki)
  • Allow it to roll out with the rest of the train, and document it as a "risky change".
Legoktm added a comment.Aug 23 2021, 11:32 PM
In T288848#7303412, @Legoktm wrote:
  • Enable proxy in mwdebug k8s deployment too.

Note that this proxy should point to the mwdebug k8s deployment, not the main api_appserver cluster.

gerritbot added a comment.Aug 25 2021, 4:08 PM

Change 714420 merged by Legoktm:

[operations/puppet@production] services_proxy: Add mwapi envoyproxy for MediaWiki-internal requests

https://gerrit.wikimedia.org/r/714420

Maintenance_bot removed a project: Patch-For-Review.Aug 25 2021, 4:10 PM
dancy mentioned this in T285298: Make all httpbb tests pass on the mwdebug deployment..Sep 13 2021, 6:08 PM
Joe added a comment.Sep 14 2021, 7:06 AM

I think it would be interesting to actually do what @TK-999 suggested and actually intercept all HTTP requests and route them to the correct proxy transparently.

That would allow us to do more complex stuff on our side, while keeping configuration easily intelligible to developers.

For now, though, this should be enough.

gerritbot added a comment.Sep 14 2021, 10:14 AM

Change 720944 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/mediawiki-config@master] Make the apple dictionary bridge work in kubernetes

https://gerrit.wikimedia.org/r/720944

gerritbot added a project: Patch-For-Review.Sep 14 2021, 10:14 AM
gerritbot added a comment.Sep 14 2021, 10:19 AM

Change 720944 abandoned by Giuseppe Lavagetto:

[operations/mediawiki-config@master] Make the apple dictionary bridge work in kubernetes

Reason:

the search bridge is being split to a separate service.

https://gerrit.wikimedia.org/r/720944

Maintenance_bot removed a project: Patch-For-Review.Sep 14 2021, 11:10 AM
Legoktm added a comment.Sep 15 2021, 6:13 PM

My overdue status update:

Joe added a comment.Sep 15 2021, 7:31 PM

Using envoy as a proper proxy instead than a transparent one makes me uneasy a bit; it's not how we've used it, and I don't think I'm happy testing new code paths there with mediawiki specifically.

Legoktm added a comment.Sep 15 2021, 11:37 PM

We could teach MediaWiki how to use a transparent proxy instead, I'll poke at that.

Joe added a comment.Sep 16 2021, 6:17 AM
In T288848#7357474, @Legoktm wrote:

We could teach MediaWiki how to use a transparent proxy instead, I'll poke at that.

Yeah that's basically being able to change the IP:PORT you're connecting to, and possibly use HTTP instead of HTTPS.

But at this point I think it would be interesting to have a way to intercept all http calls, look up the hostname in a table, and if it corresponds, intercept the call and direct it to our middleware.

The third alternative is to actually intercept the calls at the container level, maybe even as simple as adding entries to /etc/hosts...

jijiki added a parent task: T290536: Serve production traffic via Kubernetes.Oct 7 2021, 10:20 AM
gerritbot added a comment.Oct 8 2021, 7:26 PM

Change 728622 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@master] Allow using a transparent proxy for local HTTP requests

https://gerrit.wikimedia.org/r/728622

gerritbot added a project: Patch-For-Review.Oct 8 2021, 7:26 PM
Legoktm added a comment.Oct 8 2021, 7:30 PM

After reading https://en.wikipedia.org/wiki/Proxy_server#Transparent_proxy I'm not exactly sure "transparent proxy" is the correct terminology, but the patch should work.

gerritbot added a comment.Oct 16 2021, 4:41 AM

Change 728622 merged by jenkins-bot:

[mediawiki/core@master] Allow using a reverse proxy for local HTTP requests

https://gerrit.wikimedia.org/r/728622

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.5; 2021-10-19).Oct 16 2021, 5:00 AM
Maintenance_bot removed a project: Patch-For-Review.Oct 16 2021, 5:10 AM
gerritbot added a comment.Oct 18 2021, 5:35 PM

Change 731757 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@wmf/1.38.0-wmf.4] Allow using a reverse proxy for local HTTP requests

https://gerrit.wikimedia.org/r/731757

gerritbot added a project: Patch-For-Review.Oct 18 2021, 5:35 PM
gerritbot added a comment.Oct 18 2021, 10:54 PM

Change 731757 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.4] Allow using a reverse proxy for local HTTP requests

https://gerrit.wikimedia.org/r/731757

Stashbot added a comment.Oct 18 2021, 10:56 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-18T22:56:33Z] <legoktm@deploy1002> Synchronized php-1.38.0-wmf.4/includes/http/MWHttpRequest.php: Allow using a reverse proxy for local HTTP requests (T288848) (duration: 00m 56s)

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.4; 2021-10-12); removed MW-1.38-notes (1.38.0-wmf.5; 2021-10-19).Oct 18 2021, 11:00 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 18 2021, 11:11 PM
gerritbot added a comment.Oct 19 2021, 12:12 AM

Change 731861 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] Add framework for setting $wgLocalHTTPProxy

https://gerrit.wikimedia.org/r/731861

gerritbot added a project: Patch-For-Review.Oct 19 2021, 12:12 AM

Change 731862 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] Enable $wgLocalHTTPProxy on group0 wikis

https://gerrit.wikimedia.org/r/731862

gerritbot added a comment.Oct 19 2021, 5:57 PM

Change 731861 merged by jenkins-bot:

[operations/mediawiki-config@master] Add framework for setting $wgLocalHTTPProxy

https://gerrit.wikimedia.org/r/731861

Stashbot added a comment.Oct 19 2021, 5:59 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T17:59:26Z] <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Add framework for setting $wgLocalHTTPProxy (T288848) (1/2) (duration: 01m 05s)

Stashbot added a comment.Oct 19 2021, 6:00 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T18:00:48Z] <legoktm@deploy1002> Synchronized wmf-config/: Add framework for setting $wgLocalHTTPProxy (T288848) (2/2) (duration: 01m 06s)

gerritbot added a comment.Oct 20 2021, 4:37 AM

Change 731862 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable $wgLocalHTTPProxy on group0 wikis

https://gerrit.wikimedia.org/r/731862

Stashbot added a comment.Oct 20 2021, 4:40 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-20T04:40:02Z] <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Enable $wgLocalHTTPProxy on group0 wikis (T288848) (duration: 01m 05s)

Legoktm added a comment.Oct 21 2021, 6:17 PM

Thanks to other activity, I realized my patch didn't cover MultiHttpClient, which is used by Echo for cross-wiki notifications, and probably other things.

Legoktm added a comment.Oct 25 2021, 11:21 PM

MultiHttpClient is more complicated than the previous part since it's in includes/libs/ and isn't supposed to depend upon MediaWiki. Three approaches I came up with:

  1. Have similar isLocalURL() function, and just inject the domain list. This is code-wise the simplest but feels a weird fit for a library.
  2. Inject some kind of mapping for the domains to point to the reverse proxy.
  3. Add a callback hook to normalizeRequests() that checks if URL matches a local vhost and if so, adjusts the domain + host header.

I'm working on implementing #1 since it's the most straightforward and just mirrors what was done in MWHttpRequest.

gerritbot added a comment.Oct 27 2021, 11:27 PM

Change 735073 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@master] [WIP] Support $wgLocalHTTPProxy in MultiHttpClient

https://gerrit.wikimedia.org/r/735073

gerritbot added a comment.Nov 16 2021, 6:45 PM

Change 739323 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@master] http: Don't set X-Forwarded-Proto when using a reverse proxy

https://gerrit.wikimedia.org/r/739323

gerritbot added a comment.Nov 17 2021, 11:35 PM

Change 739323 merged by jenkins-bot:

[mediawiki/core@master] http: Don't set X-Forwarded-Proto when using a reverse proxy

https://gerrit.wikimedia.org/r/739323

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.12; 2021-12-06); removed MW-1.38-notes (1.38.0-wmf.4; 2021-10-12).Nov 18 2021, 12:00 AM
gerritbot added a comment.Nov 22 2021, 9:36 PM

Change 735073 merged by jenkins-bot:

[mediawiki/core@master] Support $wgLocalHTTPProxy in MultiHttpClient

https://gerrit.wikimedia.org/r/735073

gerritbot added a comment.Dec 20 2021, 7:00 PM

Change 748787 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/mediawiki-config@master] Enable $wgLocalHTTPProxy on Kubernetes, regardless of group

https://gerrit.wikimedia.org/r/748787

gerritbot added a comment.Dec 20 2021, 7:27 PM

Change 748787 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable $wgLocalHTTPProxy on Kubernetes, regardless of group

https://gerrit.wikimedia.org/r/748787

Legoktm added a comment.Dec 20 2021, 7:51 PM

With the above two patches, I was able to successfully load cross-wiki notifications, aka make cross-wiki HTTP requests from en.wikipedia to test.wikidata and meta.wikimedia, then from test.wikidata to meta.wikimedia. If there are any remaining issues, it probably means we just need to add more domains to $wgLocalVirtualHosts.

Legoktm closed this task as Resolved.Dec 23 2021, 6:56 PM

I will file a separate follow-up task for finishing the rollout on traditional appservers, the issue with Kubernetes is fixed.

Legoktm mentioned this in T298265: Have internal MediaWiki to MediaWiki HTTP requests use an envoyproxy on appservers.Dec 23 2021, 6:59 PM
Tgr mentioned this in T223413: Broken (empty) cross-wiki notification when using $wgLocalHTTPProxy (e.g. on Kubernetes).Oct 12 2023, 6:07 AM
Maintenance_bot removed a project: Patch-For-Review.Oct 12 2023, 6:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL