Page MenuHomePhabricator

Have internal MediaWiki to MediaWiki HTTP requests use an envoyproxy on appservers
Closed, ResolvedPublic

Description

This is split from T288848: Make HTTP calls work within mediawiki on kubernetes .

Currently this functionality is enabled on group0 wikis, it just needs to be rolled out to remaining wikis, with a bit of monitoring of logs/potential user complaints.

// Temporary setting to stage the rollout of $wgLocalHTTPProxy (T288848)
'wmgUseLocalHTTPProxy' => [
	'default' => false,
	'group0' => true,
],

Event Timeline

Group0 is done, I can pick this up and finish it. Any preferences on wikis? I was thinking about adding a couple low traffic ones from group1, then commons, then all of group1 and then group2. Any objections?

Change 1004135 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/mediawiki-config@master] Enable $wgLocalHTTPProxy on group1 wikis

https://gerrit.wikimedia.org/r/1004135

Change 1004135 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable $wgLocalHTTPProxy on group1 wikis

https://gerrit.wikimedia.org/r/1004135

Mentioned in SAL (#wikimedia-operations) [2024-02-22T14:28:09Z] <cgoubert@deploy2002> Started scap: Backport for [[gerrit:1004135|Enable $wgLocalHTTPProxy on group1 wikis (T298265)]]

Mentioned in SAL (#wikimedia-operations) [2024-02-22T14:29:43Z] <cgoubert@deploy2002> cgoubert: Backport for [[gerrit:1004135|Enable $wgLocalHTTPProxy on group1 wikis (T298265)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-02-22T14:45:56Z] <cgoubert@deploy2002> Finished scap: Backport for [[gerrit:1004135|Enable $wgLocalHTTPProxy on group1 wikis (T298265)]] (duration: 17m 46s)

Appserver clusters are starting to use the envoy listener

image.png (921×2 px, 120 KB)

Change 1006497 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/mediawiki-config@master] Enable $wgLocalHTTPProxy on all wikis

https://gerrit.wikimedia.org/r/1006497

Change 1006497 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable $wgLocalHTTPProxy on all wikis

https://gerrit.wikimedia.org/r/1006497

Mentioned in SAL (#wikimedia-operations) [2024-02-26T14:34:40Z] <cgoubert@deploy2002> Started scap: Backport for [[gerrit:1006497|Enable $wgLocalHTTPProxy on all wikis (T298265)]]

Mentioned in SAL (#wikimedia-operations) [2024-02-26T14:36:06Z] <cgoubert@deploy2002> cgoubert: Backport for [[gerrit:1006497|Enable $wgLocalHTTPProxy on all wikis (T298265)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-02-26T14:48:05Z] <cgoubert@deploy2002> Finished scap: Backport for [[gerrit:1006497|Enable $wgLocalHTTPProxy on all wikis (T298265)]] (duration: 13m 24s)

All wikis are now using the local envoy

image.png (924×2 px, 175 KB)

The Y-axis not starting at 0 is misleading, but the numbers show a quasi-perfect 50/50 split of requests through envoy between those originating from mw-on-k8s (app=mediawiki in yellow) and from the baremetal cluster (cluster=api_appserver in blue and cluster=appserver in orange), which is consistent with the 50% of traffic mw-on-k8s is getting since T357507: Move 50% of mediawiki external requests to mw on k8s was completed this morning.

Clement_Goubert claimed this task.

Not seeing obvious issues in logstash or graphs, considering resolved. Feel free to reopen if something looks wrong.