Page MenuHomePhabricator
Create Task
Maniphest T333120

Migrate internal traffic to k8s
Open, In Progress, MediumPublic
Actions

Description

We need to progressively migrate traffic from our services to call the api in the mw-api-int cluster on k8s.

Right now we have (via this thanos query:

  • Mobileapps making 3k rps to the mediawiki API (!!!) <- Moved in 2nd stage
  • restbase making 600 rps <- Moved in 2nd stage
  • ores making 75-100 rps <- Deprecated
  • wikifeeds making ~ 70 rps <- Moved in 2nd stage
  • flink making ~ 40 rps <- Moved in 2nd stage

Everything else is basically marginal.

I propose we start moving all services on kubernetes to use mw-api-int now, with the exception of the ones named above.

Kubernetes services calling mediawiki

servicemoving to mw-api-intdoneCR/Task
api-gatewayyes✅ (some straggling traffic)T334065: Migrate api-gateway to mw-api-int T357907: Migrate remaining internal MW API traffic to k8s
cxserveryes✅T334204: Migrate cxserver to mw-api-int
linkrecommendationyes✅T334060: Migrate linkrecommendations to mw-api-int
mediawikiyes✅
push-notificationsyes✅T334061: Migrate push-notifications to mw-api-int
recommendation-apiyes✅T334062: Migrate recommendation-api to mw-api-int
termboxyes✅T334064: Migrate termbox to mw-api-int
flink-session-cluster-taskmanageryes✅T342252: Migrate rdf-streaming-updater to connect to mw-on-k8s
mobileappsyes✅T350846: Migrate mobileapps to k8s
wikifeedsyes✅T346447: Migrate wikifeeds to mw-api-int
eventgatesyes✅T346448: Migrate all eventgate installations to mw-api-int
mw-page-content-change-enrichyes✅T357785: Migrate mw-page-content-change-enrich to mw-api-int
changepropyes✅T360767: Migrate changeprop to mw-api-int
restbaseyes✅T358213: Migrate restbase from mwapi-async to mw-api-int

Details

Other Assignee
Joe
SubjectRepoBranchLines +/-
mediawiki: Switch backend calls to mw-api-intoperations/puppetproduction+1 -1
cxserver: Add mesh egressoperations/deployment-chartsmaster+34 -28
admin_ng: Add mw-on-k8s Egress rulesoperations/deployment-chartsmaster+27 -0
P:services_proxy::envoy: Add mw-api-intoperations/puppetproduction+14 -0
mw-api-int: add discovery recordsoperations/dnsmaster+4 -0
service_catalog: Add mw-api-int k8s service - 3operations/puppetproduction+1 -1
service_catalog: Add mw-api-int k8s service - 2operations/puppetproduction+1 -1
service_catalog: Add mw-api-int k8s service - 1operations/puppetproduction+42 -2
mw-api-int: Add recordsoperations/dnsmaster+4 -0
mw-api-int: Add recordsoperations/dnsmaster+8 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT255792 Quibble runs core:unit tests twice!
 OpenNoneT328919 Upgrade to PHPUnit 10
 OpenNoneT338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+
 OpenNoneT328921 Drop PHP 7.4 support from MediaWiki
 StalledNoneT334726 Use return type `never` in Wikibase
 OpenNoneT328922 Drop PHP 8.0 support from MediaWiki
 StalledNoneT319055 Upgrade to psr/container 2.x
 StalledFeatureNoneT364249 New upstream release for Pygments (2.18.0)
 StalledKrinkleT319432 Migrate WMF production from PHP 7.4 to PHP 8.1
 OpenNoneT291916 Tracking task for Bullseye migrations in production
 OpenNoneT368366 Upgrade K8s docker images to running in production on Buster with either Bullseye or Bookworm
 StalledNoneT356293 Migrate MW appservers' base images to bullseye
 OpenNoneT290536 Serve production traffic via Kubernetes
 In ProgressClement_GoubertT333120 Migrate internal traffic to k8s
 ResolvedClement_GoubertT334060 Migrate linkrecommendations to mw-api-int
 ResolvedClement_GoubertT334061 Migrate push-notifications to mw-api-int
 ResolvedClement_GoubertT334062 Migrate recommendation-api to mw-api-int
 InvalidClement_GoubertT334063 Migrate tegola-vector-tiles to mw-api-int
 ResolvedJoeT334064 Migrate termbox to mw-api-int
 ResolvedClement_GoubertT334065 Migrate api-gateway to mw-api-int
 ResolvedClement_GoubertT334204 Migrate cxserver to mw-api-int
 ResolvedClement_GoubertT334456 Monitor all mw-on-k8s deployments with httpbb
 ResolvedClement_GoubertT334561 Add mw-on-k8s deployments to mediawiki certificates
 ResolvedJoeT342252 Migrate rdf-streaming-updater to connect to mw-on-k8s
 ResolvedJoeT346447 Migrate wikifeeds to mw-api-int
 ResolvedJoeT346448 Migrate all eventgate installations to mw-api-int
 ResolvedJMeybohmT347397 Migrate functions-orchestrator service to mw-api-int
 ResolvedJoeT350846 Migrate mobileapps to k8s
 ResolvedClement_GoubertT357785 Migrate mw-page-content-change-enrich to mw-api-int
 ResolvedClement_GoubertT358213 Migrate restbase from mwapi-async to mw-api-int
 ResolvedClement_GoubertT298265 Have internal MediaWiki to MediaWiki HTTP requests use an envoyproxy on appservers
 DeclinedClement_GoubertT360625 Alter changeprop chart to use the service mesh
 ResolvedClement_GoubertT360767 Migrate changeprop to mw-api-int
 ResolvedelukeyT362316 Migrate ml-services to mw-api-int

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Mar 29 2023, 9:09 AM

Change 904061 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] service_catalog: Add mw-api-int k8s service - 3

https://gerrit.wikimedia.org/r/904061

gerritbot added a comment.Mar 29 2023, 9:43 AM

Change 904065 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/dns@master] mw-api-int: add geo and metafo records

https://gerrit.wikimedia.org/r/904065

Stashbot added a comment.Mar 29 2023, 9:57 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T09:57:03Z] <claime> Adding mw-api-int to service_catalog in service_setup - T333120

gerritbot added a comment.Mar 29 2023, 9:57 AM

Change 903217 merged by Clément Goubert:

[operations/puppet@production] service_catalog: Add mw-api-int k8s service - 1

https://gerrit.wikimedia.org/r/903217

Stashbot added a comment.Mar 29 2023, 9:58 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T09:58:56Z] <claime> running puppet on O:kubernetes::worker and O:lvs::balancer - T333120

Stashbot added a comment.Mar 29 2023, 10:37 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T10:37:09Z] <claime> Switching mw-api-int to lvs_setup - T333120

gerritbot added a comment.Mar 29 2023, 10:37 AM

Change 904060 merged by Clément Goubert:

[operations/puppet@production] service_catalog: Add mw-api-int k8s service - 2

https://gerrit.wikimedia.org/r/904060

Stashbot added a comment.Mar 29 2023, 10:41 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T10:41:03Z] <cgoubert@cumin1001> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1020*,lvs2010*} and A:lvs (T333120)

Stashbot added a comment.Mar 29 2023, 10:43 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T10:42:59Z] <cgoubert@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1020*,lvs2010*} and A:lvs (T333120)

Stashbot added a comment.Mar 29 2023, 10:46 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T10:46:23Z] <cgoubert@cumin1001> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1019*,lvs2009*} and A:lvs (T333120)

Stashbot added a comment.Mar 29 2023, 10:49 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T10:49:17Z] <cgoubert@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1019*,lvs2009*} and A:lvs (T333120)

Stashbot added a comment.Mar 29 2023, 10:50 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T10:50:22Z] <claime> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1019*,lvs2009*} and A:lvs (T333120)

Stashbot added a comment.Mar 29 2023, 10:50 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T10:50:57Z] <claime> Switching mw-api-int to production - T333120

gerritbot added a comment.Mar 29 2023, 10:51 AM

Change 904061 merged by Clément Goubert:

[operations/puppet@production] service_catalog: Add mw-api-int k8s service - 3

https://gerrit.wikimedia.org/r/904061

Stashbot added a comment.Mar 29 2023, 10:52 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T10:52:30Z] <claime> Running puppet on dns-auth - T333120

gerritbot added a comment.Mar 29 2023, 10:56 AM

Change 904065 merged by Clément Goubert:

[operations/dns@master] mw-api-int: add discovery records

https://gerrit.wikimedia.org/r/904065

Stashbot added a comment.Mar 29 2023, 10:58 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-29T10:58:30Z] <claime> authdns-update successful on all nodes - T333120

Clement_Goubert added a comment.Mar 29 2023, 11:00 AM

mw-api-int and mw-api-int-ro services now in production, we can proceed with creating the envoy listeners in https://gerrit.wikimedia.org/r/c/operations/puppet/+/903595/ and then switching services to use them.

Clement_Goubert updated the task description. (Show Details)Mar 29 2023, 11:04 AM
gerritbot added a comment.Mar 29 2023, 12:49 PM

Change 903595 merged by Clément Goubert:

[operations/puppet@production] P:services_proxy::envoy: Add mw-api-int

https://gerrit.wikimedia.org/r/903595

Clement_Goubert updated the task description. (Show Details)Mar 29 2023, 2:05 PM
KartikMistry subscribed.Apr 4 2023, 4:37 PM
Clement_Goubert updated the task description. (Show Details)Apr 5 2023, 9:06 AM
Clement_Goubert updated the task description. (Show Details)Apr 5 2023, 9:14 AM
Clement_Goubert closed subtask T334063: Migrate tegola-vector-tiles to mw-api-int as Invalid.
Clement_Goubert updated the task description. (Show Details)Apr 5 2023, 9:21 AM
Clement_Goubert updated the task description. (Show Details)Apr 6 2023, 1:08 PM
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert changed the status of subtask T334456: Monitor all mw-on-k8s deployments with httpbb from Open to In Progress.Apr 11 2023, 9:51 AM
JMeybohm mentioned this in T333575: mediawiki-event-enrichment in k8s should use mwapi-async envoy listener for MW api requests.Apr 11 2023, 11:20 AM
Clement_Goubert closed subtask T334456: Monitor all mw-on-k8s deployments with httpbb as Resolved.Apr 12 2023, 8:42 AM
Clement_Goubert closed subtask T334561: Add mw-on-k8s deployments to mediawiki certificates as Resolved.Apr 13 2023, 11:23 AM
gerritbot added a comment.Apr 13 2023, 1:15 PM

Change 908542 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] admin_ng: Add mw-on-k8s Egress rules

https://gerrit.wikimedia.org/r/908542

gerritbot added a comment.Apr 13 2023, 1:28 PM

Change 908542 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: Add mw-on-k8s Egress rules

https://gerrit.wikimedia.org/r/908542

gerritbot added a comment.Apr 13 2023, 2:01 PM

Change 908553 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] cxserver: Add mesh egress

https://gerrit.wikimedia.org/r/908553

gerritbot added a comment.Apr 13 2023, 2:35 PM

Change 908553 merged by jenkins-bot:

[operations/deployment-charts@master] cxserver: Add mesh egress

https://gerrit.wikimedia.org/r/908553

Clement_Goubert closed subtask T334204: Migrate cxserver to mw-api-int as Resolved.Apr 13 2023, 3:23 PM
Clement_Goubert updated the task description. (Show Details)Apr 14 2023, 9:05 AM
Clement_Goubert closed subtask T334060: Migrate linkrecommendations to mw-api-int as Resolved.Apr 17 2023, 2:14 PM
Clement_Goubert updated the task description. (Show Details)Apr 18 2023, 8:52 AM
Clement_Goubert closed subtask T334061: Migrate push-notifications to mw-api-int as Resolved.Apr 24 2023, 1:33 PM
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert closed subtask T334062: Migrate recommendation-api to mw-api-int as Resolved.May 3 2023, 11:03 AM
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert changed the status of subtask T334064: Migrate termbox to mw-api-int from Open to Stalled.May 3 2023, 1:13 PM
Clement_Goubert mentioned this in T290536: Serve production traffic via Kubernetes.Jul 13 2023, 11:11 AM
Joe closed subtask T334065: Migrate api-gateway to mw-api-int as Resolved.Jul 18 2023, 2:38 PM
Joe updated the task description. (Show Details)
Joe changed the status of subtask T342252: Migrate rdf-streaming-updater to connect to mw-on-k8s from Open to In Progress.Jul 20 2023, 11:57 AM
Joe closed subtask T342252: Migrate rdf-streaming-updater to connect to mw-on-k8s as Resolved.Jul 24 2023, 9:15 AM
Krinkle removed a project: Performance-Team (Radar).Aug 6 2023, 10:15 PM
Joe closed subtask T334064: Migrate termbox to mw-api-int as Resolved.Aug 24 2023, 9:21 AM
Joe updated the task description. (Show Details)Sep 15 2023, 1:21 PM
Joe updated the task description. (Show Details)Sep 15 2023, 1:30 PM
Joe updated the task description. (Show Details)Sep 15 2023, 1:34 PM
Joe closed subtask T346448: Migrate all eventgate installations to mw-api-int as Resolved.Sep 26 2023, 10:35 AM
Joe updated the task description. (Show Details)
Joe closed subtask T346447: Migrate wikifeeds to mw-api-int as Resolved.Sep 26 2023, 1:12 PM
Joe updated the task description. (Show Details)
Jdforrester-WMF changed the status of subtask T347397: Migrate functions-orchestrator service to mw-api-int from Open to In Progress.Sep 27 2023, 12:58 PM
JMeybohm closed subtask T347397: Migrate functions-orchestrator service to mw-api-int as Resolved.Sep 27 2023, 3:04 PM
kamila closed subtask T350846: Migrate mobileapps to k8s as Resolved.Jan 17 2024, 5:29 PM
Clement_Goubert reopened subtask T357785: Migrate mw-page-content-change-enrich to mw-api-int as In Progress.Feb 19 2024, 2:20 PM
akosiaris updated the task description. (Show Details)Feb 20 2024, 4:58 PM
Clement_Goubert closed subtask T357785: Migrate mw-page-content-change-enrich to mw-api-int as Resolved.Feb 21 2024, 12:30 PM
Clement_Goubert updated the task description. (Show Details)Feb 21 2024, 4:54 PM
Clement_Goubert changed the status of subtask T358213: Migrate restbase from mwapi-async to mw-api-int from Open to In Progress.Feb 22 2024, 1:07 PM
Clement_Goubert added a subtask: T298265: Have internal MediaWiki to MediaWiki HTTP requests use an envoyproxy on appservers.Feb 26 2024, 4:48 PM
Clement_Goubert closed subtask T298265: Have internal MediaWiki to MediaWiki HTTP requests use an envoyproxy on appservers as Resolved.
Clement_Goubert updated the task description. (Show Details)Mar 21 2024, 12:15 PM
Clement_Goubert closed subtask T360625: Alter changeprop chart to use the service mesh as Declined.Mar 22 2024, 11:52 AM
Clement_Goubert updated the task description. (Show Details)Mar 22 2024, 12:03 PM
Clement_Goubert updated the task description. (Show Details)Mar 22 2024, 12:12 PM
Clement_Goubert changed the status of subtask T360767: Migrate changeprop to mw-api-int from Open to In Progress.Mar 25 2024, 10:47 AM
Clement_Goubert closed subtask T360767: Migrate changeprop to mw-api-int as Resolved.Mar 25 2024, 12:18 PM
Clement_Goubert updated the task description. (Show Details)Mar 25 2024, 12:24 PM
Clement_Goubert closed subtask T358213: Migrate restbase from mwapi-async to mw-api-int as Resolved.Mar 28 2024, 11:55 AM
Clement_Goubert updated the task description. (Show Details)Mar 28 2024, 11:58 AM
Clement_Goubert updated the task description. (Show Details)
elukey closed subtask T362316: Migrate ml-services to mw-api-int as Resolved.Fri, May 31, 2:54 PM
gerritbot added a comment.Thu, Jun 13, 1:24 PM

Change #1043062 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] mediawiki: Switch backend calls to mw-api-int

https://gerrit.wikimedia.org/r/1043062

gerritbot added a comment.Thu, Jun 13, 2:03 PM

Change #1043062 merged by Clément Goubert:

[operations/puppet@production] mediawiki: Switch backend calls to mw-api-int

https://gerrit.wikimedia.org/r/1043062

Stashbot added a comment.Thu, Jun 13, 2:15 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-13T14:15:25Z] <cgoubert@deploy1002> Started scap: Change mwapi listener to mw-api-int - T333120

Clement_Goubert updated the task description. (Show Details)Thu, Jun 13, 2:21 PM
Stashbot added a comment.Thu, Jun 13, 2:21 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-13T14:21:24Z] <cgoubert@deploy1002> Finished scap: Change mwapi listener to mw-api-int - T333120 (duration: 06m 47s)

Jdforrester-WMF subscribed.Thu, Jun 13, 3:39 PM

Looks like this is now done except for "some straggling traffic" for the api-gateway?

image.png (958×2 px, 146 KB)

Clement_Goubert added a comment.Thu, Jun 13, 3:54 PM

Yes, but I will close it when I'm sure I have zero internal traffic on the bare metal clusters.

hnowlan subscribed.EditedThu, Jun 13, 3:56 PM

I believe the straggling traffic here is a misnomer/a graph misunderstanding - the API gateway's envoy config refers to traffic to the mediawiki API as "mwapi_cluster" internally before and after the hostname was changed. I believe these requests are normal and are being routed to k8s already - drop mwapi_cluster from the graph and we're at zero! 🎉

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits