Re-using the mw-api-async-transition listener that was used for mobileapps, progressively transfer restbase's backend mediawiki api calls to mw-api-int
- 10%
- 50%
- 100%
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | None | T255792 Quibble runs core:unit tests twice! | |||
| Open | None | T328919 Upgrade to PHPUnit 10 | |||
| Open | None | T338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+ | |||
| Open | None | T328921 Drop PHP 7.4 support from MediaWiki | |||
| Stalled | None | T334726 Use return type `never` in Wikibase | |||
| Open | None | T328922 Drop PHP 8.0 support from MediaWiki | |||
| Stalled | None | T319055 Upgrade to psr/container 2.x | |||
| Stalled | Krinkle | T319432 Migrate WMF production from PHP 7.4 to PHP 8.1 | |||
| Open | None | T291916 Tracking task for Bullseye migrations in production | |||
| Stalled | None | T356293 Migrate MW appservers' base images to bullseye | |||
| Open | None | T290536 Serve production traffic via Kubernetes | |||
| In Progress | Clement_Goubert | T333120 Migrate internal traffic to k8s | |||
| Resolved | Clement_Goubert | T358213 Migrate restbase from mwapi-async to mw-api-int |
Event Timeline
Comment Actions
Change 1005756 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/puppet@production] restbase: Start moving mwapi calls to mw-on-k8s
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-26T11:15:18Z] <claime> Stopping puppet on P:restbase to deploy 1005756 - T358213
Comment Actions
Change #1005756 merged by Clément Goubert:
[operations/puppet@production] restbase: Start moving mwapi calls to mw-on-k8s
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-26T11:19:13Z] <claime> enabling and running puppet on restbase2021.codfw.wmnet - T358213
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-26T11:24:04Z] <claime> enabling and running puppet on restbase1035.eqiad.wmnet - T358213
Comment Actions
Change #1014493 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/puppet@production] restbase: Migrate backend traffic to mw-api-int
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-26T12:54:15Z] <claime> enabling and running puppet on restbase2021.codfw.wmnet - T358213
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-26T12:54:20Z] <claime> enabling and running puppet on restbase1035.eqiad.wmnet - T358213
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-26T14:19:45Z] <claime> enabling and running puppet on P:restbase - T358213
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-26T14:20:51Z] <claime> Deploying split listener for 10% of backend restbase traffic to mw-api-int - T358213
Comment Actions
Change #1015016 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/puppet@production] restbase: Moving 50% of mwapi calls to mw-on-k8s
Comment Actions
Things to keep an eye on:
- Upstream error rate is higher on mw-api-int than bare-metal
- Connection establishment time is way higher on mw-api-int
- Upstream latencies are consistently higher on mw-api-int
As the envoy listener configuration is the same for both, it may come from the tls termination envoy configuration in mw-on-k8s, since we are seeing similar numbers for connections to mw-parsoid.
Some of it can also be explained by routing through kubernetes, as connection establishment time from restbase to other k8s services (for example mobileapps or cxserver) are around 25ms average p99.
In any case, I can't see an impact on client side latency or request rate at 10% of traffic, and propose moving forward to 50%.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-27T15:08:47Z] <claime> Disabling puppet on P:restbase - T358213
Comment Actions
Change #1015016 merged by Clément Goubert:
[operations/puppet@production] restbase: Moving 50% of mwapi calls to mw-on-k8s
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-27T15:11:28Z] <claime> enabling and running puppet on restbase2021.codfw.wmnet - T358213
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-27T15:14:26Z] <claime> enabling and running puppet on restbase1035.eqiad.wmnet - T358213
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-27T15:17:15Z] <claime> enabling and running puppet on P:restbase - T358213
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-27T15:51:21Z] <claime> 50% of backend RESTbase traffic to mw-api-int - T358213
Comment Actions
Change #1015278 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] mw-api-int: Double envoy concurrency
Comment Actions
Change #1015278 merged by jenkins-bot:
[operations/deployment-charts@master] mw-api-int: Double envoy concurrency
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-28T11:04:37Z] <claime> RESTbase: Migrate backend traffic to mw-api-int - T358213
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-28T11:04:44Z] <claime> Disabling puppet on P:restbase - T358213
Comment Actions
Change #1014493 merged by Clément Goubert:
[operations/puppet@production] restbase: Migrate backend traffic to mw-api-int
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-28T11:09:50Z] <claime> enabling and running puppet on restbase2021.codfw.wmnet - T358213
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-28T11:12:27Z] <claime> enabling and running puppet on restbase1035.eqiad.wmnet - T358213
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-03-28T11:15:26Z] <claime> enabling and running puppet on P:restbase - T358213