| Clement_Goubert | |
| Aug 23 2023, 11:46 AM |
| F37730389: image.png | |
| Sep 18 2023, 11:16 AM |
| F37621959: image.png | |
| Aug 23 2023, 11:46 AM |
| F37621947: image.png | |
| Aug 23 2023, 11:46 AM |
| F37621955: image.png | |
| Aug 23 2023, 11:46 AM |
| F37621941: image.png | |
| Aug 23 2023, 11:46 AM |
As we can see from the following screenshots, our two outwards facing mw-on-k8s deployments are experiencing throttling of the tls-proxy container while their load is nowhere near CPU limit (similar to T342748: mw-on-k8s app container CPU throttling at low average load)
We may want to implement a similar solution (remove CPU limits) if we know envoy CPU isn't going to run away from us. Thoughts?
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | None | T255792 Quibble runs core:unit tests twice! | |||
| Open | None | T328919 Upgrade to PHPUnit 10 | |||
| Open | None | T338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+ | |||
| Open | None | T328921 Drop PHP 7.4 support from MediaWiki | |||
| Stalled | None | T334726 Use return type `never` in Wikibase | |||
| Open | None | T328922 Drop PHP 8.0 support from MediaWiki | |||
| Stalled | None | T319055 Upgrade to psr/container 2.x | |||
| Stalled | Krinkle | T319432 Migrate WMF production from PHP 7.4 to PHP 8.1 | |||
| Open | None | T291916 Tracking task for Bullseye migrations in production | |||
| Stalled | None | T356293 Migrate MW appservers' base images to bullseye | |||
| Open | None | T290536 Serve production traffic via Kubernetes | |||
| Resolved | Clement_Goubert | T344814 mw-on-k8s tls-proxy container CPU throttling at low average load | |||
| Open | None | T354532 Limit the concurrency of envoy in service mesh | |||
| Open | Clement_Goubert | T345243 Remove tls-proxy cpu limits on eventstreams | |||
| In Progress | Clement_Goubert | T345244 Remove tls-proxy cpu limits on eventgate |
Dumping the envoy configuration in one of our containers as well as there being no CLI flag set for it means envoy is setting its number of threads to the number of hardware cores.
In other words, we have a CPU limit of 500mCPU, a 48 core machine, so each worker thread gets ~10mCPU (this is just for illustration purposes because the allocation doesn't work like that).
In effect, we are letting envoy spawn 48 threads and throttling all of them
istio basically takes the CPU limit and rounds up in order to avoid this, but in our case that would give a concurrency of 1, which is probably not what we want (unless we think envoy can handle the connections with only one worker thread).
We can:
Personally, I think the first solution is the easiest and requires the least amount of finetuning, but it doesn't guarantee us against runaway effects.
Change 952148 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/docker-images/production-images@master] envoy: Add concurrency control to envoy cmdline
Change 952158 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] mesh: Add concurrency control for envoy workers
Change 952159 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] mediawiki: Remove limits for tls-proxy container
Change 952171 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] mediawiki: Generalize tls-proxy limits removal
Change 952148 merged by Clément Goubert:
[operations/docker-images/production-images@master] envoy: Add concurrency control to envoy cmdline
Change 952158 merged by jenkins-bot:
[operations/deployment-charts@master] mesh: Add concurrency control for envoy workers
Change 952159 merged by jenkins-bot:
[operations/deployment-charts@master] mediawiki: Remove limits for tls-proxy container
Mentioned in SAL (#wikimedia-operations) [2023-08-25T08:44:24Z] <claime> mw-debug: Remove limits for tls-proxy container - T344814
Change 952314 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] admin_ng: Remove limits on mw-debug namespace
Change 952314 merged by jenkins-bot:
[operations/deployment-charts@master] admin_ng: Remove limits on mw-debug namespace
Change 952316 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] mw-debug: Fix resourcequota on namespace
Change 952316 merged by jenkins-bot:
[operations/deployment-charts@master] mw-debug: Fix resourcequota on namespace
Change 952319 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] mediawiki: Fix revert
Change 952319 merged by jenkins-bot:
[operations/deployment-charts@master] mediawiki: Fix revert
Change 952812 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] mediawiki: Switch to set worker envoy threads
Change 952812 merged by jenkins-bot:
[operations/deployment-charts@master] mediawiki: Switch to set worker envoy threads
Mentioned in SAL (#wikimedia-operations) [2023-08-28T10:10:21Z] <claime> Deploying 952812 for T344814 to mw-debug and mw-api-ext
Change 952823 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] mediawiki: Global values override cpu limit removal
Change 952823 merged by jenkins-bot:
[operations/deployment-charts@master] mediawiki: Global values override cpu limit removal
Change 952867 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/deployment-charts@master] mediawiki: Remove tls-proxy CPU limits
Change 952171 abandoned by Clément Goubert:
[operations/deployment-charts@master] mediawiki: Generalize tls-proxy limits removal
Reason:
Superseded by I94e475715a5abda59ba48421cc61925fa83959da
Change 953230 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/puppet@production] kubernetes: Bump envoy image version to 1.23.10-2-s1
Change 953230 merged by Clément Goubert:
[operations/puppet@production] kubernetes: Bump envoy image version to 1.23.10-2-s2
Mentioned in SAL (#wikimedia-operations) [2023-08-29T10:30:03Z] <claime> Running puppet on deploy servers to bump envoy image version - T344814
Change 952867 merged by jenkins-bot:
[operations/deployment-charts@master] mediawiki: Remove tls-proxy CPU limits
Mentioned in SAL (#wikimedia-operations) [2023-08-29T10:39:31Z] <cgoubert@deploy1002> Started scap: Removing mw-on-k8s tls-proxy CPU limits - T344814
Mentioned in SAL (#wikimedia-operations) [2023-08-29T10:41:59Z] <cgoubert@deploy1002> Finished scap: Removing mw-on-k8s tls-proxy CPU limits - T344814 (duration: 02m 27s)
CPU limits have now been removed on all mw-on-k8s deployments except mw-misc. We'll wait a few days to see how the reduced concurrency impacts latency if at all, then resolve this task.
p50 latency increased slightly, we may want to up the concurrency a little to see what shakes.
Example mw-web eqiad
This is tweaking though, so I'm considering this task Resolved.