| thcipriani | |
| Mar 20 2025, 9:27 PM |
| F58924465: integration_agent_cpu_pressure_8days.png | |
| Mar 26 2025, 11:25 AM |
| F58916956: jenkins_executors_label_docker.png | |
| Mar 25 2025, 2:41 PM |
CPU load on integration machines is high. It has been high during periods of high use for a long time.
Let's lower the number of executors per node and add new nodes to see if that helps, we already have needed quota.
Current:
Remaining quota:
What I'd like to do:
Needed (additional) capacity:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T388416 CI jobs failing with various timeouts (March 2025) | |||
| Resolved | thcipriani | T389554 Add some integration executors to spread the load |
I'm making integration-agent-docker-(1060-1062)
Brennen is making integration-agent-docker-(1063–1065)
Mentioned in SAL (#wikimedia-releng) [2025-03-20T21:41:39Z] <brennen> integration: launched integration-agent-docker-106{3,4,5} (T389554)
Mentioned in SAL (#wikimedia-releng) [2025-03-20T22:50:38Z] <brennen> integration: added jenkins nodes for integration-agent-docker-106{3,4,5} with 3 executors per each (T389554)
Looking at the last 30 days (all we have) at least that long. But looking at the pressure stall: that's getting worse.
It looks like these are picking up jobs. Will have to monitor and make sure they don't blow up in some unusual way.
Whilst I failed to log mine (unlike @brennen :D), mine are also launched.
I also lowered the number of executors on all other instances to 3.
We have the same number of executors, but spread across more machines now. Updated our instance creation runbook to say Docker agents get 3 executors.
Calling this complete.
Mentioned in SAL (#wikimedia-releng) [2025-03-20T23:31:41Z] <bd808> integration: thcipriani added integration-agent-docker-106{0,1,2} earlier today (T389554)
I have checked Jenkins built in graph for the Docker label and I can barely see a raise in number of executors. The graphs shows we had 68 and now 73. Brennen and Tyler each created three instances for a total of six we should be a total of 24 more executors.
Am I confused and those new instances were created as replacement of old instances that have been deleted?
https://integration.wikimedia.org/ci/label/Docker/load-statistics , the green line is the number of online executors.
At the same time 6 new nodes were added the per-node concurrency was lowered from 4 to 3. The intent was to lower per-cpu core utilization across the group by adding more cores and reducing the number of parallel jobs that could hit each core.
At the same time 6 new nodes were added the per-node concurrency was lowered from 4 to 3.
Somehow I missed that :)
The intent was to lower per-cpu core utilization across the group by adding more cores and reducing the number of parallel jobs that could hit each core.
I think that should reflect in less pressure being put on the node. When we worked on the parallelization of PHPUnit tests CPU pressure on Node was my concern, I have discovered [ https://docs.kernel.org/accounting/psi.html | Linux Pressure Stall Information ] which is a fine grained version of old Load Average measurement. I did add a graph for it a while back and yesterday I have added one so we can easily find the aggregated CPU pressure.
Tentatively:
If spreading the load is a thing, we should see less pressure. Below is a view over 8 day. Monday to Thursday for a reference baseline, Monday and Tuesday this week seems to have less pressure:
There is also less IO and memory stalling (= all non-idle tasks stalled on either resource)