To add some more data/justification for this task: the number of Gerrit repositories has grown:

The number of commits has grown:

We got a spike of mediawiki/core patches right after the hackathon due to ¯\_(ツ)_/¯:

This spike coincided with some long runtimes for a few repos, which slowed everyone working on code since every test job we run is using the same pool of a limited number of workers:

^ this graph is *just jenkins* zuul waiting for a worker + waiting for dependent changes to merge was even longer during that time period:

The combination of more repos + more patches + more jobs using the same pool of workers we had a few months ago led to lots of backup in the system.

We're running close to capacity, the late August slow downs would have been mitigated (somewhat, although there were/are other issues) by additional capacity.

Current quota/utilization:
22 / 37 instances. 125 / 130 VCPUs. 434.0 GB / 458.984375 GB RAM.

Similarly large projects:

• deployment-prep: 77 / 100 instances. 191 / 200 VCPUs. 366.0 GB / 415.203125 GB RAM.
• tools: 152 / 158 instances. 603 / 625 VCPUs. 1206.0 GB / 1281.73828125 GB RAM.

The relatively large RAM usage by the mediumram and bigram instances in the integration project changes how things can fit into our cloudvirt hosts (hypervisors). RAM is the only resource that we do not oversubscribe, so these instances "consume" more of the underlying host. The dashboard at https://grafana.wikimedia.org/d/000000579/wmcs-openstack-eqiad1?refresh=30s&orgId=1 gives an overview of the current OpenStack cluster. At the moment I'm typing this, our global CPU usage is 138% and our RAM usage is 44%.

@hashar / @thcipriani can you make a more concrete ask than "whatever you can spare"? How many more mediumram instances would you ideally like to have? How many existing instances (if any) can you get rid of? Do you have other wish list items for an ideal setup like "must be on SSD" or "must have 10G network"?

The relatively large RAM usage by the mediumram and bigram instances in the integration project changes how things can fit into our cloudvirt hosts (hypervisors). RAM is the only resource that we do not oversubscribe, so these instances "consume" more of the underlying host.

I remember those conversations which followed WMCS overflowing a few years ago. The root cause is that Nova never got taught how to use memory ballooning with libvirt/kvm (but it is available on other hypervisors). I blame OpenStack for defaulting that to ram_allocation_ratio=1.5 (50% over commit). But I digress.

The dashboard at https://grafana.wikimedia.org/d/000000579/wmcs-openstack-eqiad1?refresh=30s&orgId=1 gives an overview of the current OpenStack cluster. At the moment I'm typing this, our global CPU usage is 138% and our RAM usage is 44%.

That is a nice dashboard, very helpful. Discounting cloudvirt1014 and cloudvirt1028 which have more than 90% memory usage, the next four at ~ 70% memory usage still have ~ 100G free and could potentially fit 4 24G instances each. Overall there is 6.5TBytes of free memory free, which sounds like the resource is abundant.

The CPU oversubscription could be a concern though, if I got it right we oversubscribe by a ratio of 4 (cpu_allocation_ratio=4.0). It is usually fine assuming most VM are idling with some spike of CPU from time to time. With CI, the CPU demand kind of defeat the over subscription, an instance with 8vCPUs might well have 6 of them busy and that single instance could probably represent 25% of the real CPUs on the machine. But I guess that can be deal with by spreading those CI instances across the cloudvirt.

We still have 7 bigram instances for a consumption of 7 * 36 GB = 252 GB. Since they got created we have lowered the number of concurrent builds from 6 to 4 per instances since the jobs are typically CPU bound. That in turns mean we need less memory and since I am well aware that the whole memory of the instance is actually used on the cloudvirt I went to migrate them to mediumram (24G). That would save 7 * (36G - 24GB) = 84G (T226233), or down from 434GB to 350GB memory usage for the project.

The whole list:

We can probably drop integration-slave-jessie-1004 and saucelabs-02, though they are rather small and offer some kind of redundancy for the jobs being build on them.

The bulk of integration-slave-docker are being rebuild to save 84G of RAM.

@hashar / @thcipriani can you make a more concrete ask than "whatever you can spare"? How many more mediumram instances would you ideally like to have? How many existing instances (if any) can you get rid of? Do you have other wish list items for an ideal setup like "must be on SSD" or "must have 10G network"?

Eventually we will have 12 mediumram instances able to run up to 48 jobs concurrently for a total usage of:

I would like to aim at progressively doubling that pool, but stop earlier once the level of service is deemed good enough (ie the time jobs start fast enough / the queue stays reasonably low).

Then reviewing dashboards:

• Max launch wait: https://grafana.wikimedia.org/d/000000321/zuul?panelId=33&fullscreen&orgId=1&from=now-30d&to=now
• Gearman wait queue: https://grafana.wikimedia.org/d/000000322/zuul-gearman?panelId=10&fullscreen&orgId=1&from=now-30d&to=now

There is less pressure on CI right now due to other actions (dropping some old php versions, running slightly less tests etc). So I guess we should just add 4 mediumram instances (from 12 to 16) or an increase of:

Given 84G of RAM are going to be saved by rebuilding the bigram machines. To add four instances I could use 32 vCPU and 12G RAM to be added to the quota.

"must be on SSD"

I think the jobs are typically CPU bound rather than disk IO bound, but I could check. Maybe some jobs could benefit from using SSD.

or "must have 10G network"?

I had the concern for the central cache, but found out its egress traffic was shaped, I have raised the traffic shaping value and there is no more issue for now :)

Thank you!

I have added four more mediumram instances (integration-agent-docker-[1013-1016])