Page MenuHomePhabricator
Create Task
Maniphest T139535

Drop some Trusty permanent slaves from integration labs project
Closed, ResolvedPublic
Actions

Description

We have 15 Trusty instances each with 8 GBytes of RAM. Some jobs have been migrated to Nodepool trusty images already so we should be able to reclaim some of the instances.

They got spawned en masse back when we switched MediaWiki from PHP5.3 to PHP5.5 due to a surge of demand.

Seems we can drop at least half of them.

WARNING: due to labs capacity there is no return. Those instances will NOT be able to be recreated in a short term.

The Jenkins graph for executors business:

https://integration.wikimedia.org/ci/label/UbuntuTrusty/load-statistics?type=hour

graph-3.png (500×950 px, 33 KB)

Event Timeline

hashar created this task.Jul 6 2016, 10:16 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJul 6 2016, 10:16 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript
Paladox subscribed.Jul 6 2016, 10:18 PM
hashar claimed this task.Jul 7 2016, 1:09 PM

The Jenkins graph above is average so it does not accomodate for spikes :(

I created some more graphs on https://grafana.wikimedia.org/dashboard/db/releng-zuul to shows the max wait and # of builds for each of the three labels we use on permanent slaves (DebianJessie, UbuntuPrecise and UbuntuTrusty).

We have spikes for jobs on UbuntuTrusty , but I am pretty sure they are due to a conflict with the Jenkins Throttle plugin and the Gearman plugin, were a build is incorrectly assigned on a node.

I am thus dropping a couple 8GBytes RAM nodes: integration-slave-trusty-1024 and integration-slave-trusty-1025

Stashbot subscribed.Jul 7 2016, 1:10 PM

Mentioned in SAL [2016-07-07T13:10:38Z] <hashar> deleting integration-slave-trusty-1024 and integration-slave-trusty-1025 to free up some RAM. We have enough permanent Trusty slaves. T139535

hashar added a comment.Jul 7 2016, 1:37 PM

They were both on labvirt1010 which recovered 16GBytes of memory :-]

labvirt1010-2x8GB_deletion.png (422×677 px, 17 KB)

hashar closed this task as Resolved.Jul 22 2016, 11:48 AM

2 big ones got dropped. That is good enough for now. More will be deleted as jobs are shifted to Nodepool instances.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL