The VMs running on labvirt1001 seem mostly OK. And yet, actual behavior on the box itself is strange. Creating new VMs is very slow, and puppet runs take 10x longer than on equivalent labvirts.
Actual atop stats for disk IO look fine, so something interesting is going on :(
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Repool labvirt1001 | operations/puppet | production | +1 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T159721 labvirt1001 and 1002 cannot launch new VMs | |||
| Resolved | Andrew | T159835 Labvirt1001 has insanely slow IO |
Might well be related to T161006 which suggest the Scheduler prioritize mostly based on RAM usage. So we end up with Nodepool instances spawning mostly on the same host which most probably overload the CPUs.
@hashar, it's nothing to do with load. there are no VMs running on labvirt1001 and it still has the problem.
The one symptom I'm fixating on is puppet runs. A puppet run on labvirt1001 takes 811.99.
The same run on labvirt1002 (which is actually doing useful things) takes 37.36 seconds.
Mentioned in SAL (#wikimedia-operations) [2017-03-22T21:18:47Z] <andrewbogott> rebooting labvirt1001 because it is being terrible. https://phabricator.wikimedia.org/T159835
what about doing puppet agent -tv --debug --verbose to see what it is taking so long on?
@Paladox what kind of things should I be looking for when running puppet agent -tv --debug --verbose?
@yuvipanda hi, For example running it on gerrit-test3 returns P5113
So maybe it will tell us what bit it gets stuck on the longest.
It will go through bit by bit.
@yuvipanda puppet agent -tv --debug --verbose --evaltrace
https://ask.puppet.com/question/2755/howto-trace-execution-time-of-components-of-agent-run/
or doing "If you have reports=true in your puppet.conf on the agent, you can see the time spent on each resource type. Reports are stored on the agent in /var/lib/puppet/reports."
The current state of this is: I rebooted labvirt1001 and it got better. I've migrated a handful of tools exec nodes back to labvirt1001 and I'm going to keep an eye on it for a few weeks. If everything is still good come mid-April, we can just shrug and repool it.
Change 347887 had a related patch set uploaded (by Andrew Bogott):
[operations/puppet@production] Repool labvirt1001.
Change 347887 merged by Andrew Bogott:
[operations/puppet@production] Repool labvirt1001