Page MenuHomePhabricator

Maintained cpu usage (60%) on Vagrant managed Virtualbox VM with NFS sharing
Closed, DeclinedPublic

Description

Hi,

For the last 6 months I've been seeing a weird behavior with mediawiki-vagrant.

From time to time, cpu usage on process vboxheadless will spike to 250% for 2 or 3 minutes, and then it goes down to 100% and stays that way forever.

I've been trying to diagnose, but htop or top don't show any processes inside the VM taking any cpu (max 5%) so I'm not sure what's going on.

osx 10.9.5
Virtualbox 4.3.20 r96996
Vagrant 1.7.2

I've recreated the image a few times over the last 6 months but it always happens again.

Any ideas of what it could be? What can I do to help diagnose?

Doing a vagrant reload (which re-mounts NFS) brings everything back to normal, which is why I suspect it has to do with the NFS volumes.

Any ideas/pointers?

Event Timeline

Jhernandez raised the priority of this task from to Needs Triage.
Jhernandez updated the task description. (Show Details)
Jhernandez subscribed.

Hey @bd808, do you have any suggestions as to how to diagnose what's happening when the problem shows up again?

bd808 renamed this task from Maintained 100% cpu usage to Maintained 100% cpu usage on Vagrant managed Virtualbox VM with NFS sharing.May 12 2015, 3:05 PM

High guest CPU utilization seems to be a not uncommon complaint against Virtualbox: https://www.google.com/?q=virtualbox+100%25+cpu

A Vagrant reload, in addition to remounting NFS, does a cold boot of the VM so it would reset literally everything related to the VM. If you want to test the NFS hypothesis, you can disable NFS sharing with vagrant config nfs_shares no followed by vagrant reload.

Related ticket. https://www.virtualbox.org/ticket/14137

If I disable nfs_shares how is it going to share folders? I'm guessing performance is going to be a lot worse.

I may try it out on a weekend when I don't have to work.

Thanks for the answer bd808

If I disable nfs_shares how is it going to share folders? I'm guessing performance is going to be a lot worse.

When explicit NFS sharing is disabled Vagrant will fall back to configuring the default transport for the container in use. For VirtualBox the fallback is the built-in "shared folders" service. Testing has shown that NFS is a bit more performant than this default sharing solution which is why we made NFS the default configuration but the VM will be useable with native VirtualBox sharing.

I've since updated vagrant and everything else, re-provisioned, etc.

I've noticed vagrant is faster now than before (i hadn't updated the vagrant folder for months), and I haven't had again the 100% issues, but now VBoxHeadless is always at a sustained ~60% of CPU. I haven't seen those 250% spikes or 100% maintained cpu usage.

I've tried disabling nfs as @bd808 suggested above but I got 130% maintained cpu usage all the time. Re-enabling nfs gave me back a 60% maintained CPU usage.

This is making the laptop be like a frying pan on an oven...

Can't spot any weird activity on top, this is what it usually looks like (there is sometimes a 9% or 10% spike by a process but every 20 secs or so)

Screen Shot 2015-06-22 at 2.23.25 PM.png (597×1 px, 184 KB)

Jhernandez renamed this task from Maintained 100% cpu usage on Vagrant managed Virtualbox VM with NFS sharing to Maintained cpu usage (60%) on Vagrant managed Virtualbox VM with NFS sharing.Jun 22 2015, 12:26 PM

Have you tried a complete rebuild of the VM (vagrant destroy -f && vagrant up)? This will lose any state that is stored only in the VM (eg wiki content) but it ensures that the VM environment doesn't have any strange leftover mess from roles that you tried at some point and then removed or old bugs that have been fixed. I try to do this myself about once a month.

Short of that, audit your enabled roles and see if there are things that run extra processes that you can do without. Your htop shows elasticsearch running which can be a resource hog if you don't have a lot of cpu and ram to give over to the VM.

After a destroy -f and up and git-update instance seems to be stable at ~7% cpu usage.

I also upgraded to Yosemite by mistake yesterday... so I'm not sure if that's affected...

I've talked with @TheDJ over irc:

13:19 <thedj> joakino: that's a powersafe bug in vmware + nfs vagrant
13:20 <thedj> basically every time you suspend your mac, this will start happening
13:20 <joakino> thedj: is there any solution? i tried disabling nfs but cpu usage was even worse
13:20 <joakino> oh really
13:21 <thedj> lemme check.
13:22 <thedj> i remember i recently changed something to attempt to fix this, and i only now realize it's actually no longer happening...
13:24 <thedj> in Vagrantfile, i changed: root_share_options[:mount_options] = ['noatime','rsize=32767','wsize=32767','async']
13:24 <thedj> to root_share_options[:mount_options] = ['nolock','noatime','rsize=32767','wsize=32767','async']
13:24 <thedj> i'm not sure if i have been using vagrant less than before, or if it fixed it :)
13:26 <joakino> thedj: thanks, i'll try it out soon!
13:26 <joakino> i'll test also if it only happens after suspending

I'm going to keep an eye on cpu usage and make some tests suspending. If usage spikes I'll try the nolock change to the vagrantfile and will report back any findings.

bd808 changed the task status from Open to Stalled.Jan 26 2016, 3:41 AM
bd808 triaged this task as Low priority.Jan 26 2016, 3:41 AM

I'm closing this declined as I haven't experienced this exact behavior for some time and the task is old enough.