Page MenuHomePhabricator

Archives directory /vagrant/apt-cache/partial is missing.
Closed, ResolvedPublic

Description

tl;dr instructions to fix if you run into this. ssh into the vagrant instance, run sudo mkdir -p /vagrant/cache/apt/partial, and then run vagrant reload.

Steps to reproduce:

  1. start with an old vagrant installation (haven't had time yet to try and reproduce on a new one)
  2. update mediawiki-vagrant to current master
  3. run setup.sh
  4. run vagrant reload

vagrant reload will exit with this message:

Failed to mount folders in Linux guest. This is usually because
the "vboxsf" file system is not available. Please verify that
the guest additions are properly installed in the guest and
can work properly. The command attempted was:

mount -t vboxsf -o uid=id -u www-data,gid=getent group www-data | cut -d: -f3 /vagrant/logs /vagrant/logs
mount -t vboxsf -o uid=id -u www-data,gid=id -g www-data /vagrant/logs /vagrant/logs

Running the command manually just yields "mount: Protocol error". Googling suggests that this is caused by virtualbox-guest-dkms not being present (it isn't). Running apt-get install virtualbox-guest-dkms will give this error:

E: Archives directory /vagrant/apt-cache/partial is missing. - Acquire (2: No such file or directory)

After creating the directory, the install and then the mount works, but running vagrant reload again resets to the bad state (virtualbox-guest-dkms not installed, /vagrant/apt-cache/partial missing).


Version: unspecified
Severity: normal

Details

Reference
bz67976

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:28 AM
bzimport added a project: MediaWiki-Vagrant.
bzimport set Reference to bz67976.
Tgr created this task.Jul 14 2014, 9:14 AM

I can't recreate this in a clean checkout of MediaWiki-Vagrant. The cache/apt/partial directory exists in the git repository. Your problem may have been a bug in the cache migration code put in place when the directory structure was changed.

Tgr added a comment.Jul 19 2014, 11:24 PM

Yes, this doesn't happen in a new vagrant installation, but breaks an upgrade attempt of an old one in a really ugly way (in the end I had to migrate everything manually to a new box).

bd808 added a comment.Jul 20 2014, 3:50 AM

Looking at this closer, the problem is caused by cleaning up the previous apt-cache/** directories before puppet runs on the VM.

We install a file at /etc/apt/apt.conf.d/20shared-cache to configure an alternate cache directory for apt in manifests/base.pp. This file will change to reference the new /vagrant/cache/apt directory as soon as puppet runs on the VM. The vagrant-vbguest plugin however is running before puppet has changed the file to point to the new location and attempting to use apt.

A possible fix would be to remove the code in Vagrantfile that deletes the old directories and instead add puppet code to remove the directories. This would leave the old directory structure intact until puppet has run and reconfigured apt to use the new cache location.

Change 147878 had a related patch set uploaded by BryanDavis:
Defer deleting legacy cache directories

https://gerrit.wikimedia.org/r/147878

Tgr added a comment.Jul 20 2014, 5:26 AM

After applying the patch, running vagrant reload fails with the same message. However, virtualbox-guest-dkms is now installed on the VM, and the mount fails with a different error message:

/sbin/mount.vboxsf: mounting failed with the error: No such file or directory

Tgr added a comment.Jul 20 2014, 7:21 AM

Ignore me. I got lost while context switching and ran the commands on the host. The error is still the same as described in comment 0.

This might be only because this box is already in a messed-up state; I can try upgrading another box if you think it will work now.

bd808 added a comment.Jul 23 2014, 4:03 AM

(In reply to Tisza Gergő from comment #6)

This might be only because this box is already in a messed-up state; I can
try upgrading another box if you think it will work now.

If you have the energy to try and recreate on a "clean" install I'd appreciate the data. If not, I'll try to find some time to setup a test starting from 3c6a6606b5217a2fb16cfde4db31d5a6c8e102c5 and then updating to the current HEAD of master plus the patch.

Tgr added a comment.Jul 31 2014, 12:02 AM

Not sure if this is really related to the change you mention. The most recent box I am experiencing this on is at I91d51f6deb0b12b795f705cf8895783a11fa8695. Looks like all the old boxes became unusable when I upgraded one of them. Any chance this is somehow related to one of the two vagrant plugins which the setup script installs globally?

Tgr added a comment.Jul 31 2014, 12:08 AM

The error message after running vagrant reload is slightly different though:

$ vagrant reload
[default] Attempting graceful shutdown of VM...
[default] Clearing any previously set forwarded ports...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 3335 (adapter 1)
[default] -- 80 => 9095 (adapter 1)
[default] Running 'pre-boot' VM customizations...
[default] Booting VM...
[default] Waiting for machine to boot. This may take a few minutes...
[default] Machine booted and ready!
GuestAdditions versions on your host (4.3.10) and guest (4.2.16) do not match.
Reading package lists...
Building dependency tree...
Reading state information...
linux-headers-3.2.0-56-generic is already the newest version.
The following packages will be upgraded:

dkms

1 upgraded, 0 newly installed, 0 to remove and 112 not upgraded.
E: Archives directory /vagrant/apt-cache/partial is missing. - Acquire (2: No such file or directory)
E: Archives directory /vagrant/apt-cache/partial is missing. - Acquire (2: No such file or directory)
[default] The guest additions on this VM do not match the installed version of
VirtualBox! In most cases this is fine, but in rare cases it can
prevent things such as shared folders from working properly. If you see
shared folder errors, please make sure the guest additions within the
virtual machine match the version of VirtualBox you have installed on
your host and reload your VM.

Guest Additions Version: 4.2.16
VirtualBox Version: 4.3
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

apt-get update

Stdout from the command:

Stderr from the command:

E: Archives directory /vagrant/apt-cache/partial is missing. - Acquire (2: No such file or directory)

(In reply to Tisza Gergő from comment #9)

The error message after running vagrant reload is slightly different though:
$ vagrant reload

[...snip...]

[default] Machine booted and ready!
GuestAdditions versions on your host (4.3.10) and guest (4.2.16) do not
match.

[...snip...]

The following packages will be upgraded:

dkms

1 upgraded, 0 newly installed, 0 to remove and 112 not upgraded.
E: Archives directory /vagrant/apt-cache/partial is missing. - Acquire (2:
No such file or directory)

The output you give seems to show that the vagrant-vbguest plugin is attempting to update your guest's GuestAdditions. This would happen before the puppet run.

The change I mentioned (3c6a6606b5217a2fb16cfde4db31d5a6c8e102c5) was the last commit before Ori added logic to Vagrantfile to move any existing /vagrant/apt-cache directory to /vagrant/cache/apt. My patch is an attempt to give existing hosts that are expecting the cache to be in the old location to at least see empty directories before puppet tells them to use the new location.

I think what has happened (and continues to happen) for you is that the move from /vagrant/apt-cache to /vagrant/cache/apt has happened in the working directory for Vagrant on your local machine but puppet has not run inside the VM to tell it to look for cached apt packages in the new location.

I think if you mkdir -p apt-cache/partial in the MediaWiki-Vagrant checkout directory on your host computer with my patch applied before you vagrant reload or vagrant up then the VM will start. Then you can vagrant provision to fix the apt cache config via puppet.

Change 147878 merged by jenkins-bot:
Defer deleting legacy cache directories

https://gerrit.wikimedia.org/r/147878

bd808 added a comment.Aug 9 2014, 9:16 PM

I have seen this happen for some other user during the London hackathon (I don't remember who). The fix was to make the old git cache happy by mkdir -p apt-cache/partial and then vagrant up. This is not an ongoing issue for new installs but a horrible inconvenience we inflicted on users from before the consolidation of the guest hosted cache directories.

Tgr added a comment.Aug 18 2014, 9:04 AM

(In reply to Bryan Davis from comment #12)

The fix was to make the old git cache happy by mkdir -p apt-cache/partial
and then vagrant up.

This does not work for me, presumably because folder sharing is not set up yet at the point where this error happens, so anything done to the folder of the vagrantbox on the host machine does not affect /vagrant on the guest machine.

In T69976#718954, @Tgr wrote:

(In reply to Bryan Davis from comment #12)

The fix was to make the old git cache happy by mkdir -p apt-cache/partial
and then vagrant up.

This does not work for me, presumably because folder sharing is not set up yet at the point where this error happens, so anything done to the folder of the vagrantbox on the host machine does not affect /vagrant on the guest machine.

Me too, same situation (new virtualbox and latest vagrant code but using an old vagrant box). The workaround was to vagrant ssh to the guest, then on the guest

cd /vagrant
sudo mkdir -p apt-cache/partial
exit

Then`vagrant reload` worked,

Installing Virtualbox Guest Additions 4.3.10 - guest version is 4.2.16
Verifying archive integrity... All good.
Uncompressing VirtualBox 4.3.10 Guest Additions for Linux............
...

However, I had further problems

...
An error occurred during installation of VirtualBox Guest Additions 4.3.10. Some functionality may not work as intended.
...
mount -t vboxsf -o uid=`id -u root`,gid=`getent group vagrant | cut -d: -f3` tmp_vagrant-puppet-4_manifests /tmp/vagrant-puppet-4/manifests

I thought vagrant would use NFS, not vboxsf, hmm.

see P187

bd808 added a comment.Dec 31 2014, 1:03 AM

Me too, same situation (new virtualbox and latest vagrant code but using an old vagrant box).

Don't do that. :) The change to the cache structure was made ~6 months ago. If you have a VM that is that old without updates I would suggest deleting it and starting over rather than trying to fix it. I'd actually encourage all MediaWiki-Vagrant users to recreate their VMs from scratch (vagrant destroy -f; vagrant up) at least once a month. If you are using the roles and local puppet customizations all you should lose is accumulated state in the wiki database. Virtual machines are cattle, not pets.

bd808 moved this task from Backlog to Done on the MediaWiki-Vagrant board.Feb 26 2015, 12:53 AM
Tgr added a comment.Dec 10 2015, 11:03 PM

Just for the record, I am getting this on all my boxes after a host system upgrade (nothing in MW-Vagrant changed; must be caused by some change in the Vagrant upstream). sudo mkdir -p /vagrant/cache/apt/partial worked fine in this case.

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 10 2015, 11:03 PM

That directory exists in the rMWVA mediawiki/vagrant repo -- https://phabricator.wikimedia.org/diffusion/MWVA/browse/master/cache/apt/partial/

I wonder if something is trying to get at that directory before it has actually been mounted into the VM? @Tgr can you get a PUPPET_DEBUG=1 VAGRANT_LOG=debug vagrant ... log trace that reproduces the problem?

Tgr added a comment.EditedDec 10 2015, 11:59 PM

The directory definitely existed on those boxes before. With the system update, virtualbox got updated and vagrant tried to update the hosted virtualbox plugin accordingly; the error occurred in the process of that. Logfile:

I'm not sure it is worth the effort investigating though, it only happens once per box and is easily fixed by creating the directory.

In T69976#1871652, @Tgr wrote:

The directory definitely existed on those boxes before. With the system update, virtualbox got updated and vagrant tried to update the hosted virtualbox plugin accordingly; the error occurred in the process of that. Logfile:

I'm not sure it is worth the effort investigating though, it only happens once per box and is easily fixed by creating the directory.

Right. I've seen that before. Here's how it happens:

  1. VM is provisioned using our Puppet rules.
  2. Puppet changes the apt config to look for cache in /vagrant/cache/apt rather than the default /var/cache/apt location.
  3. VM is restarted.
  4. Vagrant's plugin for updating VirtualBox runs before any shares are mounted in the VM so it can ensure that vboxsf mounts actually work.
  5. The plugin runs apt.
  6. apt barfs because it is looking for a directory that is not mounted.

I'm not sure there is much we can do about this one. If we were using a custom base image I guess we could ensure that a /vagrant/apt/partial directory existed in the base image. That tree would normally be hidden by the /vagrant mount but it would make the directory exist when the share was not mounted.

A similar but inverted approach would be to add a separate mount of the mw-vagrant cache/apt directory over the top of the VM's existing /var/cache/apt directory rather than changing Apt's cache location in the first place.

Legoktm added a subscriber: Legoktm.Feb 8 2016, 5:50 AM

tl;dr instructions to fix if you run into this. ssh into the vagrant instance, run sudo mkdir -p /vagrant/cache/apt/partial, and then run vagrant reload.

bd808 updated the task description. (Show Details)Jul 18 2016, 10:16 PM
bd808 set Security to None.
bd808 added a comment.Jul 26 2016, 5:28 PM

If you get sick and tired of this causing you problems, you can globally disable the vbguest plugin by creating a ~/.vagrant/Vagrantfile with this content:

Vagrant.configure("2") do |config|
  # Disable vbguest updates
  config.vbguest.auto_update = false
end
bd808 added a comment.Mar 16 2017, 6:15 PM

We stopped installing vbguest in rMWVAc8ca59051837: Drop the vbguest plugin and sideload the mediawiki-vagrant plugin for T151928: Vagrant 1.9.0 unable to install local mediawiki-vagrant plugin. Lets close the loop and disable the plugin for anyone who may still have it installed by adding this to our Vagrantfile:

if Vagrant.has_plugin?('vagrant-vbguest')
    config.vbguest.auto_update = false
end
bd808 moved this task from Done to Awaiting review on the MediaWiki-Vagrant board.Mar 16 2017, 7:57 PM

Change 343103 merged by jenkins-bot:
[mediawiki/vagrant] Disable vbguest plugin auto updates

https://gerrit.wikimedia.org/r/343103

bd808 closed this task as Resolved.Mar 22 2017, 1:05 AM