Page MenuHomePhabricator

Migrate /data/project from math project into eqiad
Closed, ResolvedPublic

Description

Author: physik

Description:
Hi,

could you please migrate or create a new folder for math in the eqiad project.
Otherwise new instances in eqiad can not be created. I deleted all custom puppet roles... but that did not help.
Thanks.
The puppet error is:
notice: /Stage[first]/Apt::Update/Exec[/usr/bin/apt-get update]/returns: executed successfully
notice: /Stage[main]/Role::Labs::Instance/Mount[/home]/ensure: ensure changed 'unmounted' to 'mounted'
err: /Stage[main]/Role::Labs::Instance/Mount[/home]: Could not evaluate: Execution of '/bin/mount -o rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,nofsc /home' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/math/home failed, reason given by server:

No such file or directory

notice: /Stage[main]/Base::Puppet/Exec[puppet snmp trap]/returns: executed successfully
notice: instanceproject: math
notice: /Stage[main]/Base::Puppet/Notify[instanceproject: math]/message: defined 'message' as 'instanceproject: math'
notice: /Stage[main]/Role::Labs::Instance/Mount[/data/project]/ensure: ensure changed 'unmounted' to 'mounted'
err: /Stage[main]/Role::Labs::Instance/Mount[/data/project]: Could not evaluate: Execution of '/bin/mount -o rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,nofsc /data/project' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/math/project failed, reason given by server:

No such file or directory

Version: unspecified
Severity: minor

Details

Reference
bz62470

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:56 AM
bzimport added a project: Cloud-VPS.
bzimport set Reference to bz62470.
bzimport added a subscriber: Unknown Object (MLST).

Ah, this is because shared volumes (both home and project) were turned off in the project settings. We're going to have shared volumes for everyone in eqiad but that setting is still observed by some of the scripts.

I'm not sure what the proper general fix is, but in the meantime your project should be all set.

physik wrote:

This seems to break compatibilty with labsvagrant

err: /Stage[main]/Labs_vagrant/User[vagrant]/ensure: change from absent to present failed: Could not create user vagrant: Execution of '/usr/sbin/useradd -m vagrant' returned 12: useradd: cannot create directory /home/vagrant

physik wrote:

My feeling is that this is more a bug in the labsvagrant role.

If the labsvagrant class is broken then this should maybe have its own bug, or needs to be taken up with the people who wrote that class (yuvi and/or matanya). This doesn't look like a failure of eqiad, although it may have to do with the way the nfs /home share is set up.

Change 118053 had a related patch set uploaded by Physikerwelt:
Change home directory of vagrant user

https://gerrit.wikimedia.org/r/118053

(In reply to physikerwelt from comment #2)

err: /Stage[main]/Labs_vagrant/User[vagrant]/ensure: change from absent to
present failed: Could not create user vagrant: Execution of
'/usr/sbin/useradd -m vagrant' returned 12: useradd: cannot create directory
/home/vagrant

My best guess is that this was caused by the intermittent problem of the NFS server making shared read-only due to a race in application of ACLs. Coren knows about this problem and is trying to track down it's root cause. It seems to be cache related and will typically clear after a reboot or a 10-15 minute wait.

There was a related problem that the NFSv4 server in eqiad is enforcing idmapd identities and the NFS server didn't know about the locally created vagrant user. Coren has fixed this by adding a vagrant user to the eqiad LDAP.

You may have to create the /home/vagrant directory manually at this point. This may be something that needs to be fixed in the labs_vagrant role.

physik wrote:

In fact I had to create the /home/vagrant directory manually and fix permissions. Thus, I do not see the bug as fixed.

Change 118053 merged by Ori.livneh:
Change home directory of vagrant user

https://gerrit.wikimedia.org/r/118053