Mentioned in SAL (#wikimedia-releng) [2017-01-20T10:14:14Z] <hashar> puppet fails on "integration" labs instances due to an attempt to unmount the non existing NFS /home. Filled T155820

Is this running the latest puppet code?

The puppetmaster was stall with 2-3 days of lag and I rebased yesterday just before that happened.

I did create a new instance hashar-nfs.integration.eqiad.wmflabs and it has no trouble.

Looking at puppet:

# Allows per-host overriding of NFS mounts
$mount_nfs = hiera('mount_nfs', true)
# No NFS on labs metal for now.
if $::virtual == 'kvm' and $mount_nfs{
    require role::labs::nfsclient
}

Yes, there's probably going to be a refactor at some point for that. Does puppet run fine on older hosts too now?

On integration-slave-jessie-1001

# /usr/local/sbin/nfs-mount-manager check /home
It's mounted.
It seems healthy.

Which is wrong, there is some grep pattern that match another mount point:

# mount|grep /home
none on /srv/home/jenkins-deploy/tmpfs type tmpfs (rw,noatime,size=262144k)

check)

    if cat /proc/mounts | /usr/bin/awk '{print $2}' \
        | /bin/grep -qs $2; then
        echo "It's mounted."
    else
        echo "It's not mounted."
        exit 1
    fi

# echo /srv/home/jenkins-deploy/tmpfs|grep /home
/srv/home/jenkins-deploy/tmpfs

Change 333230 had a related patch set uploaded (by Hashar):
labstore: check should search for exact mount match

https://gerrit.wikimedia.org/r/333230

That matches the hosts failing puppet:

# salt --output=text '*' cmd.run 'grep -q /srv/home/jenkins-deploy/tmpfs /proc/mounts && echo yes || echo no'|sort
buildlog.integration.eqiad.wmflabs: no
castor.integration.eqiad.wmflabs: no
hashar-nfs.integration.eqiad.wmflabs: no
integration-publisher.integration.eqiad.wmflabs: no
integration-puppetmaster01.integration.eqiad.wmflabs: no
integration-saltmaster.integration.eqiad.wmflabs: no
integration-slave-docker-1000.integration.eqiad.wmflabs: no
integration-slave-jessie-1001.integration.eqiad.wmflabs: yes
integration-slave-jessie-1002.integration.eqiad.wmflabs: yes
integration-slave-jessie-android.integration.eqiad.wmflabs: no
integration-slave-precise-1002.integration.eqiad.wmflabs: yes
integration-slave-precise-1011.integration.eqiad.wmflabs: yes
integration-slave-precise-1012.integration.eqiad.wmflabs: yes
integration-slave-trusty-1001.integration.eqiad.wmflabs: yes
integration-slave-trusty-1003.integration.eqiad.wmflabs: yes
integration-slave-trusty-1004.integration.eqiad.wmflabs: yes
integration-slave-trusty-1006.integration.eqiad.wmflabs: yes
integration-slave-trusty-1011.integration.eqiad.wmflabs: yes
repository.integration.eqiad.wmflabs: no

https://gerrit.wikimedia.org/r/#/c/333230/ cherry picked on CI puppet master and that fixed it :-}

Change 333230 merged by Rush:
labstore: check should search for exact mount match

https://gerrit.wikimedia.org/r/333230

The fix here is actually a bit of a misnomer, and while it's probably best to anchor that regex the actual resolution is in 2791f51fda9b5d4e7252ae23034ce0822a39a5dc. i.e. home is not actually a mount only a symlink to one and should never actually be matched.

So I merged https://gerrit.wikimedia.org/r/#/c/333230/ after running through a few test cases but this was 2 levels of weirdness to begin with.

In T155820#2955685, @hashar wrote:

https://gerrit.wikimedia.org/r/#/c/333230/ cherry picked on CI puppet master and that fixed it :-}

Can you un-cherry pick?

In T155820#3036397, @chasemp wrote:

The fix here is actually a bit of a misnomer, and while it's probably best to anchor that regex the actual resolution is in 2791f51fda9b5d4e7252ae23034ce0822a39a5dc. i.e. home is not actually a mount only a symlink to one and should never actually be matched.

So I merged https://gerrit.wikimedia.org/r/#/c/333230/ after running through a few test cases but this was 2 levels of weirdness to begin with.

2791f51fda9b5d4e7252ae23034ce0822a39a5dc would not apply on projects that are not using NFS. That is the case for deployment-prep and integration at least. But yeah looks like that fix another oddity :-}

In T155820#2955685, @hashar wrote:

https://gerrit.wikimedia.org/r/#/c/333230/ cherry picked on CI puppet master and that fixed it :-}

Can you un-cherry pick?

Not sure what you mean. The Puppet repository on standalone puppetmaster is automatically rebased via a cronjob. If we want to test without https://gerrit.wikimedia.org/r/#/c/333230/ we can revert it on puppet master and run puppet to see what happens.

Regardless, looks like the task is solved for good now.