nfsiostat.py has

def get_default_config(self):
    """
    Returns the default collector settings
    """
    config = super(NfsiostatCollector, self).get_default_config()
    config.update({
        'enabled':  False,
        'path':     'nfsiostat',
        'devices': ['/home',
                    '/public/dumps',
                    '/data/scratch',
                    '/data/project'],
        'attributes': ['attr',
                       'read',
                       'write',
                       'getattr',
                       'lookup',
                       'dir_cache',
                       'page_stats']
    })

The directories listed as devices are all now symlinks, and should probably be changed to the underlying devices, which is all the mount_paths + project specific logic in https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/labs/nfsclient.pp.

We can override Diamond default config by supplying an external config file, which may be cleaner than continuing to hardcode this in the script (see DirectorySizeCollector - https://github.com/wikimedia/puppet/blob/9197b9c91ebb7e531705c18cded253363723266f/modules/role/manifests/labs/nfs/secondary.pp#L153 for example on doing this.

I think ideally this is smarter and figures out what local NFS mounts exist potentially. Other collector, for example disk space usage, do that and have whitelists/blacklists to manage including/excluding dynamically found resources.

+1 That sounds like the right thing to do

I discussed with @chasemp a patch like tihs:

https://phabricator.wikimedia.org/P6202

Mentioned in SAL (#wikimedia-operations) [2017-10-30T11:58:34Z] <arturo> depool tools-exec-1401 to test patch in T179024 --> aborrero@tools-bastion-03:~$ sudo exec-manage depool tools-exec-1401.tools.eqiad.wmflabs

Updated patch: https://phabricator.wikimedia.org/P6202

Now there seems to be some data after manually reading some files in the nfs mount:

https://graphite-labs.wikimedia.org/graphlot/?width=623&height=274&_salt=1509367272.694&target=tools.tools-exec-1401.nfsiostat.mounts.mnt_nfs_labstore1003-scratch.read.ops&lineMode=connected

Change 387243 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] diamond: nfsiostat: update collector to read from arbitrary NFS mount points

https://gerrit.wikimedia.org/r/387243

Mentioned in SAL (#wikimedia-operations) [2017-10-30T15:57:26Z] <arturo> depool again tools-exec-1401.tools.eqiad.wmflabs for more tests related to T179024

Change 387243 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] diamond: nfsiostat update collector to read from arbitrary NFS mount points

https://gerrit.wikimedia.org/r/387243

We run puppet merge also in the tools cluster.

After some days, stats appear to be being collected

https://graphite-labs.wikimedia.org/graphlot/?width=586&height=308&_salt=1509630717.808&from=00%3A00_20171023&until=23%3A59_20171102&target=tools.tools-exec-1401.nfsiostat.mounts.mnt_nfs_labstore-secondary-tools-home.read.ops&target=tools.tools-exec-1401.nfsiostat.mounts.mnt_nfs_labstore-secondary-tools-home.write.ops

seems fixed

https://graphite-labs.wikimedia.org/render/?width=724&height=560&_salt=1508952516.828&target=tools.tools-worker-1007.nfsiostat.labstore.ops&target=tools.tools-worker-1007.nfsiostat.labstore.ops_per_sec&target=tools.tools-worker-1007.loadavg.15&target=tools.tools-worker-1007.loadavg.05&target=tools.tools-worker-1007.loadavg.01&from=-48h

In T179024#3723763, @aborrero wrote:

We run puppet merge also in the tools cluster.

small note: this was run on the production puppetmasters