This may be the root cause of T289655: [Cloud VPS alert][paws] Puppet failure on paws-k8s-haproxy-1.paws.eqiad.wmflabs (172.16.0.191) and is related to T267082: Rebuild Toolforge servers that should not have NFS mounted (and with affinity)
The patterns for the VM names (straight out of cumin) are:
- paws-acme-chief-01.paws.eqiad1.wikimedia.cloud
- paws-k8s-haproxy-[1-2].paws.eqiad1.wikimedia.cloud
- abogott-proxy-canary.testlabs.eqiad1.wikimedia.cloud
- abogott-puppetmaster.testlabs.eqiad1.wikimedia.cloud
- krenair-t252721-test.testlabs.eqiad1.wikimedia.cloud
- paws-puppetmaster-01.paws.eqiad1.wikimedia.cloud
- tools-acme-chief-[01-02].tools.eqiad1.wikimedia.cloud
- tools-clushmaster-02.tools.eqiad1.wikimedia.cloud
- tools-elastic-[1-2].tools.eqiad1.wikimedia.cloud
- tools-legacy-redirector.tools.eqiad1.wikimedia.cloud
- tools-prometheus-05.tools.eqiad1.wikimedia.cloud
- tools-puppetmaster-02.tools.eqiad1.wikimedia.cloud
- tools-redis-[1003-1004].tools.eqiad1.wikimedia.cloud
- toolsbeta-acme-chief-01.toolsbeta.eqiad1.wikimedia.cloud
- toolsbeta-docker-imagebuilder-01.toolsbeta.eqiad1.wikimedia.cloud
- toolsbeta-docker-registry-01.toolsbeta.eqiad1.wikimedia.cloud
- toolsbeta-legacy-redirector.toolsbeta.eqiad1.wikimedia.cloud
- toolsbeta-mail-01.toolsbeta.eqiad1.wikimedia.cloud
- toolsbeta-puppetdb-02.toolsbeta.eqiad1.wikimedia.cloud
- toolsbeta-puppetmaster-04.toolsbeta.eqiad1.wikimedia.cloud
- toolsbeta-services-01.toolsbeta.eqiad1.wikimedia.cloud
- toolsbeta-sgecron-01.toolsbeta.eqiad1.wikimedia.cloud
- toolsbeta-workflow-test.toolsbeta.eqiad1.wikimedia.cloud
So 27 VMs. They need a force unmount of the two incorrect scratch volumes with removal of them from /etc/fstab *or* they should have nfs_mount: true set on the VM, which will allow puppet to correct things for the scratch mount (such as for cron servers).