Something strange is going on here. Taking tools-exec-1401 as an example:

valhallasw@tools-exec-1401:~$ uptime
 11:07:56 up 294 days, 20:36,  1 user,  load average: 148.57, 143.21, 128.77

Normally these kinds of load averages indicate an NFS issue, but NFS in the home directory feels fast. Looking at the NFS mounts:

labstore1003.eqiad.wmnet:/scratch on /mnt/nfs/labstore1003-scratch type nfs (rw,noexec,nosuid,nodev,noatime,vers=4,bg,intr,sec=sys,proto=tcp,port=0,lookupcache=all,nofsc,soft,timeo=300,retrans=3,addr=10.64.4.10,clientaddr=10.68.17.202)

nfs-tools-project.svc.eqiad.wmnet:/project/tools/project on /mnt/nfs/labstore-secondary-tools-project type nfs (rw,noatime,vers=4,bg,intr,sec=sys,proto=tcp,port=0,lookupcache=all,nofsc,hard,addr=10.64.37.18,clientaddr=10.68.17.202)

nfs-tools-project.svc.eqiad.wmnet:/project/tools/home on /mnt/nfs/labstore-secondary-tools-home type nfs (rw,noatime,vers=4,bg,intr,sec=sys,proto=tcp,port=0,lookupcache=all,nofsc,hard,addr=10.64.37.18,clientaddr=10.68.17.202)

labstore1006.wikimedia.org:/dumps on /mnt/nfs/dumps-labstore1006.wikimedia.org type nfs (ro,noatime,vers=4,bg,intr,sec=sys,proto=tcp,port=0,lookupcache=all,nofsc,soft,timeo=300,retrans=3,addr=208.80.154.7,clientaddr=10.68.17.202)

labstore1007.wikimedia.org:/dumps on /mnt/nfs/dumps-labstore1007.wikimedia.org type nfs (ro,noatime,vers=4,bg,intr,sec=sys,proto=tcp,port=0,lookupcache=all,nofsc,soft,timeo=300,retrans=3,addr=208.80.155.106,clientaddr=10.68.17.202)

• ls /mnt/nfs/labstore1003-scratch hangs
• ls /mnt/nfs/labstore-secondary-tools-project is OK
• ls /mnt/nfs/labstore-secondary-tools-home is OK
• ls /mnt/nfs/dumps-labstore1006.wikimedia.org hangs
• ls /mnt/nfs/dumps-labstore1007.wikimedia.org is OK

The high load averages seem to be there because of prometheus, which has a large number of threads in D (uninterruptible sleep -- typically 'waiting for NFS') state:

valhallasw@tools-exec-1401:~$ ps -T aux | grep " D" | wc -l
163
valhallasw@tools-exec-1401:~$ ps -T aux | grep " D" | grep "prometheus" | wc -l
161
valhallasw@tools-exec-1401:~$ uptime
 11:17:32 up 294 days, 20:45,  2 users,  load average: 163.55, 158.31, 143.06

This also makes sense -- it's probably trying to get statistics on the mounted NFS shares.

The two problematic mounts are set to soft,timeo=300, which means any activity should time out after 30 seconds. But this is not consistent with what I see: ls does not time out, and there are many prometheus processes older than 30 seconds.

sudo service prometheus-node-exporter restart seems to at least temporarily resolve the issue, but it seems likely prometheus subprocesses will hang again sooner or later.

sudo mount -o remount /mnt/nfs/labstore1003-scratch does not help to bring the scratch mount back online. Note, the mount parameters changed from

labstore1003.eqiad.wmnet:/scratch on /mnt/nfs/labstore1003-scratch type nfs (rw,noexec,nosuid,nodev,noatime,vers=4,bg,intr,sec=sys,proto=tcp,port=0,lookupcache=all,nofsc,soft,timeo=300,retrans=3,addr=10.64.4.10,clientaddr=10.68.17.202)

to

labstore1003.eqiad.wmnet:/scratch on /mnt/nfs/labstore1003-scratch type nfs (rw,noexec,nosuid,nodev,noatime,vers=4,bg,intr,sec=sys,proto=tcp,port=0,lookupcache=all,nofsc,soft,timeo=300,retrans=3)

likely because no connection could be made.

I cannot ping labstore1003:

valhallasw@tools-exec-1401:~$ ping labstore1003.eqiad.wmnet
PING labstore1003.eqiad.wmnet (10.64.4.10) 56(84) bytes of data.
From ae2-1118.cr2-eqiad.wikimedia.org (10.64.20.3) icmp_seq=1 Packet filtered

but it does respond on the nfs port:

valhallasw@tools-exec-1401:~$ telnet labstore1003.eqiad.wmnet 111
Trying 10.64.4.10...
Connected to labstore1003.eqiad.wmnet.

https://grafana.wikimedia.org/d/000000032/labs-monitoring?refresh=5m&orgId=1&from=1550921996943&to=1551526796943&var-LabstoreServers=All does not show anything particularly different on the labstore servers.

I'm not sure whether the Puppet are related -- the puppetmaster does not (always?) start correctly when it needs to respond to requests:

Mar  2 11:45:28 tools-exec-1421 puppet-agent[25767]: (/Stage[main]/Apt/File[/usr/local/bin/apt2xml]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/apt/apt2xml.py: Error 500 on SERVER: <!DOCTYPE HTML PUBLIC "-
//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
[...]
Mar  2 11:45:28 tools-exec-1421 puppet-agent[25767]: (/Stage[main]/Apt/File[/usr/local/bin/apt2xml]) #011#011#011#011An error occurred while starting up the preloader: it did not write a startup response in time. Please read <a href="https://github.com/phusion/passenger/wiki/Debugging-application-startup-problems">this article</a> for more information about this problem.<br>

Does the puppetmaster use NFS for some storage?

labstore1006 seems to have recovered:

valhallasw@tools-exec-1401:~$ ls  /mnt/nfs/dumps-labstore1006.wikimedia.org
lost+found  xmldatadumps

There were some alerts in #wikimedia-operations:

[09:25:21] <icinga-wm>	 PROBLEM - Host labstore1006 is DOWN: PING CRITICAL - Packet loss = 100%
[09:29:13] <icinga-wm>	 RECOVERY - Host labstore1006 is UP: PING WARNING - Packet loss = 28%, RTA = 36.35 ms
[09:31:51] <icinga-wm>	 PROBLEM - NFS on labstore1006 is CRITICAL: connect to address 208.80.154.7 and port 2049: Connection refused
[09:43:55] <icinga-wm>	 PROBLEM - toolschecker: check mtime mod from tools cron job on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/toolscron - 185 bytes in 0.086 second response time
[11:56:47] <icinga-wm>	 RECOVERY - NFS on labstore1006 is OK: TCP OK - 0.036 second response time on 208.80.154.7 port 2049

Not clear why it recovered, though.

It seems SGE has a hard dependency on the NFS dumps share.

Related T217473

@valhallasw pointed out that job not getting scheduled could be caused by high load average and SGE deciding the servers aren't fit for purpose (instead of the NFS dumps share being down and causing the SGE scheduler itself to lock).

The soft-mounted dumps shares are known to bring toolforge to its knees (kubernetes as well) T203469 mentions some of that. If one of the NFS shares must be actually offline, anything connected needs to be rebooted to preserve functionality. The interesting thing is that when this happens to a kubernetes server, the effect is generally far worse than gridengine. Failing over will not help. I'll take a look at the servers quick. There was no maintenance going on, and I believe labstore1006 is not even the active partner for toolforge (though, it is known already that even the inactive will cause this because it is monitored and connected to containers).

Finally read back and got to T217473. That really should have paged. I may copy the hiera values I added to make them page over to host records instead, I think 😭

labstore1003 is weirder: up 905 days (presuming @GTirloni already checked that but wanting it noted here that the reboots weren't contagious)
I guess that one was hanging just because it's slow and unreliable? Not sure.

Luckily, there is a fix already in store for labstore1003 in general: T209527

@valhallasw pointed out that this was not really happening on the new grid! They just gained some load and kept running. Upgrading may have actually fixed something!