This morning, the stretch grid shadow master was showing a load of 127 and login wasn't possible. After reboot, I found that it has been failing on NFS-related puppet code since earlier last week. At this time, the gridengine-shadow init won't start (goes into a restart loop and fails).
This needs to be fixed so that failover of the grid master is possible.
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| sonofgridengine: SGE_DEBUG_LEVEL required for sge shadow to run | Bstorm | operations/puppet | production | +1 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | bd808 | T218038 NFS issue affecting Toolforge SGE master | |||
| Resolved | • Bstorm | T219187 tools-sgegrid-shadow failures |
This is the start of NFS errors on the host Mar 18 15:58:41 tools-sgegrid-shadow puppet-agent[11372]: /usr/bin/timeout -k 5s 20s /bin/mkdir /mnt/nfs/labstore-secondary-tools-home returned 1 instead of one of [0]
At very least, that suggests that any problems around the root cause are long gone. However, shadowd still won't start.
Change 498973 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] sonofgridengine: SGE_DEBUG_LEVEL required for sge shadow to run
Change 498973 merged by Bstorm:
[operations/puppet@production] sonofgridengine: SGE_DEBUG_LEVEL required for sge shadow to run