| aborrero | |
| Oct 2 2018, 12:16 PM |
| F27388932: image.png | |
| Dec 5 2018, 10:18 AM |
| F26275053: imagen.png | |
| Oct 2 2018, 12:16 PM |
This server (dumps) seems to be suffering high load average issues related to rsync jobs or nfs.
On 2018-10-02 we got paged by icinga about the load issue:
I jumped into the machine and rsyncd was the most CPU intensive process (about 100%) in both htop and iotop.
@ArielGlenn created this patch: https://gerrit.wikimedia.org/r/463925
But probably the load is also high because NFS.
Apparently labstore1006 is not suffering the same issues.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T205969 labstore1007: high load avg issue | |||
| Resolved | • Bstorm | T206144 Set NFS load notifications to use a timespan-based approach |
labstore1007 is the Cloud VPS NFS partner. It gets high load whenever someone does a cp on a very large dump over NFS, basically. labstore1006 will do the same thing if exposed to VPS (currently, it's NFS for the stats servers, but not Cloud VPS).
This is essentially totally expected. Adding the monitoring task as a related task.
Mentioned in SAL (#wikimedia-operations) [2018-12-05T10:17:52Z] <arturo> T205969 icinga downtime the load avg check in labstore1007 for 1 week
This week we got several pages related to this. I just downtimed the server for 1 week.
Mentioned in SAL (#wikimedia-operations) [2018-12-12T12:35:27Z] <arturo> T205969 icinga downtime load-avg check for labstore1007 until January (1 month)
High load is now fairly normal for NFS since the upgrades for Spectre and Meltdown (and others). It is an email-only alert at this point because it is very hard to distinguish a real problem using just a load number like we used to. If load is extremely high for a long time, it might suggest a problem, but otherwise, I think it is not worth it to try to stop the load from being over 20 on a regular basis when it doesn't even slow down the system's web server.
Not saying this is fixed. I'm just thinking it's a "can't-fix" and we should get used to high load as more of a possible warning than a real problem.