Change Details

On the jessie upload-cache test host (cp1064), we've had some issues in the kernel vm subsystem related to the heavy disk i/o of the varnish :3128 backend's persistent/bigobj caches. The primary symptom (the one that must be fixed) is periodic extremely heavy, short-lived spikes of CPU utilization (sys, as opposed to user/iowait) which seem to be happening in a contended lock in the kswapd threads during something related to cache page eviction. There are numerous ongoing experiments on how to resolve this (vm kernel parameter tuning, switching filesystems, changing varnish storage configuration/rules, possible kernel downgrades, or upgrades with new bugfix patches).

On the jessie upload-cache test host (cp1064), we've had some issues in the kernel vm subsystem related to the heavy disk i/o of the varnish :3128 backend's persistent/bigobj caches. The primary symptom (the one that must be fixed) is periodic extremely heavy, short-lived spikes of CPU utilization (sys, as opposed to user/iowait) which seem to be happening in a contended lock in the kswapd threads during something related to cache page eviction. These temporary spike to loadavg in the hundreds or thousands and pause normal network/disk i/o significantly enough to cause 503 spikes. There are numerous ongoing experiments on how to resolve this (vm kernel parameter tuning, switching filesystems, changing varnish storage configuration/rules, possible kernel downgrades, or upgrades with new bugfix patches).

On the jessie upload-cache test host (cp1064), we've had some issues in the kernel vm subsystem related to the heavy disk i/o of the varnish :3128 backend's persistent/bigobj caches. The primary symptom (the one that must be fixed) is periodic extremely heavy, short-lived spikes of CPU utilization (sys, as opposed to user/iowait) which seem to be happening in a contended lock in the kswapd threads during something related to cache page eviction. These temporary spike to loadavg in the hundreds or thousands and pause normal network/disk i/o significantly enough to cause 503 spikes. There are numerous ongoing experiments on how to resolve this (vm kernel parameter tuning, switching filesystems, changing varnish storage configuration/rules, possible kernel downgrades, or upgrades with new bugfix patches).