I think I may have tracked down a cause of Trusty VMs (and labs instances) occasionally locking up.

The php5 package shipped with Ubuntu 14.04 includes a cron job to cleanup stale session files stored on disk. A similar cron job has been present in previous versions of Ubuntu, but the version shipped with 14.04 attempts to fix a long standing bug https://bugs.debian.org/626640 by scanning for sessions in active use before purging. To do this, the upstream maintainer decided that running /usr/bin/lsof -w -l +d "/var/lib/php5" would be a great way to find out if any processes had session files open. As pointed out in an upstream bug report https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113 lsof can have a highly variable runtime cost depending on the processes running at the time it is invoked. I have now several times caught multiple versions of the /usr/lib/php5/sessionclean script running on my VM. Since this script is only invoked at :09 and :39 of each hour by cron, if multiple instances are seen this implies that the older version has been running for at least 30 minutes.

The Ubuntu 12.04 version of the cleanup script was inlined in /etc/cron.d/php5:

[ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete

The proposed fix in the upstream Debian package is at http://anonscm.debian.org/cgit/pkg-php/php.git/tree/debian/sessionclean. This fix is not currently scheduled for backporting to Ubuntu 14.04.

I think we should either, 1) revert the cron job to the 12.04 version, 2) backport the Debian fix as part of our Puppet configuration, or 3) remove the clean up job entirely as part of our Puppet configuration.

Since we do not currently set $wgSessionsInObjectCache, we are actively using files to store php sessions. I think this implies that option 3 would be less than optimal.

Version: unspecified
Severity: normal