Page MenuHomePhabricator

PHP 5.5 sessionclean cron job hanging on tool labs bastions
Closed, ResolvedPublic

Description

The buggy sessionclean cron task described in T73645: php5 session cleanup script can go nuts is now having problems on tools-bastion-02.tools.eqiad.wmflabs and tools-bastion-05.tools.eqiad.wmflabs. This script runs every half hour and uses lsof to check to see if a session file is active before cleaning it up.

Yesterday I caught 60+ copies of the script running on each trusty bastion. I manually killed them and now less than 24 hours later there are 40+ stuck again. The magic command to kill them off is ps ax|grep lsof|grep -v grep|awk '{print $1}'|xargs sudo kill.

The fix applied for MediaWiki-Vagrant in T73645 was to backport a replacement script from the upstream Debian 5.5 package via Puppet. The most current PHP 5.5 cleaning script from upstream is https://anonscm.debian.org/cgit/pkg-php/php.git/tree/debian/sessionclean?h=master-5.5&id=64d94baca49c60041d02902c4ba7d564a17a3dbf

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript

Have the new packages Nish Aravamudan referred to in https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/comments/42 been tested in Labs? It would be nice not to manage local patches.

To my knowledge, no that PPA build has not been tested.

Rather than maintain a patch to the deb, I would suggest a small bit of puppet code to replace the /etc/cron.d/php5 file packaged by Ubuntu with a new crontab entry that executes a copy of the upstream sessionclean script. The net maintenance burden should be 2 Puppet file:// resources and the associated puppet code to ensure that they are deployed after the php5-common package has been provisioned. This can all basically be lifted from the https://gerrit.wikimedia.org/r/#/c/164877 patch to mediawiki-vagrant.

Change 289973 had a related patch set uploaded (by BryanDavis):
toollabs: Replace trusty PHP5 session cleanup script

https://gerrit.wikimedia.org/r/289973

Change 289973 merged by Ori.livneh:
toollabs: Replace trusty PHP5 session cleanup script

https://gerrit.wikimedia.org/r/289973

Mentioned in SAL [2016-05-20T23:39:15Z] <bd808> Forced puppet run on bastion-02 & bastion-05 to apply fix for T135861