Page MenuHomePhabricator

Maps-warper instance very slow application start up times, passenger timeout
Closed, ResolvedPublic

Description

After the recent labs reboot due to kernel vulnerability, the maps-warper application at warper.wmflabs.org failed to start.
Running rake -T would take somewhere from 9 to 11 Minutes to run.

I have moved some of the rvm files from the labstore.svc.eqiad.wmnet:/project/maps/home to the /srv partition at /dev/mapper/vd-second--local--disk and now the time to run rake -T takes just over 1 minute.

Further investigation is needed. But I suspect that possibly related to T124133 (Labs instances sometimes freeze) and T122743 "(labs NFS slowness / high load) running things from the home directory is currently much slower and not recommended.
(It runs very quickly with the same setup locally.)

If unable to fix, a new instance would need to be created.

Note that the wikimaps project doesn't have a phabricator project set up yet. https://commons.wikimedia.org/wiki/Commons:Wikimaps

Event Timeline

Chippyy raised the priority of this task from to Needs Triage.
Chippyy updated the task description. (Show Details)
Chippyy subscribed.
Dereckson set Security to None.

I installed a new version of Passenger (Apache mod which runs Rails apps) and bumped up the timeout time.
So it takes 1-2 mins to start the application and once the app is running, it should be okay.

I do need to run Ruby Prof to profile where there is a problem and to go over https://github.com/phusion/passenger/wiki/Debugging-application-startup-problems and hope that the ruby dev ops fairy blesses me with the knowledge of how to understand it all!

Ruby and NFS do not mix well at all, since ruby makes a *lot* of stat calls. rvm doubly so. So putting anything on NFS is going to make everything *really* slow, so highly reccomend putting them on /srv (as you seem to have done!).

I also don't think it's related to the instances freezing - those instances just fully do not allow ssh, so probably unrelated. I'd recommend attaching strace to the process to see how much filesystem calls it's making to see if there's ways to optimize that.

I think Asaf Bartov (@abartov) ran into similar issues and found ways around them, so perhaps he can help

chasemp claimed this task.
chasemp subscribed.

Ruby and NFS do not mix well at all, since ruby makes a *lot* of stat calls. rvm doubly so. So putting anything on NFS is going to make everything *really* slow, so highly reccomend putting them on /srv (as you seem to have done!).