Page MenuHomePhabricator

Maps-warper instance very slow application start up times, passenger timeout
Closed, ResolvedPublic

Description

After the recent labs reboot due to kernel vulnerability, the maps-warper application at warper.wmflabs.org failed to start.
Running rake -T would take somewhere from 9 to 11 Minutes to run.

I have moved some of the rvm files from the labstore.svc.eqiad.wmnet:/project/maps/home to the /srv partition at /dev/mapper/vd-second--local--disk and now the time to run rake -T takes just over 1 minute.

Further investigation is needed. But I suspect that possibly related to T124133 (Labs instances sometimes freeze) and T122743 "(labs NFS slowness / high load) running things from the home directory is currently much slower and not recommended.
(It runs very quickly with the same setup locally.)

If unable to fix, a new instance would need to be created.

Note that the wikimaps project doesn't have a phabricator project set up yet. https://commons.wikimedia.org/wiki/Commons:Wikimaps

Event Timeline

Chippyy created this task.Jan 23 2016, 7:46 PM
Chippyy updated the task description. (Show Details)
Chippyy raised the priority of this task from to Needs Triage.
Chippyy added a subscriber: Chippyy.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 23 2016, 7:46 PM
Dereckson updated the task description. (Show Details)Jan 23 2016, 7:49 PM
Dereckson set Security to None.
Restricted Application added a project: Discovery. · View Herald TranscriptJan 23 2016, 8:05 PM

I installed a new version of Passenger (Apache mod which runs Rails apps) and bumped up the timeout time.
So it takes 1-2 mins to start the application and once the app is running, it should be okay.

I do need to run Ruby Prof to profile where there is a problem and to go over https://github.com/phusion/passenger/wiki/Debugging-application-startup-problems and hope that the ruby dev ops fairy blesses me with the knowledge of how to understand it all!

Ruby and NFS do not mix well at all, since ruby makes a *lot* of stat calls. rvm doubly so. So putting anything on NFS is going to make everything *really* slow, so highly reccomend putting them on /srv (as you seem to have done!).

I also don't think it's related to the instances freezing - those instances just fully do not allow ssh, so probably unrelated. I'd recommend attaching strace to the process to see how much filesystem calls it's making to see if there's ways to optimize that.

I think Asaf Bartov (@abartov) ran into similar issues and found ways around them, so perhaps he can help

Yurik moved this task from All map-related tasks to Tracking on the Maps board.Feb 7 2016, 9:58 PM
EBernhardson moved this task from Needs triage to Maps on the Discovery board.Feb 11 2016, 11:21 PM
chasemp closed this task as Resolved.May 31 2016, 3:25 PM
chasemp claimed this task.
chasemp added a subscriber: chasemp.

Ruby and NFS do not mix well at all, since ruby makes a *lot* of stat calls. rvm doubly so. So putting anything on NFS is going to make everything *really* slow, so highly reccomend putting them on /srv (as you seem to have done!).