The passenger docs about this are pretty clear: in the floss version of passenger, a restart is blocking - that is passenger will wait for all currently spawned workers to stop, and at least a new one to be spawned, before serving a new request. Those requests are going to be queued, but if we have some expensive catalog being built, it means ~ 30 seconds of blocking.

I don't think this is feasible unless we do something along the following lines:

• Make the puppetmaster-backend an LVS service
• Have the puppetmaster frontends connect to that LVS service (with one twist: the frontends should not host a backend as well, since we use LVS-DR)
• depool/restart/pool the single backend

I will nonetheless experiment with restarting passenger on our backends to see what's the effect in our live environment.

Mentioned in SAL (#wikimedia-operations) [2017-07-03T08:58:56Z] <_joe_> restarting the passenger app on puppetmaster1002 for T169493

Mentioned in SAL (#wikimedia-operations) [2017-07-03T09:07:30Z] <_joe_> restarting the passenger app on puppetmasters in codfw serially with a sleep of 3 seconds for T169493

So all my attempts at restarting one or multiple instances of the puppetmaster backends luckily didn't cause any puppet errors, but it takes up to 4 minutes for a restart to take effect. I'm not sure this is acceptable with our current practices.

Given the fact our puppetmasters are so underloaded that we can safely assume they won't get a perf hit unless we create very complex environments.

So I'll decline this ticket for now and get back to it only if performance hits are significant.