Page MenuHomePhabricator

Allow RB to be programmatically pooled/depooled during restarts
Closed, ResolvedPublic

Description

During deploys and restarts, RESTBase can take up to 20 seconds to start up and start listening to incoming connections. Given the rate of requests received by the services, there is a big chance that LVS will send a portion of the volume to a node that is in the start-up process, which will end up in connections being refused. We need a way to programmatically depool a given server before restarting RESTBase there as well as a way to repool it back once RB starts accepting connections.

Event Timeline

@mobrovac which user on the rb-systems should be able to repool/depool the server itself?

Once I have that info, it's a 10 minutes job.

@mobrovac which user on the rb-systems should be able to repool/depool the server itself?

The only user we use there is the restbase user, which runs the service and doesn't have shell access (shell=/bin/false). Perhaps we ought to create a user specific to that purpose? Otherwise, using restbase should be ok too for the time being(TM).

Change 304033 had a related patch set uploaded (by Giuseppe Lavagetto):
restbase: allow pooling/depooling programmatically

https://gerrit.wikimedia.org/r/304033

Change 304033 merged by Giuseppe Lavagetto:
restbase: allow pooling/depooling programmatically

https://gerrit.wikimedia.org/r/304033

mobrovac claimed this task.

With the above patch merged and PR #99 for our ansible script repo, this is now fully functional. Resolving.