During deploys and restarts, RESTBase can take up to 20 seconds to start up and start listening to incoming connections. Given the rate of requests received by the services, there is a big chance that LVS will send a portion of the volume to a node that is in the start-up process, which will end up in connections being refused. We need a way to programmatically depool a given server before restarting RESTBase there as well as a way to repool it back once RB starts accepting connections.
Description
Description
Details
Details
Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
restbase: allow pooling/depooling programmatically | operations/puppet | production | +2 -0 |
Related Objects
Related Objects
Event Timeline
Comment Actions
@mobrovac which user on the rb-systems should be able to repool/depool the server itself?
Once I have that info, it's a 10 minutes job.
Comment Actions
The only user we use there is the restbase user, which runs the service and doesn't have shell access (shell=/bin/false). Perhaps we ought to create a user specific to that purpose? Otherwise, using restbase should be ok too for the time being(TM).
Comment Actions
Change 304033 had a related patch set uploaded (by Giuseppe Lavagetto):
restbase: allow pooling/depooling programmatically
Comment Actions
Change 304033 merged by Giuseppe Lavagetto:
restbase: allow pooling/depooling programmatically
Comment Actions
With the above patch merged and PR #99 for our ansible script repo, this is now fully functional. Resolving.