Page MenuHomePhabricator

Add support for setting weight=0 when depooling
Open, HighPublic

Description

We currently use the LVS source hash (sh) scheduler for HTTPS traffic, as a way to keep client connected to specific HTTPS terminators and avoid renegotiating their TLS session parameters. However, this means that currently a a penalty is incured for all users whenever a server is depooled (all IPs are reshuffled).

To address this, we need to switch our depooling method from removing the server from the pool entirely, to keeping it in the pool but setting it at weight 0, which is explicitly supported by LVS. Pybal needs to explicitly support this.

Event Timeline

faidon assigned this task to BBlack.
faidon raised the priority of this task from to High.
faidon updated the task description. (Show Details)
faidon added subscribers: Aklapper, faidon, mark.

Let's make this a configurable boolean option per LVS service.

gerritbot added a subscriber: gerritbot.

Change 187346 had a related patch set uploaded (by Mark Bergsma):
Add no-gravity configuration option

https://gerrit.wikimedia.org/r/187346

Patch-For-Review

This way of working with server state variables in PyBal is becoming quite unpleasant. It would probably make sense to rewrite this with a decent state machine at some point.

This changes a lot and is likely to break things. We could do a minimal version instead, which doesn't support the current model of removing/readding servers at all, and ONLY does 0 weight.

FWIW, relevant patch upstream http://archive.linuxvirtualserver.org/html/lvs-devel/2013-06/msg00055.html to support choosing a fallback realserver on weight=0 for sh and optionally hash on client source port

Yes, and that patch only exists in kernel v3.11 and higher. Luckily, even on the LVS balancers that are still precise, we upgraded to 3.13 kernels manually some time back, so we should be good here even if we continue to be lazy about upgrading the LVS clusters to jessie.

That made me go and review all of this, though, and there is one minor issue: the flag for doing smart things on weight=0 exists in kernels 3.11 or higher, but there has never been a new release of the commandline ipvsadm tools which support setting that flag AFAICS. I checked the source just in case, but it should be obvious from the fact that the kernel patch for these new flags was in 2013, and the last upstream ipvsadm software release was in 2011 :/ I double-checked debian's package in case they had locally patched it in, but nope.

So, in addition to the generic weight=0 support in pybal, we probably need to hack up ipvsadm a bit. The patch should be fairly trivial (new C-level flag defines, new commandline arguments to flip them on in the existing flags field). We can do it as a debian/patches/ in a local build here, and then maybe try to get one of the two levels of upstream to update as well, either debian or the ipvs project (although given they haven't released in over 4 years, so who knows on that front?).

looks like the last ipvsadm release is 1.28 from feb 2015 at https://www.kernel.org/pub/linux/utils/kernel/ipvsadm/ not yet packaged in debian tho

I didn't even see the 1.28 package hiding out on kernel.org! It does have the appropriate flag in it. Perhaps we can put this through the debian process and then pick it from unstable/testing/backports instead of locally hacking on it.

hah I was actually wrong, 1.28 is in debian experimental, https://packages.debian.org/source/experimental/ipvsadm see also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775930 I've asked for unstable upload plans

BBlack removed BBlack as the assignee of this task.Aug 7 2015, 5:16 PM
BBlack lowered the priority of this task from High to Low.

I think at this point, this has fallen down to the nice-to-have category, but may not be worth the effort. The majority of our sessions are fresh (re-)negotiations even with the ipvs sh algorithm in place and we have tons of CPU headroom, so it just hasn't been a big issue that we spike re-negotiation a bit on depools (which aren't that common anyways). We're likely better off at this point investing our effort into things that make it even less of an issue or even get us back to using wrr (such as RFC5077 support).

FWIW ipvsadm 1.28 is in stretch, once all pybal machines are jessie we could backport it to jessie-backports

ema raised the priority of this task from Low to High.Jul 28 2017, 4:19 PM

Change 187346 abandoned by Mark Bergsma:
Add no-gravity configuration option

Reason:
Pybal is being redesigned around FSMs, and it would make a lot of sense and be much more robust to implement this feature on top of that work instead of the current code.

https://gerrit.wikimedia.org/r/187346