Let's make this a configurable boolean option per LVS service.

Change 187346 had a related patch set uploaded (by Mark Bergsma):
Add no-gravity configuration option

https://gerrit.wikimedia.org/r/187346

Patch-For-Review

This way of working with server state variables in PyBal is becoming quite unpleasant. It would probably make sense to rewrite this with a decent state machine at some point.

This changes a lot and is likely to break things. We could do a minimal version instead, which doesn't support the current model of removing/readding servers at all, and ONLY does 0 weight.

FWIW, relevant patch upstream http://archive.linuxvirtualserver.org/html/lvs-devel/2013-06/msg00055.html to support choosing a fallback realserver on weight=0 for sh and optionally hash on client source port

Yes, and that patch only exists in kernel v3.11 and higher. Luckily, even on the LVS balancers that are still precise, we upgraded to 3.13 kernels manually some time back, so we should be good here even if we continue to be lazy about upgrading the LVS clusters to jessie.

That made me go and review all of this, though, and there is one minor issue: the flag for doing smart things on weight=0 exists in kernels 3.11 or higher, but there has never been a new release of the commandline ipvsadm tools which support setting that flag AFAICS. I checked the source just in case, but it should be obvious from the fact that the kernel patch for these new flags was in 2013, and the last upstream ipvsadm software release was in 2011 :/ I double-checked debian's package in case they had locally patched it in, but nope.

So, in addition to the generic weight=0 support in pybal, we probably need to hack up ipvsadm a bit. The patch should be fairly trivial (new C-level flag defines, new commandline arguments to flip them on in the existing flags field). We can do it as a debian/patches/ in a local build here, and then maybe try to get one of the two levels of upstream to update as well, either debian or the ipvs project (although given they haven't released in over 4 years, so who knows on that front?).

looks like the last ipvsadm release is 1.28 from feb 2015 at https://www.kernel.org/pub/linux/utils/kernel/ipvsadm/ not yet packaged in debian tho

I didn't even see the 1.28 package hiding out on kernel.org! It does have the appropriate flag in it. Perhaps we can put this through the debian process and then pick it from unstable/testing/backports instead of locally hacking on it.

hah I was actually wrong, 1.28 is in debian experimental, https://packages.debian.org/source/experimental/ipvsadm see also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775930 I've asked for unstable upload plans

I think at this point, this has fallen down to the nice-to-have category, but may not be worth the effort. The majority of our sessions are fresh (re-)negotiations even with the ipvs sh algorithm in place and we have tons of CPU headroom, so it just hasn't been a big issue that we spike re-negotiation a bit on depools (which aren't that common anyways). We're likely better off at this point investing our effort into things that make it even less of an issue or even get us back to using wrr (such as RFC5077 support).

FWIW ipvsadm 1.28 is in stretch, once all pybal machines are jessie we could backport it to jessie-backports

Change 187346 abandoned by Mark Bergsma:
Add no-gravity configuration option

Reason:
Pybal is being redesigned around FSMs, and it would make a lot of sense and be much more robust to implement this feature on top of that work instead of the current code.

https://gerrit.wikimedia.org/r/187346

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

This ticket seems contingent on our usage of the sh scheduler. How relevant is it when we eventually switch to the mh scheduler? (T263797)