I was looking at creating a bunch of k8s workers using "soft anti-affinity" instead of typical anti-affinity because workers should schedule wherever possible, but with as great a spread as available, right? Also we have that policy exposed in Horizon.
That said, it's not quite configured apparently. I got this error message:
ServerGroupSoftAntiAffinityWeigher not configured
Seems both solvable and a nifty feature to unlock for Toolforge and friends in the future.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | • Bstorm | T246122 Upgrade the Toolforge Kubernetes cluster to v1.16 | |||
| Resolved | • Bstorm | T211096 PAWS: Rebuild and upgrade Kubernetes | |||
| Resolved | • Bstorm | T253267 Configure the soft anti-affinity (and presumably the soft affinity) server policy |
I briefly looked into this.
The docs at https://docs.openstack.org/python-openstackclient/rocky/cli/command-objects/server-group.html read Specify –os-compute-api-version 2.15 or higher for the ‘soft-affinity’ or ‘soft-anti-affinity’ policy.
So it seems we need a compute API with version >= 2.15, but apparently we have:
| eqiad1-r | nova | compute | True | public | http://openstack.eqiad1.wikimediacloud.org:8774/v2.1
@Andrew can you take a quick look to see if we can disable the UI feature as it appears our control plane does not support this yet?
Change 607825 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Nova: enable soft affinity (and soft anti-affinity) server groups
api versions are weird in nova... 2.1 is the version but there are later 'microversions' that can be requested specifically via http headers. We support up to microversion 2.65. It looks to me like horizon is properly requesting a higher microversion, so we shouldn't have API issues with using this feature.
You can turn scheduling filters on and off in nova.conf, and it looks like the soft affinity filters are currently turned off. Attached patch should resolve that.
Change 607825 merged by Andrew Bogott:
[operations/puppet@production] Openstack Nova: enable soft affinity (and soft anti-affinity) server groups
Change 607827 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Nova: enable soft affinity (and soft anti-affinity) server groups
Change 607827 merged by Andrew Bogott:
[operations/puppet@production] Openstack Nova: enable soft affinity (and soft anti-affinity) server groups
@Bstorm try now? I did a quick test and it seems to be working (at least with 3 VMs it put them on three different hosts.)
Mentioned in SAL (#wikimedia-cloud) [2020-06-25T22:52:04Z] <bstorm> created paws-k8s-worker-5/6/7 as x-large nodes to bring the cluster up to roughly the same capacity as the existing one using soft anti-affinity T211096 T253267
Worked for my new workers. I presume that it will also place them together when the capacity gets more spread tighter (thus the "soft"), which is great. I'm imagining soft anti-affinity is ideal for worker node-type things and might deprecate the spread monitor.