Change 887282 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] codfw1dev: cloudgw: make VIPs use /32 netmask

https://gerrit.wikimedia.org/r/887282

Change 887288 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] eqiad1: cloudgw: make VIPs use /32 netmask

https://gerrit.wikimedia.org/r/887288

Change 887282 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] codfw1dev: cloudgw: make VIPs use /32 netmask

https://gerrit.wikimedia.org/r/887282

Mentioned in SAL (#wikimedia-cloud) [2023-02-07T11:26:42Z] <arturo> [codfw1dev] testing network changes in cloudgw, expect unrealiable network (T295774)

Change 887373 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: introduce additional route for VIPs

https://gerrit.wikimedia.org/r/887373

Change 887373 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: introduce additional route for VIPs

https://gerrit.wikimedia.org/r/887373

Mentioned in SAL (#wikimedia-cloud) [2023-02-08T17:08:31Z] <arturo> changing to cloudgw network setup, make VIPs /32 (T295774)

Change 887811 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: eqiad1: move remaining VIP to /32 netmask

https://gerrit.wikimedia.org/r/887811

Change 887811 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: eqiad1: move remaining VIP to /32 netmask

https://gerrit.wikimedia.org/r/887811

Change 887288 abandoned by Arturo Borrero Gonzalez:

[operations/puppet@production] eqiad1: cloudgw: make VIPs use /32 netmask

Reason:

merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/887373 instead

https://gerrit.wikimedia.org/r/887288

I think they should be converted all to be /32 both on Netbox and on the instances. This will also let the automation know that they are proper VIPs and will prevent it to attach them specifically to any host on Netbox (because as VIPs they can float from one host to another).

I'm not convinced this logic is best on the network/host side.

We have a normal Ethernet segment (vlan) at play, over which ARP should function to discover the link-layer (MAC) address of devices owning a given IP. Using a /32 netmask on the interface tells the system "there are no other devices connected to this vlan".

On the Juniper's we don't have the problem of importing the IPs back into Netbox, which means we don't have this constraint. Ultimately I think the fix to the problem should be changes that mean we don't need the PuppetDB import script (see T347411).

In the meantime, for the cloudgw issue in particular, we should widen the /30 to a /29 on the cloudgw-transport vlans. That will allow ARP to function normally without the workarounds the were required to make things work with the /32.

@cmooney this change would affect a lot of VIPs assigned by puppet all over production so we must check carefully the consequences of any changes. That said I'm happy to evaluate/discuss alternative approaches.
See for example vipexempt , _process_binding_address() and _handle_vip() in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/netbox-extras/+/refs/heads/master/customscripts/interface_automation.py

Change 962614 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/software/netbox-extras@master] Interface automation: skip import of existing int IPs and VIPs

https://gerrit.wikimedia.org/r/962614

@Volans yep thanks.

I created a provisional patch but I agree we need to consider all the cases. I believe from looking through the code the vipexempt and handle_vip parts only relate to when we need to import new IPs to netbox that aren't already present. So my approach was to just skip things that already exist, which I think should be safe, I avoided touching any of the other code for vip processing as indeed it looks fairly complex.

Very much a quick thing to allow for discussion. I've no strong feelings on this one way or other. I guess the main difference I stumbled on is "VIPs" that are on the main ethernet interface/subnet (and should have wider mask technically) versus those on loopback device (where /32 is the correct choice).

Trying to run the import puppetdb script on cloudgw1002 is now a noop, but for cloudgw2002-dev fails with this exception:

Failure	
An exception occurred: MultipleObjectsReturned: get() returned more than one IPAddress -- it returned 2!

Traceback (most recent call last):
  File "/srv/deployment/netbox/current/src/netbox/extras/scripts.py", line 461, in _run_script
    script.output = script.run(data=data, commit=commit)
  File "/srv/deployment/netbox-extras//customscripts/import_server_facts.py", line 70, in run
    messages.extend(self._import_interfaces_for_device(device, net_driver, networking, lldp, False))
  File "/srv/deployment/netbox-extras//customscripts/_common.py", line 750, in _import_interfaces_for_device
    self._assign_ip_to_interface(address, nbiface, networking, iface, is_primary, False)
  File "/srv/deployment/netbox-extras//customscripts/_common.py", line 203, in _assign_ip_to_interface
    ipaddr = IPAddress.objects.get(address=str(address))
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/deployment/netbox/venv/lib/python3.9/site-packages/django/db/models/query.py", line 499, in get
    raise self.model.MultipleObjectsReturned(
ipam.models.ip.IPAddress.MultipleObjectsReturned: get() returned more than one IPAddress -- it returned 2!