Change #1306672 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] dumps: temp allow production_networks for nfs healthchecks

https://gerrit.wikimedia.org/r/1306672

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306672 is the bandaid I propose for now to get unblocked

All the IPs allowed by LOAD_BALANCER_HEALTH_CHECKS are defined in Hiera, for production under common.yaml in haproxy_allowed_healthcheck_sources. I'm not familiar with the finer details of how the IPs are managed on the LVSes, but it sounds to me that the public IPs of the LVS host are missing in Hiera?

The workaround sounds good to me, +1d.

Change #1306672 merged by Filippo Giunchedi:

[operations/puppet@production] dumps: temp allow production_networks for nfs healthchecks

https://gerrit.wikimedia.org/r/1306672

Change #1306690 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] LVS: add public vlan IPs/subnets for LVS still connected to L2 vlans

https://gerrit.wikimedia.org/r/1306690

@fgiunchedi thanks for the task. I think @MoritzMuehlenhoff is correct, for some unknown reason the public vlan IPs that lvs1019/lvs1020 are using have never been added to the list of allowed health-check sources for LVS.

I guess that's because things normally only go on the public vlans if they are exposing a service directly to the internet. Anything that is behind LVS normally goes on a private vlan instead. So it never came up.

This messy config will thankfully go away once we can get the remaining K8s services moved to IPIP encapsulation (see T352956#11722069), and remove the spiders-web of cables across the datacentre and vlan sub-interfaces to support direct layer-2 adjacency for the low-traffic class.

For now I've made a patch to include the required public vlan IPs the LVS servers who are using them still have.

This service is using LVS with IPIP so I don't understand why health-checks are coming from the public vlan IPs. Maybe something worth checking with Traffic?

ipip_encapsulation:
  - eqiad

lvs1020:~$ sudo -i ipvsadm -Ln |grep -A2 208.80.154.243
TCP  208.80.154.243:2049 mh
  -> 208.80.154.71:2049           Tunnel  100    0          0         
  -> 208.80.154.142:2049          Tunnel  100    0          0

In T430651#12074704, @ayounsi wrote:

This service is using LVS with IPIP so I don't understand why health-checks are coming from the public vlan IPs. Maybe something worth checking with Traffic?

The destination IP for the IPIP packets is on a directly connected interface, so it gets used.

Once we remove the vlan interfaces it'll use the default gateway over the primary int instead.

Change #1306690 merged by Cathal Mooney:

[operations/puppet@production] LVS: add public vlan IPs/subnets for LVS still connected to L2 vlans

https://gerrit.wikimedia.org/r/1306690

I merged the patch to add the lvs public IPs to the healtcheck list now, and the rule is on the clouddumps servers and being used:

cmooney@clouddumps1002:~$ sudo iptables -L -v --line -n | grep 208.80.154.106 
103      5   300 ACCEPT     tcp  --  *      *       208.80.154.106       0.0.0.0/0            tcp dpt:2049

However thinking further about Arzhel's point I think he is correct.

The health checks should be IPIP encapsulated, and come from sources in 172.16.0.0/12

Health check packets from lvs1020 to say clouddumps1002 will indeed be sent directly at layer-2 over vlan1003 (public1-c-eqiad). But as the clouddumps are using IPIP it should be sending IPIP packets over this interface, from a source IP in 172.16.0.0/12. It should not be using its own interface IP for these, so the rule we added shouldn't be needed. This existing rule should allow the IPIP-encapsulated traffic:

cmooney@clouddumps1002:~$ sudo iptables -L -v --line -n | grep 172.16
753    31M 2176M ACCEPT     4    --  *      *       172.16.0.0/12        0.0.0.0/0

lvs1020 appears to be doing health-checks, but directly to the clouddumps IP (not service IP), and not over IPIP

lvs1020 does not appear to be sending any IPIP packets to the host though:

cmooney@clouddumps1002:~$ sudo timeout 900 tcpdump -i enp94s0f0np0 -l -p -nn ether src "e4:3d:1a:51:fa:86" and ip src net 172.16.0.0/12 
listening on enp94s0f0np0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

0 packets captured

Instead we see periodic connections from lvs1020's interface IP, directly to clouddumps main IP (not the service IP):

cmooney@clouddumps1002:~$  sudo tcpdump -c 4 -i enp94s0f0np0 -l -p -nn ip and ether src "e4:3d:1a:51:fa:86" and port 2049
listening on enp94s0f0np0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:36:41.066431 IP 208.80.154.106.56032 > 208.80.154.71.2049: Flags [.], ack 1676371535, win 83, options [nop,nop,TS val 3144368992 ecr 2644264254], length 0
11:37:11.786471 IP 208.80.154.106.56032 > 208.80.154.71.2049: Flags [.], ack 1, win 83, options [nop,nop,TS val 3144399712 ecr 2644294974], length 0
11:37:42.506450 IP 208.80.154.106.56032 > 208.80.154.71.2049: Flags [.], ack 1, win 83, options [nop,nop,TS val 3144430432 ecr 2644325694], length 0

lvs1018 seems to be sending no health checks

If I send traffic to the nfs endpoint, lvs1018 forwards them using IPIP as it should:

14:39:25.000011 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto IPIP (4), length 80)
    172.16.169.76 > 208.80.154.71: IP (tos 0x0, ttl 62, id 64765, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.16.154.47472 > 208.80.154.243.2049: Flags [S], cksum 0x3276 (correct), seq 266343076, win 42340, options [mss 1460,sackOK,TS val 1124516617 ecr 0,nop,wscale 9], length 0

But I didn't observe any packets generated by lvs1018 itself for health checks. It seems it's not sending any?

I think we need traffic to clear up what the expected behaviour is here.

Change #1307111 abandoned by Cathal Mooney:

[operations/puppet@production] Clouddumps fw rules for lvs healthcheck - remove production networks

Reason:

already done in another patch

https://gerrit.wikimedia.org/r/1307111

@Fabfur have you any insight into the above?

On the face of it it looks like we no longer do health-checks? I'm almost certainly wrong about that but it would put my mind at ease if someone from Traffic could explain. Thanks.

In T430651#12109242, @cmooney wrote:

@Fabfur have you any insight into the above?

On the face of it it looks like we no longer do health-checks? I'm almost certainly wrong about that but it would put my mind at ease if someone from Traffic could explain. Thanks.

Thanks for flagging this again, @cmooney. We will be discussing this tomorrow and I will follow up then. Thanks for the patience!