The other LVS hosts in codfw seem to be accessible from cumin2002, just not lvs2012 for some reason.

sukhe@cumin2002:~$ ping -4 lvs2011.codfw.wmnet
PING  (10.192.0.29) 56(84) bytes of data.
64 bytes from lvs2011.codfw.wmnet (10.192.0.29): icmp_seq=1 ttl=64 time=0.126 ms

I tried to ping from lvs2012 few hosts in row C and all fails, so I think is the connection with the row C that is misconfigured on the lvs host.

In T336428#8842727, @Volans wrote:

I tried to ping from lvs2012 few hosts in row C and all fails, so I think is the connection with the row C that is misconfigured on the lvs host.

Thanks @Volans! That's an important bit of info re: the debugging. Adding @Papaul and @Jhancock.wm since it's related to codfw.

Definitely odd.

I can ping fine with v6 (default) from cumin2002 as of now:

cmooney@cumin2002:~$ ping lvs2012
PING lvs2012(lvs2012.codfw.wmnet (2620:0:860:102:10:192:16:140)) 56 data bytes
64 bytes from lvs2012.codfw.wmnet (2620:0:860:102:10:192:16:140): icmp_seq=1 ttl=64 time=0.206 ms
64 bytes from lvs2012.codfw.wmnet (2620:0:860:102:10:192:16:140): icmp_seq=2 ttl=64 time=0.291 ms

IPv4 ping fails, however I see the ping requests hitting the host:

cmooney@lvs2012:~$ sudo tcpdump -i eno12399np0 -l -p -nn host 10.192.16.140 and icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eno12399np0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
19:57:03.289310 IP 10.192.32.49 > 10.192.16.140: ICMP echo request, id 28367, seq 29, length 64
19:57:04.313322 IP 10.192.32.49 > 10.192.16.140: ICMP echo request, id 28367, seq 30, length 64

No ping responses are sent however, the box has no ARP entry for the gateway address on the vlan2019 interface:

cmooney@lvs2012:~$ sudo ip neigh show dev vlan2019
10.192.32.123  FAILED
10.192.32.70  FAILED
10.192.32.49  FAILED
10.192.32.67  FAILED
10.192.32.24  FAILED

Looking closer in Netbox/switch config I could see the issue. Vlan2019 was set as the "native" or untagged vlan on the switch. Which meant that host generated packets with vlan tag 2019 were not getting properly processed, nor were outbound ARPs for the lvs2012 vlan2019 interface getting sent by the switch with tag 2019 as required. I changed in Netbox and ran Homer:

Configuration diff for asw-c-codfw.mgmt.codfw.wmnet:

[edit interfaces xe-2/0/45]
-   native-vlan-id 2019;
[edit interfaces xe-2/0/45 unit 0 family ethernet-switching vlan]
-       members [ public1-c-codfw private1-c-codfw ];
+       members [ private1-c-codfw public1-c-codfw ];

Type "yes" to commit, "no" to abort.
> yes

I assumed this would sort things. I can now ping lvs2019's vlan2019 interface from cumin:

cmooney@cumin2002:~$ ping 10.192.33.9
PING 10.192.33.9 (10.192.33.9) 56(84) bytes of data.
64 bytes from 10.192.33.9: icmp_seq=1 ttl=64 time=0.199 ms
64 bytes from 10.192.33.9: icmp_seq=2 ttl=64 time=0.220 ms

And the arp issues are fixed:

cmooney@lvs2012:~$ sudo ip -4 neigh show dev vlan2019
10.192.32.1 lladdr 00:00:5e:00:01:13 STALE
10.192.32.49 lladdr 2c:ea:7f:87:f1:2a STALE
10.192.32.2 lladdr 64:87:88:f2:73:c8 REACHABLE
10.192.32.3 lladdr a8:d0:e5:e3:87:c8 REACHABLE

However I still don't see a ping response being sent out the vlan2019 interface:

cmooney@lvs2012:~$ sudo tcpdump -i vlan2019 -nn -l icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vlan2019, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C
0 packets captured

Routing looks ok, and ping from the vlan2019 interface towards cumin work:

cmooney@lvs2012:~$ ping 10.192.32.49
PING 10.192.32.49 (10.192.32.49) 56(84) bytes of data.
64 bytes from 10.192.32.49: icmp_seq=1 ttl=64 time=0.181 ms
64 bytes from 10.192.32.49: icmp_seq=2 ttl=64 time=0.204 ms
^C
--- 10.192.32.49 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.181/0.192/0.204/0.011 ms

From the primary IP it fails though:

cmooney@lvs2012:~$ ping -I 10.192.16.140 10.192.32.49
PING 10.192.32.49 (10.192.32.49) from 10.192.16.140 : 56(84) bytes of data.
^C
--- 10.192.32.49 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3070ms

Tcpdump when I do that reveals the host is sending out the ping requests ok:

cmooney@lvs2012:~$ sudo tcpdump -l -nn -e -i enp152s0f0np0 icmp 
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp152s0f0np0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:42:45.987576 00:62:0b:cb:55:d0 > 2c:ea:7f:87:f1:2a, ethertype 802.1Q (0x8100), length 102: vlan 2019, p 0, ethertype IPv4 (0x0800), 10.192.16.140 > 10.192.32.49: ICMP echo request, id 24260, seq 135, length 64
20:42:47.011553 00:62:0b:cb:55:d0 > 2c:ea:7f:87:f1:2a, ethertype 802.1Q (0x8100), length 102: vlan 2019, p 0, ethertype IPv4 (0x0800), 10.192.16.140 > 10.192.32.49: ICMP echo request, id 24260, seq 136, length 64

These packets are received on cumin2002, and it replies:

cmooney@cumin2002:~$ sudo tcpdump -i eno1 -l -p -nn host 10.192.16.140
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:44:24.579250 IP 10.192.16.140 > 10.192.32.49: ICMP echo request, id 19571, seq 20, length 64
20:44:24.579285 IP 10.192.32.49 > 10.192.16.140: ICMP echo reply, id 19571, seq 20, length 64
20:44:25.603246 IP 10.192.16.140 > 10.192.32.49: ICMP echo request, id 19571, seq 21, length 64
20:44:25.603278 IP 10.192.32.49 > 10.192.16.140: ICMP echo reply, id 19571, seq 21, length 64

uRPF looks like it should be ok:

cmooney@lvs2012:~$ sudo sysctl -a | grep \\.rp_filter | egrep "all|eno12399np0|enp152s0f0np0|vlan2019"
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.eno12399np0.rp_filter = 0
net.ipv4.conf.enp152s0f0np0.rp_filter = 1
net.ipv4.conf.vlan2019.rp_filter = 0

It's disabled for the primary interface where this traffic is coming in. It's enabled on the vlan2019's physical parent, but not on the sub-interface. I tried changing it on enp152s0f0np0 but that made no difference, however toggling it for "all" seems to make things work:

cmooney@lvs2012:~$ sudo sysctl -w net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.all.rp_filter = 0

cmooney@cumin2002:~$ ping -4 lvs2012
PING  (10.192.16.140) 56(84) bytes of data.
64 bytes from lvs2012.codfw.wmnet (10.192.16.140): icmp_seq=19 ttl=64 time=0.283 ms
64 bytes from lvs2012.codfw.wmnet (10.192.16.140): icmp_seq=20 ttl=64 time=0.213 ms

So not 100% sure what the issue is here. From what I know about the rp_filter settings the previous was ok. Certainly there was an issue with the switch config initially, but this host-side issue needs more checks. I set the global rp_filter back to 1 now as I don't want to leave it off in general, will need to do more deeper analysis tomorrow.

I took a quick look at lvs2012, the server can ping 10.192.16.1 and 10.192.32.1 but the server can not ping 10.192.0.1 and 10.192.48.1 which explain why the server can not reach any server in row A and Row D. The why, i will have to check the fiber connected to nic 2 and nic 4 when i am on site tomorrow.

In T336428#8843550, @Papaul wrote:

I took a quick look at lvs2012, the server can ping 10.192.16.1 and 10.192.32.1 but the server can not ping 10.192.0.1 and 10.192.48.1 which explain why the server can not reach any server in row A and Row D. The why, i will have to check the fiber connected to nic 2 and nic 4 when i am on site tomorrow.

@Papaul I think the connections are ok. Thanks for the heads up, the issue was the same as with vlan2019, vlans were set up as untagged/native vlan on the switch side.

I've fixed that now for vlan2017 on asw-a-codfw xe-7/0/45, and vlan2020 on asw-d-codfw xe-4/0/47 and it looks ok.

PING 10.192.16.1 (10.192.16.1) from 10.192.16.140 : 56(84) bytes of data.
64 bytes from 10.192.16.1: icmp_seq=1 ttl=64 time=0.526 ms
64 bytes from 10.192.16.1: icmp_seq=2 ttl=64 time=0.783 ms
--- 10.192.16.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1012ms
rtt min/avg/max/mdev = 0.526/0.654/0.783/0.128 ms

PING 10.192.0.1 (10.192.0.1) from 10.192.1.8 : 56(84) bytes of data.
64 bytes from 10.192.0.1: icmp_seq=1 ttl=64 time=0.478 ms
64 bytes from 10.192.0.1: icmp_seq=2 ttl=64 time=0.804 ms
--- 10.192.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1022ms
rtt min/avg/max/mdev = 0.478/0.641/0.804/0.163 ms

PING 10.192.32.1 (10.192.32.1) from 10.192.33.9 : 56(84) bytes of data.
64 bytes from 10.192.32.1: icmp_seq=1 ttl=64 time=0.510 ms
64 bytes from 10.192.32.1: icmp_seq=2 ttl=64 time=0.732 ms
--- 10.192.32.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1022ms
rtt min/avg/max/mdev = 0.510/0.621/0.732/0.111 ms

PING 10.192.48.1 (10.192.48.1) from 10.192.49.9 : 56(84) bytes of data.
64 bytes from 10.192.48.1: icmp_seq=1 ttl=64 time=0.474 ms
64 bytes from 10.192.48.1: icmp_seq=2 ttl=64 time=0.543 ms
--- 10.192.48.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1021ms
rtt min/avg/max/mdev = 0.474/0.508/0.543/0.034 ms

PING 208.80.153.1 (208.80.153.1) from 208.80.153.19 : 56(84) bytes of data.
64 bytes from 208.80.153.1: icmp_seq=1 ttl=64 time=0.542 ms
64 bytes from 208.80.153.1: icmp_seq=2 ttl=64 time=0.595 ms
--- 208.80.153.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1021ms
rtt min/avg/max/mdev = 0.542/0.568/0.595/0.026 ms

PING 208.80.153.33 (208.80.153.33) from 208.80.153.56 : 56(84) bytes of data.
64 bytes from 208.80.153.33: icmp_seq=1 ttl=64 time=0.527 ms
64 bytes from 208.80.153.33: icmp_seq=2 ttl=64 time=0.654 ms
--- 208.80.153.33 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1022ms
rtt min/avg/max/mdev = 0.527/0.590/0.654/0.063 ms

PING 208.80.153.65 (208.80.153.65) from 208.80.153.80 : 56(84) bytes of data.
64 bytes from 208.80.153.65: icmp_seq=1 ttl=64 time=0.747 ms
64 bytes from 208.80.153.65: icmp_seq=2 ttl=64 time=0.557 ms
--- 208.80.153.65 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1022ms
rtt min/avg/max/mdev = 0.557/0.652/0.747/0.095 ms

PING 208.80.153.97 (208.80.153.97) from 208.80.153.113 : 56(84) bytes of data.
64 bytes from 208.80.153.97: icmp_seq=1 ttl=64 time=1.57 ms
64 bytes from 208.80.153.97: icmp_seq=2 ttl=64 time=0.769 ms
--- 208.80.153.97 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.769/1.169/1.569/0.400 ms

Ok I think I see what the issue is. Looking at the kernel docs they state that "the max value from conf/{all,interface}/rp_filter is used when doing source validation on the {interface}."

This effectively means that this setting:

net.ipv4.conf.all.rp_filter = 1

Nullifies the per-interface setting on eno12399np0:

net.ipv4.conf.eno12399np0.rp_filter = 0

I guess we need to analyse exactly what we want from uRPF on the LVS nodes. Certainly it should be disabled for "all", and on the primary interface, to allow traffic from any of the other row subnets to the primary IP (e.g. from cumin host).

Not sure whether it should be on for the vlan interfaces themselves or not. As things stand it is on for those (by virtue of the 'all' setting). This also means you can't ping, for instance, the vlan2017 IP from a host on vlan2020. The server rejects the packet arriving on the vlan2017 interface with a source IP from the vlan2020 subnet.

Perhaps a sensible option would be:

net.ipv4.conf.default.rp_filter = 2 # loose mode by default
net.ipv4.conf.all.rp_filter = 0 # not enabled on 'all', allowing per-interface override
net.ipv4.conf.eno12399np0.rp_filter = 0    # disabled on primary int

But I'm not convinced what benefit the uRPF checking brings us here in general. I'm also not sure if this config is something we are deliberately setting? Doesn't seem that way. There are related commands in (strangely named) /etc/sysctl.d/10-ubuntu-defaults.conf and /etc/network/if-up.d/ip.

@BBlack @jbond @ayounsi interested in your thoughts here.

Change 919034 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] O:traffic: Add lvs::kernel_config during insetup to allow reimages

https://gerrit.wikimedia.org/r/919034

Ok I think I see what the issue is

Nice work on the investigation

I'm also not sure if this config is something we are deliberately setting

For LVS hosts its seems like we have two conflicting settings one in the base profile (10-ubuntu_defaults.conf) which enables rp_filter both globally and by default and on specific to lvs host (70-lvs.conf) which disables it both globally and by default. As the later one has a higher priority it is the one that takes affect.

The issues with theses hosts is that they are still in the insetup_noferm or insetup::traffic role and as such dont have the later sysctl rules applied. We could work around this by adding lvs::kernel_config to insetup::traffic

Perhaps a sensible option would be:

Seems like it would be an improvement to what we currently have in lvs::kernel_config but it is a bit stricter so could introduces some unexpected issues

Ah cool John thanks for the explanation.

Seems like it would be an improvement to what we currently have in lvs::kernel_config but it is a bit stricter so could introduces some unexpected issues

I think it's probably safe, but I'm not really convinced it would bring much benefit. So I'd vote to keep things as they are, and merge your patch to make sure the settings are applied earlier in the install role.

In T336428#8844009, @cmooney wrote:

In T336428#8843550, @Papaul wrote:

I took a quick look at lvs2012, the server can ping 10.192.16.1 and 10.192.32.1 but the server can not ping 10.192.0.1 and 10.192.48.1 which explain why the server can not reach any server in row A and Row D. The why, i will have to check the fiber connected to nic 2 and nic 4 when i am on site tomorrow.

@Papaul I think the connections are ok. Thanks for the heads up, the issue was the same as with vlan2019, vlans were set up as untagged/native vlan on the switch side.

I've fixed that now for vlan2017 on asw-a-codfw xe-7/0/45, and vlan2020 on asw-d-codfw xe-4/0/47 and it looks ok.

PING 10.192.16.1 (10.192.16.1) from 10.192.16.140 : 56(84) bytes of data.
64 bytes from 10.192.16.1: icmp_seq=1 ttl=64 time=0.526 ms
64 bytes from 10.192.16.1: icmp_seq=2 ttl=64 time=0.783 ms
--- 10.192.16.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1012ms
rtt min/avg/max/mdev = 0.526/0.654/0.783/0.128 ms

PING 10.192.0.1 (10.192.0.1) from 10.192.1.8 : 56(84) bytes of data.
64 bytes from 10.192.0.1: icmp_seq=1 ttl=64 time=0.478 ms
64 bytes from 10.192.0.1: icmp_seq=2 ttl=64 time=0.804 ms
--- 10.192.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1022ms
rtt min/avg/max/mdev = 0.478/0.641/0.804/0.163 ms

PING 10.192.32.1 (10.192.32.1) from 10.192.33.9 : 56(84) bytes of data.
64 bytes from 10.192.32.1: icmp_seq=1 ttl=64 time=0.510 ms
64 bytes from 10.192.32.1: icmp_seq=2 ttl=64 time=0.732 ms
--- 10.192.32.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1022ms
rtt min/avg/max/mdev = 0.510/0.621/0.732/0.111 ms

PING 10.192.48.1 (10.192.48.1) from 10.192.49.9 : 56(84) bytes of data.
64 bytes from 10.192.48.1: icmp_seq=1 ttl=64 time=0.474 ms
64 bytes from 10.192.48.1: icmp_seq=2 ttl=64 time=0.543 ms
--- 10.192.48.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1021ms
rtt min/avg/max/mdev = 0.474/0.508/0.543/0.034 ms

PING 208.80.153.1 (208.80.153.1) from 208.80.153.19 : 56(84) bytes of data.
64 bytes from 208.80.153.1: icmp_seq=1 ttl=64 time=0.542 ms
64 bytes from 208.80.153.1: icmp_seq=2 ttl=64 time=0.595 ms
--- 208.80.153.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1021ms
rtt min/avg/max/mdev = 0.542/0.568/0.595/0.026 ms

PING 208.80.153.33 (208.80.153.33) from 208.80.153.56 : 56(84) bytes of data.
64 bytes from 208.80.153.33: icmp_seq=1 ttl=64 time=0.527 ms
64 bytes from 208.80.153.33: icmp_seq=2 ttl=64 time=0.654 ms
--- 208.80.153.33 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1022ms
rtt min/avg/max/mdev = 0.527/0.590/0.654/0.063 ms

PING 208.80.153.65 (208.80.153.65) from 208.80.153.80 : 56(84) bytes of data.
64 bytes from 208.80.153.65: icmp_seq=1 ttl=64 time=0.747 ms
64 bytes from 208.80.153.65: icmp_seq=2 ttl=64 time=0.557 ms
--- 208.80.153.65 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1022ms
rtt min/avg/max/mdev = 0.557/0.652/0.747/0.095 ms

PING 208.80.153.97 (208.80.153.97) from 208.80.153.113 : 56(84) bytes of data.
64 bytes from 208.80.153.97: icmp_seq=1 ttl=64 time=1.57 ms
64 bytes from 208.80.153.97: icmp_seq=2 ttl=64 time=0.769 ms
--- 208.80.153.97 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.769/1.169/1.569/0.400 ms

@cmooney thnaks

Change 919034 merged by Jbond:

[operations/puppet@production] O:traffic: Add lvs::kernel_config during insetup to allow reimages

https://gerrit.wikimedia.org/r/919034

Thanks @cmooney and @jbond for the extensive debugging!

Looking at the above discussion, I think I should have mentioned that we have been doing a bunch of LVS reimages in the past few months (bullseye upgrades first and now the new hardware provisioning). So if the issue was indeed related to RPF, we should have observed these on the other hosts that we reimaged, including lvs2011 that we re-reimaged from insetup to lvs::balancer.

I guess I am trying to role out possible causes for it but unless I am missing something, I don't think we need the change to the insetup role to turn off rp_filter as when the host is reimaged to the LVS role, it should be set anyway. In this case, the problem arose when the reimaging started (it couldn't ping lvs2012 to downtime it) and then finally when the reimaging was completed, it failed because lvs2012 was not accessible and it couldn't do the reboot.

I think the more probable cause is the switch issue @cmooney fixed above and while it can't hurt to turn off rp_filter in the insetup role, I am just trying to exclude the various possible causes of this issue and hence this comment.

Change 919358 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] insetup::traffic: properly handle LVS case

https://gerrit.wikimedia.org/r/919358

Change 919358 merged by BBlack:

[operations/puppet@production] insetup::traffic: properly handle LVS case

https://gerrit.wikimedia.org/r/919358

In T336428#8847879, @gerritbot wrote:

Change 919358 merged by BBlack:

[operations/puppet@production] insetup::traffic: properly handle LVS case

https://gerrit.wikimedia.org/r/919358

See my comment post-merge on the patch, I don't think that was needed as there is the special insetup_noferm exactly for that purpose.

In T336428#8844635, @ssingh wrote:

I think the more probable cause is the switch issue @cmooney fixed above and while it can't hurt to turn off rp_filter in the insetup role, I am just trying to exclude the various possible causes of this issue and hence this comment.

I don't have much insight into the exact sequence of events that lead to the problem for this one, and not on others. But it's clear that there is some series of events that can get us into that state, so either way it seems like a good idea to have rpf disabled earlier if we can.

In T336428#8844099, @cmooney wrote:

Ok I think I see what the issue is. Looking at the kernel docs they state that "the max value from conf/{all,interface}/rp_filter is used when doing source validation on the {interface}."

This effectively means that this setting:

net.ipv4.conf.all.rp_filter = 1

Nullifies the per-interface setting on eno12399np0:

net.ipv4.conf.eno12399np0.rp_filter = 0

(...)

Yep, that's a pitfall. See https://wikitech.wikimedia.org/wiki/Incidents/20140203-LVS for a similar occurrence. As long as the LVS boxes don't route traffic, disabling the filter shouldn't be too much of a (security) problem.

@Southparkfan are still have issue here or we can close this task?

Hi @Papaul: I plan to come back to this once we do the LVS next week and see if we have the same issue, otherwise I will close this task. Thanks!