We had a strange issue this evening when Daniel tried to re-image contint2002 on the public1-b-codfw vlan.
Symptoms
PXEboot worked, so it seemed DHCP was ok. But during the Debian installer the system was not able to get an IP address. What was strange is that when I ran a "monitor traffic" command on lsw1-b5-codfw, where the host was connected, I could see the normal DHCP message flow working. Or at least it seemed to, we had the normal DISCOVER / OFFER / REQUEST / ACK series of messages, however the host just sent another DISCOVER every time after the ACK, instead of completing the interface configuration and moving on.
I was surprised at this as we had some issues with the DHCP config for the public vlans in testing before migrating hosts to the new switches. Those issues were resolved in T358488, and I did multiple reimage tests on the very same vlan that was having problems today.
Troubleshooting
It took us a while to spot what was going on. Daniel via the serial console dropped to a BusyBox shell and was able to execute udhcpd manually, which showed this:
udhcpc: received DHCP NAK
That seemed quite strange, as at the same time I was running a "monitor" on the switch port and no NAK could be observed, instead we seen the normal message exchange with a final ACK. To troubleshoot further I booted the system from a debian live cd image, so we could run the ISC dhclient and observe the output:
The DHCP process worked, but you can see there are several 'NAKs' prior to the actual ACK that completes the exchange. The multiple packets immediately gave me a clue as to what might be going on.
Problem
When we originally tested this we observed that as well as the connected switch relaying the DHCP packets, every other switch participating in the vlan did also. This duplication of packets was far from ideal, but as the other switches inserted different option-82 strings (with their own hostname, and VXLAN vtep interface listed), the requests were ignored by the install server. So the only DISCOVER that got an answer was the one from the actual top-of-rack, and the DHCP exchange worked successfully.
The multiple NAKs seen today made me think it could be related to the duplicate relay packets, even though we'd tested reimage on this very vlan prior to migration. As a test I disabled the DHCP relay function for private1-b-codfw on all switches apart from the one contint2002 was connected to and ran dhclient again:
No NAKs at all this time. So the fact that multiple switches were relaying packets was causing the problem. The multiple NAKs weren't visible when I was running "monitor" on the switch side, as they were being bridged in the vlan from the other switches on receipt from the install server ('monitor' on a Juniper only catches packets from the actual system's control plane).
Daniel tried the reimage again - with relay still disabled on the other switches - and both PXE and DHCP within the Debian installer worked.
What we know
- The DHCP relay config on all switches participating in the stretched vlan is the cause of the NAK messages
- The NAKs are ignored by ISC's dhclient, and the PXEboot DHCP, but they cause DHCP in the Debian installer to fail
- This didn't happen on the exact same vlan in multiple reimage tests prior to the switch migration
- Reason is probably that at that point the only port in the vlan across all switches was the test host
- No other ports in an 'up' state meant the vlan and irb interface were DOWN on all other devices
- That probably stopped them relaying dhcp packets during our tests
- Now we have several devices in the vlan connected to different switches, so the irb int is UP on multiple switches
- It is unclear if this only affects the public vlans, which use an 'anycast gw' with only the GW VIP on it, or if it also affects the row-wide private vlans, which have a unique per-switch IP on the irb int in addition to the GW address.
- The significant different for the irb ints with a unicast and anycast IP is they use their individual unicast IP to send responses from the install server back to hosts.
- On the public vlans, where the irb only has the anycast GW IP on it, all switches use that same IP to source the responses from.
- That means the NAKs arrive on the host from the same IP as the DHCP OFFER, unlike on the stretched private vlans.
Next steps
It's unclear exactly how to tackle this one. Off the top of my head I suspect we may need to create some kind of firewall filter on the switches to block relay of DHCP packets that have been bridged from remote devices. But not sure exactly where we could place that. We might be able to contact Juniper to ask, but they mostly refuse to assist with configuration issues these days.
What we will definitely need is to get a test server set up in codfw row A or B so we can try to get to the bottom of it. @Papaul do you think that might be possible? Is there any hardware we could use?
Workaround
Until we have it fixed, if anyone needs to reimage a host on one of these vlans, we'll need to manually configure the relay function on just the switch their host is connected to.