We have observed a bug with Nokia SR-Linux that affects ARP resolution from directly connected hosts. The specific problem is that the switch does not transmit an ARP reply to requests from the host. If the switch needs to ARP for the host it will do so, the host will reply, and both sides will have a working arp-cache for a time. But if the ARP is initiated from the host towards an irb0 sub-interface on the switch the request goes unanswered.
We have an open case with Nokia on this matter and have been trying to work through it with their engineers.
Problem Detail
We have a network-instance configured for the private1-d3-eqiad vlan:
A:lsw1-d3-eqiad# info flat network-instance vlan-1079 set / network-instance vlan-1079 type mac-vrf set / network-instance vlan-1079 admin-state enable set / network-instance vlan-1079 description private1-d3-eqiad set / network-instance vlan-1079 interface ethernet-1/48.1079 set / network-instance vlan-1079 interface irb0.1079
We have a local irb0 sub-interface configured to act as the gateway for hosts in this vlan:
set / interface irb0 subinterface 1079 admin-state enable set / interface irb0 subinterface 1079 ipv4 admin-state enable set / interface irb0 subinterface 1079 ipv4 address 10.64.181.1/24 set / interface irb0 subinterface 1079 ipv4 arp timeout 600 set / interface irb0 subinterface 1079 ipv4 arp learn-unsolicited true set / interface irb0 subinterface 1079 ipv4 arp host-route populate dynamic set / interface irb0 subinterface 1079 ipv4 arp evpn advertise dynamic
set / network-instance vlan-1079 interface irb0.1079 set / network-instance PRODUCTION interface irb0.1079
And a sub-interface on a port connected to a server configured which is also part of vlan-1079:
set / interface ethernet-1/48 subinterface 1079 type bridged set / interface ethernet-1/48 subinterface 1079 admin-state enable set / interface ethernet-1/48 subinterface 1079 vlan encap single-tagged vlan-id 1079
The switch does learn this MAC address from the host:
A:lsw1-d3-eqiad# show network-instance vlan-1079 bridge-table mac-table all ------------------------------------------------------------------------------------------------------------------- Mac-table of network instance vlan-1079 ------------------------------------------------------------------------------------------------------------------- +--------------------+---------------------+------------+----------------+---------+--------+---------------------+ | Address | Destination | Dest Index | Type | Active | Aging | Last Update | +====================+=====================+============+================+=========+========+=====================+ | A8:E5:EC:78:4F:3C | irb-interface | 0 | irb-interface | true | N/A | 2025-11-04T12:04:38 | | B0:26:28:47:5F:A1 | ethernet-1/48.1079 | 3 | learnt | true | 1193 | 2025-11-04T12:05:30 | +--------------------+---------------------+------------+----------------+---------+--------+---------------------+
On the host we have an IP configured statically:
root@sretest1006:~# ip -4 -br addr show dev ens2f1np1.1079 ens2f1np1.1079@if4 UP 10.64.181.187/24
The switch is able to ARP and get an answer from the host, populating its own table:
A:lsw1-d3-eqiad# show arpnd arp-entries interface irb0 subinterface 1079 +-----------+-------------------+-----------------+---------+--------------------+--------------------+ | Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry | +===========+===================+=================+=========+====================+====================+ | irb0 | 1079 | 10.64.181.187 | dynamic | B0:26:28:47:5F:A1 | 4 minutes from now | +-----------+-------------------+-----------------+---------+--------------------+--------------------+
Our test server has no ARP entry for the irb0.1079 IP though:
root@sretest1006:~# ip neigh show fe80::aae5:ecff:fe78:4f3c dev ens2f1np1.1079 lladdr a8:e5:ec:78:4f:3c router STALE
When we try to ping the gateway/ARP we see the outgoing ARP requests in a tcpdump, but receive no replies:
cmooney@sretest1006:~$ sudo tcpdump -c 4 -e -i ens2f1np1 -l -p -nn vlan 1079 listening on ens2f1np1, link-type EN10MB (Ethernet), snapshot length 262144 bytes 11:30:36.254952 b0:26:28:47:5f:a1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 62: vlan 1079, p 0, ethertype ARP (0x0806), Request who-has 10.64.181.1 tell 255.255.255.255, length 44 11:30:37.256129 b0:26:28:47:5f:a1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 62: vlan 1079, p 0, ethertype ARP (0x0806), Request who-has 10.64.181.1 tell 255.255.255.255, length 44
We've replicated this on different hardware devices on servers, so it does not seem to be a faulty unit (see P74895).
Affected versions
Through trial-end-error we have been able to confirm the problem does not occur with SR-Linux v24.7.2, but it does occur with version v24.10.1, so the bug was presumably introduced in that release. I have checked through the release notes for v24.10.1 and while there are many changes nothing jumps out as a major adjustment to ARP or neighbor discovery.
Options
The obvious thing to do would be to run version v24.7.2 for now while Nokia work through the bug. However there are some obstacles to doing this:
- OpenConfig support is not present in v24.7.2
- Without this our gnmi openconfig stats collection is broken, which we depend on for graphs and BGP alerting
- The upcoming release v25.10.1 is also likely to have this bug, but it contains several new features we need:
Codfw tests
It remains to be seen if this issue will also present itself in codfw, where we have Nokia leaf switches too. The difference there is the switches are not running EVPN as well, however this problem seems local between host and switch, so there is no reason to assume it's related to enabling EVPN. When we can get a test host reimaged successfully in codfw we will test this as high priority.
The good news is that the switches in codfw are for new servers, where we should be able to select UEFI mode and not rely on DHCP option 82. We also have no "stetched" layer-2 public vlan in those rows, so we do not need the RFC3527 support. What we would still be missing is the OpenConfig support, but perhaps we could use Nokia-native paths instead to collect data and alert.