Since around 14:20-14:30 UTC all of my k8s jobs have started to fail due to not being able to connect to en.wikipedia.org.
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='en.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7eaf221d30>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))This was suspiciously paired with T330165: eqiad row B switches upgrade, which got cloudvirt1019/1020 off the network during the operation.
They are hypervisors with local storage VMs, which include Toolforge kubernetes etcd servers.
We discovered that 2/3 of etcd being down resulted in calico-node getting somehow confused, see
| 1 | 2023-03-28 14:12:39.715 [INFO][67] felix/summary.go 100: Summarising 9 dataplane reconciliation loops over 1m3.3s: avg=39ms longest=172ms (resync-nat-v4) |
|---|---|
| 2 | 2023-03-28 14:13:03.835 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 3 | 2023-03-28 14:13:43.066 [INFO][67] felix/summary.go 100: Summarising 10 dataplane reconciliation loops over 1m3.4s: avg=9ms longest=33ms (resync-routes-v4,resync-routes-v4,resync-rules-v4,resync-wg) |
| 4 | 2023-03-28 14:14:03.838 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 5 | 2023-03-28 14:14:46.909 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m3.8s: avg=33ms longest=160ms (resync-nat-v4) |
| 6 | 2023-03-28 14:15:03.842 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 7 | 2023-03-28 14:15:49.139 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m2.2s: avg=40ms longest=206ms (resync-nat-v4) |
| 8 | 2023-03-28 14:16:03.846 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 9 | 2023-03-28 14:16:52.194 [INFO][67] felix/summary.go 100: Summarising 9 dataplane reconciliation loops over 1m3.1s: avg=13ms longest=52ms (resync-routes-v4,resync-routes-v4,resync-rules-v4,resync-wg) |
| 10 | 2023-03-28 14:17:03.848 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 11 | 2023-03-28 14:17:55.498 [INFO][67] felix/summary.go 100: Summarising 10 dataplane reconciliation loops over 1m3.3s: avg=37ms longest=166ms (resync-nat-v4) |
| 12 | 2023-03-28 14:18:03.851 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 13 | 2023-03-28 14:18:58.709 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m3.2s: avg=34ms longest=170ms (resync-nat-v4) |
| 14 | 2023-03-28 14:19:03.856 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 15 | 2023-03-28 14:19:24.273 [INFO][67] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=85125 ifaceName="cali356f440101e" state="down" |
| 16 | 2023-03-28 14:19:24.273 [INFO][67] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"cali356f440101e", State:"down", Index:85125} |
| 17 | 2023-03-28 14:19:24.274 [INFO][67] felix/endpoint_mgr.go 418: Workload interface state changed; marking for status update. ifaceName="cali356f440101e" |
| 18 | 2023-03-28 14:19:24.274 [INFO][67] felix/endpoint_mgr.go 477: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"tool-glamtools/inventory-scan-27993790-fthqk", EndpointId:"eth0"} |
| 19 | 2023-03-28 14:19:24.274 [INFO][67] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"tool-glamtools/inventory-scan-27993790-fthqk", EndpointId:"eth0"} |
| 20 | 2023-03-28 14:19:24.274 [INFO][67] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"tool-glamtools/inventory-scan-27993790-fthqk", EndpointId:"eth0"} ipVersion=0x4 status="down" |
| 21 | 2023-03-28 14:19:24.275 [INFO][67] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"tool-glamtools/inventory-scan-27993790-fthqk", EndpointId:"eth0"} status="down" |
| 22 | 2023-03-28 14:19:24.276 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="cali356f440101e" |
| 23 | 2023-03-28 14:19:24.276 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"cali356f440101e", Addrs:set.mapSet{}} |
| 24 | 2023-03-28 14:19:24.276 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"cali356f440101e", Addrs:set.mapSet{}} |
| 25 | 2023-03-28 14:19:24.276 [INFO][67] felix/iface_monitor.go 201: Netlink address update. addr="fe80::ecee:eeff:feee:eeee" exists=false ifIndex=85125 |
| 26 | 2023-03-28 14:19:24.293 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=<nil> ifaceName="cali356f440101e" |
| 27 | 2023-03-28 14:19:24.293 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"cali356f440101e", Addrs:set.Set(nil)} |
| 28 | 2023-03-28 14:19:24.293 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"cali356f440101e", Addrs:set.Set(nil)} |
| 29 | 2023-03-28 14:20:01.854 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m3.1s: avg=6ms longest=24ms (resync-filter-v4) |
| 30 | 2023-03-28 14:20:03.859 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 31 | 2023-03-28 14:20:26.368 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="calib8b5df99f66" |
| 32 | 2023-03-28 14:20:26.368 [INFO][67] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=101030 ifaceName="calib8b5df99f66" state="up" |
| 33 | 2023-03-28 14:20:26.368 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calib8b5df99f66", Addrs:set.mapSet{}} |
| 34 | 2023-03-28 14:20:26.368 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calib8b5df99f66", Addrs:set.mapSet{}} |
| 35 | 2023-03-28 14:20:26.369 [INFO][67] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"calib8b5df99f66", State:"up", Index:101030} |
| 36 | 2023-03-28 14:20:26.369 [INFO][67] felix/endpoint_mgr.go 366: Workload interface came up, marking for reconfiguration. ifaceName="calib8b5df99f66" |
| 37 | 2023-03-28 14:20:26.370 [INFO][67] felix/endpoint_mgr.go 1179: Applying /proc/sys configuration to interface. ifaceName="calib8b5df99f66" |
| 38 | 2023-03-28 14:20:26.372 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 39 | 2023-03-28 14:20:26.372 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 40 | 2023-03-28 14:20:26.372 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 41 | 2023-03-28 14:20:28.113 [INFO][67] felix/iface_monitor.go 201: Netlink address update. addr="fe80::ecee:eeff:feee:eeee" exists=true ifIndex=101030 |
| 42 | 2023-03-28 14:20:28.114 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}} ifaceName="calib8b5df99f66" |
| 43 | 2023-03-28 14:20:28.114 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calib8b5df99f66", Addrs:set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}}} |
| 44 | 2023-03-28 14:20:28.114 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calib8b5df99f66", Addrs:set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}}} |
| 45 | 2023-03-28 14:20:28.115 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 46 | 2023-03-28 14:20:28.116 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 47 | 2023-03-28 14:20:28.116 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 48 | 2023-03-28 14:20:30.471 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="calif1b08e6e3d7" |
| 49 | 2023-03-28 14:20:30.472 [INFO][67] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=101031 ifaceName="calif1b08e6e3d7" state="up" |
| 50 | 2023-03-28 14:20:30.490 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{}} |
| 51 | 2023-03-28 14:20:30.490 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{}} |
| 52 | 2023-03-28 14:20:30.494 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 53 | 2023-03-28 14:20:30.496 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 54 | 2023-03-28 14:20:30.496 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 55 | 2023-03-28 14:20:30.496 [INFO][67] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"calif1b08e6e3d7", State:"up", Index:101031} |
| 56 | 2023-03-28 14:20:30.497 [INFO][67] felix/endpoint_mgr.go 366: Workload interface came up, marking for reconfiguration. ifaceName="calif1b08e6e3d7" |
| 57 | 2023-03-28 14:20:30.497 [INFO][67] felix/endpoint_mgr.go 1179: Applying /proc/sys configuration to interface. ifaceName="calif1b08e6e3d7" |
| 58 | 2023-03-28 14:20:30.500 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 59 | 2023-03-28 14:20:30.503 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 60 | 2023-03-28 14:20:30.511 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 61 | 2023-03-28 14:20:30.512 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 62 | 2023-03-28 14:20:30.512 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 63 | 2023-03-28 14:20:30.512 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 64 | 2023-03-28 14:20:31.566 [INFO][67] felix/iface_monitor.go 201: Netlink address update. addr="fe80::ecee:eeff:feee:eeee" exists=true ifIndex=101031 |
| 65 | 2023-03-28 14:20:31.566 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}} ifaceName="calif1b08e6e3d7" |
| 66 | 2023-03-28 14:20:31.566 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}}} |
| 67 | 2023-03-28 14:20:31.566 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}}} |
| 68 | 2023-03-28 14:20:31.568 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 69 | 2023-03-28 14:20:31.568 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 70 | 2023-03-28 14:20:31.569 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 71 | 2023-03-28 14:20:31.569 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 72 | 2023-03-28 14:20:31.570 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 73 | 2023-03-28 14:20:31.570 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 74 | 2023-03-28 14:20:33.792 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 75 | 2023-03-28 14:20:33.793 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 76 | 2023-03-28 14:20:33.794 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 77 | 2023-03-28 14:20:33.794 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 78 | 2023-03-28 14:20:33.795 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 79 | 2023-03-28 14:20:33.796 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 80 | 2023-03-28 14:20:43.927 [INFO][67] felix/route_table.go 896: Remove old route dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 routeProblems=[]string{"unexpected route"} tableIndex=0 |
| 81 | 2023-03-28 14:20:43.928 [INFO][67] felix/conntrack.go 90: Removing conntrack flows ip=192.168.222.147 |
| 82 | 2023-03-28 14:20:43.928 [INFO][67] felix/route_table.go 896: Remove old route dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 routeProblems=[]string{"unexpected route"} tableIndex=0 |
| 83 | 2023-03-28 14:20:43.928 [INFO][67] felix/conntrack.go 90: Removing conntrack flows ip=192.168.222.151 |
| 84 | 2023-03-28 14:21:03.863 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 85 | 2023-03-28 14:21:04.412 [INFO][67] felix/summary.go 100: Summarising 17 dataplane reconciliation loops over 1m2.6s: avg=24ms longest=181ms (resync-nat-v4) |
| 86 | 2023-03-28 14:21:13.096 [INFO][67] felix/iface_monitor.go 201: Netlink address update. addr="fe80::ecee:eeff:feee:eeee" exists=false ifIndex=101031 |
| 87 | 2023-03-28 14:21:13.096 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="calif1b08e6e3d7" |
| 88 | 2023-03-28 14:21:13.096 [INFO][67] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=101031 ifaceName="calif1b08e6e3d7" state="down" |
| 89 | 2023-03-28 14:21:13.097 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{}} |
| 90 | 2023-03-28 14:21:13.097 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{}} |
| 91 | 2023-03-28 14:21:13.097 [INFO][67] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"calif1b08e6e3d7", State:"down", Index:101031} |
| 92 | 2023-03-28 14:21:13.132 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=<nil> ifaceName="calif1b08e6e3d7" |
| 93 | 2023-03-28 14:21:13.133 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.Set(nil)} |
| 94 | 2023-03-28 14:21:13.133 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.Set(nil)} |
| 95 | 2023-03-28 14:22:03.865 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 96 | 2023-03-28 14:22:08.200 [INFO][67] felix/summary.go 100: Summarising 14 dataplane reconciliation loops over 1m3.8s: avg=24ms longest=171ms (resync-nat-v4) |
| 97 | 2023-03-28 14:23:03.868 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 98 | 2023-03-28 14:23:09.769 [INFO][67] felix/summary.go 100: Summarising 8 dataplane reconciliation loops over 1m1.6s: avg=30ms longest=176ms (resync-nat-v4) |
| 99 | 2023-03-28 14:24:03.870 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 100 | 2023-03-28 14:24:14.402 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m4.6s: avg=20ms longest=130ms () |
| 101 | 2023-03-28 14:25:03.872 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 102 | 2023-03-28 14:25:16.025 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m1.6s: avg=39ms longest=215ms (resync-nat-v4) |
| 103 | 2023-03-28 14:26:03.875 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 104 | 2023-03-28 14:26:21.225 [INFO][67] felix/summary.go 100: Summarising 10 dataplane reconciliation loops over 1m5.2s: avg=22ms longest=141ms (resync-nat-v4) |
| 105 | 2023-03-28 14:27:03.877 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 106 | 2023-03-28 14:27:23.505 [INFO][67] felix/summary.go 100: Summarising 9 dataplane reconciliation loops over 1m2.3s: avg=22ms longest=120ms () |
| 107 | 2023-03-28 14:28:03.880 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
| 108 | 2023-03-28 14:28:05.946 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="calia22ceccf57c" |
| 109 | 2023-03-28 14:28:05.947 [INFO][67] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=101032 ifaceName="calia22ceccf57c" state="up" |
| 110 | 2023-03-28 14:28:05.947 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calia22ceccf57c", Addrs:set.mapSet{}} |
| 111 | 2023-03-28 14:28:05.947 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calia22ceccf57c", Addrs:set.mapSet{}} |
| 112 | 2023-03-28 14:28:05.947 [INFO][67] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"calia22ceccf57c", State:"up", Index:101032} |
| 113 | 2023-03-28 14:28:05.948 [INFO][67] felix/endpoint_mgr.go 366: Workload interface came up, marking for reconfiguration. ifaceName="calia22ceccf57c" |
| 114 | 2023-03-28 14:28:05.948 [INFO][67] felix/endpoint_mgr.go 1179: Applying /proc/sys configuration to interface. ifaceName="calia22ceccf57c" |
| 115 | 2023-03-28 14:28:05.949 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.181/32 ifaceName="calia22ceccf57c" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 116 | 2023-03-28 14:28:05.950 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.181/32 ifaceName="calia22ceccf57c" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
| 117 | 2023-03-28 14:28:05.950 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calia22ceccf57c" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
2023-03-28 14:20:43.928 [INFO][67] felix/conntrack.go 90: Removing conntrack flows ip=192.168.222.147
2023-03-28 14:20:43.928 [INFO][67] felix/route_table.go 896: Remove old route dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 routeProblems=[]string{"unexpected route"} tableIndex=0
2023-03-28 14:20:43.928 [INFO][67] felix/conntrack.go 90: Removing conntrack flows ip=192.168.222.151If local worker node conntrack NAT information was flushed by calico, the it is expected that network flows would be affected, explaining the outage.