Since around 14:20-14:30 UTC all of my k8s jobs have started to fail due to not being able to connect to en.wikipedia.org.
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='en.wikipedia.org', port=443): Max retries exceeded with url: /w/api.php (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7eaf221d30>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
This was suspiciously paired with T330165: eqiad row B switches upgrade, which got cloudvirt1019/1020 off the network during the operation.
They are hypervisors with local storage VMs, which include Toolforge kubernetes etcd servers.
We discovered that 2/3 of etcd being down resulted in calico-node getting somehow confused, see
1 | 2023-03-28 14:12:39.715 [INFO][67] felix/summary.go 100: Summarising 9 dataplane reconciliation loops over 1m3.3s: avg=39ms longest=172ms (resync-nat-v4) |
---|---|
2 | 2023-03-28 14:13:03.835 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
3 | 2023-03-28 14:13:43.066 [INFO][67] felix/summary.go 100: Summarising 10 dataplane reconciliation loops over 1m3.4s: avg=9ms longest=33ms (resync-routes-v4,resync-routes-v4,resync-rules-v4,resync-wg) |
4 | 2023-03-28 14:14:03.838 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
5 | 2023-03-28 14:14:46.909 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m3.8s: avg=33ms longest=160ms (resync-nat-v4) |
6 | 2023-03-28 14:15:03.842 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
7 | 2023-03-28 14:15:49.139 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m2.2s: avg=40ms longest=206ms (resync-nat-v4) |
8 | 2023-03-28 14:16:03.846 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
9 | 2023-03-28 14:16:52.194 [INFO][67] felix/summary.go 100: Summarising 9 dataplane reconciliation loops over 1m3.1s: avg=13ms longest=52ms (resync-routes-v4,resync-routes-v4,resync-rules-v4,resync-wg) |
10 | 2023-03-28 14:17:03.848 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
11 | 2023-03-28 14:17:55.498 [INFO][67] felix/summary.go 100: Summarising 10 dataplane reconciliation loops over 1m3.3s: avg=37ms longest=166ms (resync-nat-v4) |
12 | 2023-03-28 14:18:03.851 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
13 | 2023-03-28 14:18:58.709 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m3.2s: avg=34ms longest=170ms (resync-nat-v4) |
14 | 2023-03-28 14:19:03.856 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
15 | 2023-03-28 14:19:24.273 [INFO][67] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=85125 ifaceName="cali356f440101e" state="down" |
16 | 2023-03-28 14:19:24.273 [INFO][67] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"cali356f440101e", State:"down", Index:85125} |
17 | 2023-03-28 14:19:24.274 [INFO][67] felix/endpoint_mgr.go 418: Workload interface state changed; marking for status update. ifaceName="cali356f440101e" |
18 | 2023-03-28 14:19:24.274 [INFO][67] felix/endpoint_mgr.go 477: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"tool-glamtools/inventory-scan-27993790-fthqk", EndpointId:"eth0"} |
19 | 2023-03-28 14:19:24.274 [INFO][67] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"tool-glamtools/inventory-scan-27993790-fthqk", EndpointId:"eth0"} |
20 | 2023-03-28 14:19:24.274 [INFO][67] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"tool-glamtools/inventory-scan-27993790-fthqk", EndpointId:"eth0"} ipVersion=0x4 status="down" |
21 | 2023-03-28 14:19:24.275 [INFO][67] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"tool-glamtools/inventory-scan-27993790-fthqk", EndpointId:"eth0"} status="down" |
22 | 2023-03-28 14:19:24.276 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="cali356f440101e" |
23 | 2023-03-28 14:19:24.276 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"cali356f440101e", Addrs:set.mapSet{}} |
24 | 2023-03-28 14:19:24.276 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"cali356f440101e", Addrs:set.mapSet{}} |
25 | 2023-03-28 14:19:24.276 [INFO][67] felix/iface_monitor.go 201: Netlink address update. addr="fe80::ecee:eeff:feee:eeee" exists=false ifIndex=85125 |
26 | 2023-03-28 14:19:24.293 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=<nil> ifaceName="cali356f440101e" |
27 | 2023-03-28 14:19:24.293 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"cali356f440101e", Addrs:set.Set(nil)} |
28 | 2023-03-28 14:19:24.293 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"cali356f440101e", Addrs:set.Set(nil)} |
29 | 2023-03-28 14:20:01.854 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m3.1s: avg=6ms longest=24ms (resync-filter-v4) |
30 | 2023-03-28 14:20:03.859 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
31 | 2023-03-28 14:20:26.368 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="calib8b5df99f66" |
32 | 2023-03-28 14:20:26.368 [INFO][67] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=101030 ifaceName="calib8b5df99f66" state="up" |
33 | 2023-03-28 14:20:26.368 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calib8b5df99f66", Addrs:set.mapSet{}} |
34 | 2023-03-28 14:20:26.368 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calib8b5df99f66", Addrs:set.mapSet{}} |
35 | 2023-03-28 14:20:26.369 [INFO][67] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"calib8b5df99f66", State:"up", Index:101030} |
36 | 2023-03-28 14:20:26.369 [INFO][67] felix/endpoint_mgr.go 366: Workload interface came up, marking for reconfiguration. ifaceName="calib8b5df99f66" |
37 | 2023-03-28 14:20:26.370 [INFO][67] felix/endpoint_mgr.go 1179: Applying /proc/sys configuration to interface. ifaceName="calib8b5df99f66" |
38 | 2023-03-28 14:20:26.372 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
39 | 2023-03-28 14:20:26.372 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
40 | 2023-03-28 14:20:26.372 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
41 | 2023-03-28 14:20:28.113 [INFO][67] felix/iface_monitor.go 201: Netlink address update. addr="fe80::ecee:eeff:feee:eeee" exists=true ifIndex=101030 |
42 | 2023-03-28 14:20:28.114 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}} ifaceName="calib8b5df99f66" |
43 | 2023-03-28 14:20:28.114 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calib8b5df99f66", Addrs:set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}}} |
44 | 2023-03-28 14:20:28.114 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calib8b5df99f66", Addrs:set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}}} |
45 | 2023-03-28 14:20:28.115 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
46 | 2023-03-28 14:20:28.116 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
47 | 2023-03-28 14:20:28.116 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
48 | 2023-03-28 14:20:30.471 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="calif1b08e6e3d7" |
49 | 2023-03-28 14:20:30.472 [INFO][67] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=101031 ifaceName="calif1b08e6e3d7" state="up" |
50 | 2023-03-28 14:20:30.490 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{}} |
51 | 2023-03-28 14:20:30.490 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{}} |
52 | 2023-03-28 14:20:30.494 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
53 | 2023-03-28 14:20:30.496 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
54 | 2023-03-28 14:20:30.496 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
55 | 2023-03-28 14:20:30.496 [INFO][67] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"calif1b08e6e3d7", State:"up", Index:101031} |
56 | 2023-03-28 14:20:30.497 [INFO][67] felix/endpoint_mgr.go 366: Workload interface came up, marking for reconfiguration. ifaceName="calif1b08e6e3d7" |
57 | 2023-03-28 14:20:30.497 [INFO][67] felix/endpoint_mgr.go 1179: Applying /proc/sys configuration to interface. ifaceName="calif1b08e6e3d7" |
58 | 2023-03-28 14:20:30.500 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
59 | 2023-03-28 14:20:30.503 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
60 | 2023-03-28 14:20:30.511 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
61 | 2023-03-28 14:20:30.512 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
62 | 2023-03-28 14:20:30.512 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
63 | 2023-03-28 14:20:30.512 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
64 | 2023-03-28 14:20:31.566 [INFO][67] felix/iface_monitor.go 201: Netlink address update. addr="fe80::ecee:eeff:feee:eeee" exists=true ifIndex=101031 |
65 | 2023-03-28 14:20:31.566 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}} ifaceName="calif1b08e6e3d7" |
66 | 2023-03-28 14:20:31.566 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}}} |
67 | 2023-03-28 14:20:31.566 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{"fe80::ecee:eeff:feee:eeee":set.empty{}}} |
68 | 2023-03-28 14:20:31.568 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
69 | 2023-03-28 14:20:31.568 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
70 | 2023-03-28 14:20:31.569 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
71 | 2023-03-28 14:20:31.569 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
72 | 2023-03-28 14:20:31.570 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
73 | 2023-03-28 14:20:31.570 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
74 | 2023-03-28 14:20:33.792 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
75 | 2023-03-28 14:20:33.793 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
76 | 2023-03-28 14:20:33.794 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
77 | 2023-03-28 14:20:33.794 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
78 | 2023-03-28 14:20:33.795 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
79 | 2023-03-28 14:20:33.796 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
80 | 2023-03-28 14:20:43.927 [INFO][67] felix/route_table.go 896: Remove old route dest=192.168.222.147/32 ifaceName="calif1b08e6e3d7" ifaceRegex="^cali.*" ipVersion=0x4 routeProblems=[]string{"unexpected route"} tableIndex=0 |
81 | 2023-03-28 14:20:43.928 [INFO][67] felix/conntrack.go 90: Removing conntrack flows ip=192.168.222.147 |
82 | 2023-03-28 14:20:43.928 [INFO][67] felix/route_table.go 896: Remove old route dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 routeProblems=[]string{"unexpected route"} tableIndex=0 |
83 | 2023-03-28 14:20:43.928 [INFO][67] felix/conntrack.go 90: Removing conntrack flows ip=192.168.222.151 |
84 | 2023-03-28 14:21:03.863 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
85 | 2023-03-28 14:21:04.412 [INFO][67] felix/summary.go 100: Summarising 17 dataplane reconciliation loops over 1m2.6s: avg=24ms longest=181ms (resync-nat-v4) |
86 | 2023-03-28 14:21:13.096 [INFO][67] felix/iface_monitor.go 201: Netlink address update. addr="fe80::ecee:eeff:feee:eeee" exists=false ifIndex=101031 |
87 | 2023-03-28 14:21:13.096 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="calif1b08e6e3d7" |
88 | 2023-03-28 14:21:13.096 [INFO][67] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=101031 ifaceName="calif1b08e6e3d7" state="down" |
89 | 2023-03-28 14:21:13.097 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{}} |
90 | 2023-03-28 14:21:13.097 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.mapSet{}} |
91 | 2023-03-28 14:21:13.097 [INFO][67] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"calif1b08e6e3d7", State:"down", Index:101031} |
92 | 2023-03-28 14:21:13.132 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=<nil> ifaceName="calif1b08e6e3d7" |
93 | 2023-03-28 14:21:13.133 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.Set(nil)} |
94 | 2023-03-28 14:21:13.133 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calif1b08e6e3d7", Addrs:set.Set(nil)} |
95 | 2023-03-28 14:22:03.865 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
96 | 2023-03-28 14:22:08.200 [INFO][67] felix/summary.go 100: Summarising 14 dataplane reconciliation loops over 1m3.8s: avg=24ms longest=171ms (resync-nat-v4) |
97 | 2023-03-28 14:23:03.868 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
98 | 2023-03-28 14:23:09.769 [INFO][67] felix/summary.go 100: Summarising 8 dataplane reconciliation loops over 1m1.6s: avg=30ms longest=176ms (resync-nat-v4) |
99 | 2023-03-28 14:24:03.870 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
100 | 2023-03-28 14:24:14.402 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m4.6s: avg=20ms longest=130ms () |
101 | 2023-03-28 14:25:03.872 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
102 | 2023-03-28 14:25:16.025 [INFO][67] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m1.6s: avg=39ms longest=215ms (resync-nat-v4) |
103 | 2023-03-28 14:26:03.875 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
104 | 2023-03-28 14:26:21.225 [INFO][67] felix/summary.go 100: Summarising 10 dataplane reconciliation loops over 1m5.2s: avg=22ms longest=141ms (resync-nat-v4) |
105 | 2023-03-28 14:27:03.877 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
106 | 2023-03-28 14:27:23.505 [INFO][67] felix/summary.go 100: Summarising 9 dataplane reconciliation loops over 1m2.3s: avg=22ms longest=120ms () |
107 | 2023-03-28 14:28:03.880 [INFO][63] monitor-addresses/startup.go 713: Using autodetected IPv4 address on interface eth0: 172.16.2.47/21 |
108 | 2023-03-28 14:28:05.946 [INFO][67] felix/int_dataplane.go 1071: Linux interface addrs changed. addrs=set.mapSet{} ifaceName="calia22ceccf57c" |
109 | 2023-03-28 14:28:05.947 [INFO][67] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=101032 ifaceName="calia22ceccf57c" state="up" |
110 | 2023-03-28 14:28:05.947 [INFO][67] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calia22ceccf57c", Addrs:set.mapSet{}} |
111 | 2023-03-28 14:28:05.947 [INFO][67] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calia22ceccf57c", Addrs:set.mapSet{}} |
112 | 2023-03-28 14:28:05.947 [INFO][67] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"calia22ceccf57c", State:"up", Index:101032} |
113 | 2023-03-28 14:28:05.948 [INFO][67] felix/endpoint_mgr.go 366: Workload interface came up, marking for reconfiguration. ifaceName="calia22ceccf57c" |
114 | 2023-03-28 14:28:05.948 [INFO][67] felix/endpoint_mgr.go 1179: Applying /proc/sys configuration to interface. ifaceName="calia22ceccf57c" |
115 | 2023-03-28 14:28:05.949 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.181/32 ifaceName="calia22ceccf57c" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
116 | 2023-03-28 14:28:05.950 [INFO][67] felix/route_table.go 892: Syncing routes: found unexpected route; ignoring due to grace period. dest=192.168.222.181/32 ifaceName="calia22ceccf57c" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
117 | 2023-03-28 14:28:05.950 [INFO][67] felix/route_table.go 567: Interface in cleanup grace period, will retry after. ifaceName="calia22ceccf57c" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0 |
2023-03-28 14:20:43.928 [INFO][67] felix/conntrack.go 90: Removing conntrack flows ip=192.168.222.147 2023-03-28 14:20:43.928 [INFO][67] felix/route_table.go 896: Remove old route dest=192.168.222.151/32 ifaceName="calib8b5df99f66" ifaceRegex="^cali.*" ipVersion=0x4 routeProblems=[]string{"unexpected route"} tableIndex=0 2023-03-28 14:20:43.928 [INFO][67] felix/conntrack.go 90: Removing conntrack flows ip=192.168.222.151
If local worker node conntrack NAT information was flushed by calico, the it is expected that network flows would be affected, explaining the outage.