| ayounsi | |
| Jul 15 2019, 6:20 PM |
| F29800469: 2019-07-18-145259_722x329_scrot.png | |
| Jul 18 2019, 12:54 PM |
| F29774900: swift retransmits.png | |
| Jul 15 2019, 6:20 PM |
See https://grafana.wikimedia.org/d/000000366/network-performances-global?panelId=18&fullscreen&orgId=1&from=now-30d&to=now and screenshot bellow:
Since 18/06, the Swift servers are showing a significant increase of TCP retransmits, from < 0.1% to >3%.
Even though it's not a high %, it could be the symptoms of a larger issue and have a performance impact.
If we narrow it down to ms-fe* hosts they regularly spike between 5% and 15% which is a bit more worrying.
https://grafana.wikimedia.org/d/SxmTH3IZk/arzhels-playground?orgId=1&panelId=3&fullscreen&from=now-30d&to=now (temporary dashboard)
Another theory is that Selective ACK has been disabled on that day on the ms-fe hosts (and is still disabled). /cc @MoritzMuehlenhoff
The same thing started to happen around the same time for labstore1007: https://grafana.wikimedia.org/d/SxmTH3IZk/arzhels-playground?orgId=1&panelId=2&fullscreen&from=now-30d&to=now (temporary dashboard)
Looking at packet capture all retransmits look like regular https traffic, that's what made me think of SACK.
net.ipv4.tcp_sack = 0 on that host as well.
Mentioned in SAL (#wikimedia-operations) [2019-07-16T10:29:37Z] <moritzm> rebooting ms-fe1005 to pick up kernel with SACK fixed (T228086)
The effect is pretty visible for ms-be1005 on https://grafana.wikimedia.org/d/SxmTH3IZk/arzhels-playground?orgId=1&panelId=3&fullscreen&from=now-1h&to=now ; I'll also reboot the other frontends.
Mentioned in SAL (#wikimedia-operations) [2019-07-16T11:12:16Z] <moritzm> rebooting remaining swift frontends in eqiad to pick up a kernel with SACK fixed (T228086)
I've also rebooted the remaining frontends, but with some more data it doesn't actually seem as if this is caused by the disabled SACKs, if e.g. one limits the dashboard to "stat1005" (the blank period is where the server was depooled for the reboot), it seems as spiky as before: https://grafana.wikimedia.org/d/SxmTH3IZk/arzhels-playground?orgId=1&panelId=3&fullscreen&from=now-3h&to=now
Not sure if relevant or not, but cluster wmcs also shows elevated retransmits around the same period:
From a cursory look on both ms-fe2005 and ms-fe1005 with tshark -NNnt -i any -Y 'tcp.analysis.retransmission' it looks like burst of retransmissions when talking to thumbor hosts, and sometimes on localhost too
2041469 81.000654601 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 3441 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39410214 Ack=9351 Win=65536 Len=3373 TSval=46103297 TSecr=1208757373 2041470 81.000659299 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 4412 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39413587 Ack=9351 Win=65536 Len=4344 TSval=46103297 TSecr=1208757373 2041474 81.000718263 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 2964 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39417931 Ack=9351 Win=65536 Len=2896 TSval=46103297 TSecr=1208757373 2041479 81.000793941 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 9233 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39420827 Ack=9351 Win=65536 Len=9165 TSval=46103297 TSecr=1208757373 2041480 81.000802213 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 4412 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39429992 Ack=9351 Win=65536 Len=4344 TSval=46103297 TSecr=1208757373 2041484 81.000870533 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 12129 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39434336 Ack=9351 Win=65536 Len=12061 TSval=46103297 TSecr=1208757373 2041485 81.000877614 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 1516 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39446397 Ack=9351 Win=65536 Len=1448 TSval=46103297 TSecr=1208757373 2041488 81.000904565 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 5860 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39447845 Ack=9351 Win=65536 Len=5792 TSval=46103297 TSecr=1208757373 2041491 81.000929362 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 8756 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39453637 Ack=9351 Win=65536 Len=8688 TSval=46103297 TSecr=1208757373 2041498 81.000990643 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 545 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39462325 Ack=9351 Win=65536 Len=477 TSval=46103297 TSecr=1208757373 2041499 81.000995002 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 13100 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39462802 Ack=9351 Win=65536 Len=13032 TSval=46103297 TSecr=1208757373 2041505 81.001025151 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 3441 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39475834 Ack=9351 Win=65536 Len=3373 TSval=46103297 TSecr=1208757373 2041506 81.001030769 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 7308 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39479207 Ack=9351 Win=65536 Len=7240 TSval=46103297 TSecr=1208757373 2041511 81.001050929 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 2964 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39486447 Ack=9351 Win=65536 Len=2896 TSval=46103297 TSecr=1208757373 2041514 81.001097907 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 6337 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39489343 Ack=9351 Win=65536 Len=6269 TSval=46103297 TSecr=1208757373 2041515 81.001104096 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 7308 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39495612 Ack=9351 Win=65536 Len=7240 TSval=46103297 TSecr=1208757373 2041521 81.001163191 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 9233 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39502852 Ack=9351 Win=65536 Len=9165 TSval=46103297 TSecr=1208757373 2041522 81.001170656 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 4412 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39512017 Ack=9351 Win=65536 Len=4344 TSval=46103297 TSecr=1208757373 2041527 81.001207300 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 2964 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39516361 Ack=9351 Win=65536 Len=2896 TSval=46103297 TSecr=1208757373 2041532 81.001230835 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 9233 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39519257 Ack=9351 Win=65536 Len=9165 TSval=46103297 TSecr=1208757373 2041533 81.001239246 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 1516 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39528422 Ack=9351 Win=65536 Len=1448 TSval=46103297 TSecr=1208757373 2041537 81.001277182 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 14548 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39529870 Ack=9351 Win=65536 Len=14480 TSval=46103297 TSecr=1208757373 2041542 81.001335141 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 8756 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39544350 Ack=9351 Win=65536 Len=8688 TSval=46103297 TSecr=1208757373 2041544 81.001352615 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 5860 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39553038 Ack=9351 Win=65536 Len=5792 TSval=46103297 TSecr=1208757373 2041547 81.001408853 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 2470 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39558830 Ack=9351 Win=65536 Len=2402 TSval=46103297 TSecr=1208757373 2041548 81.001414313 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 11652 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39561232 Ack=9351 Win=65536 Len=11584 TSval=46103297 TSecr=1208757373 2041550 81.001443148 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 4889 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39572816 Ack=9351 Win=65536 Len=4821 TSval=46103297 TSecr=1208757374
2176710 86.483963787 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 5343 [TCP Retransmission] 64628 → http(80) [PSH, ACK] Seq=197830 Ack=1 Win=44032 Len=5275 TSval=46104668 TSecr=46104663 2195481 87.107952128 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → 65522 [FIN, ACK] Seq=271932 Ack=806 Win=45568 Len=0 TSval=46104824 TSecr=46104820 2204157 87.395970158 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 42962 [TCP Retransmission] egs(1926) → http(80) [PSH, ACK] Seq=263366 Ack=1 Win=44032 Len=42894 TSval=46104896 TSecr=46104890 2208077 87.619953408 ms-fe1005.eqiad.wmnet → vl1017-eth1.lvs1015.eqiad.wmnet TCP 76 [TCP Retransmission] https(443) → 52238 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=46104952 TSecr=331036462 WS=512 2208928 87.647952967 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → pc-mta-addrmap(2246) [FIN, ACK] Seq=170066 Ack=666 Win=45056 Len=0 TSval=46104959 TSecr=46104954 2209780 87.679964990 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → 65348 [FIN, ACK] Seq=189472 Ack=764 Win=45568 Len=0 TSval=46104967 TSecr=46104963 2210294 87.691947624 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → ms-olap2(2394) [FIN, ACK] Seq=1049041 Ack=768 Win=45568 Len=0 TSval=46104970 TSecr=46104966 2232323 88.339524690 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TLSv1.2 1516 [TCP Fast Retransmission] , Ignored Unknown Record 2233000 88.352140394 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 1516 [TCP Retransmission] https(443) → 60440 [ACK] Seq=42687545 Ack=9982 Win=67584 Len=1448 TSval=46105135 TSecr=1208759211 2234850 88.395963485 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 121 [TCP Retransmission] http(80) → banyan-net(2708) [PSH, ACK] Seq=393606 Ack=521 Win=45056 Len=53 TSval=46105146 TSecr=46105142 2235445 88.415952399 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 121 [TCP Window Full] [TCP Retransmission] http(80) → banyan-net(2708) [PSH, ACK] Seq=786822 Ack=521 Win=45056 Len=53 TSval=46105151 TSecr=46105147 2237490 88.463945175 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 121 [TCP Retransmission] http(80) → banyan-net(2708) [PSH, ACK] Seq=917894 Ack=521 Win=45056 Len=53 TSval=46105163 TSecr=46105158 2259727 90.584037957 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=5045421 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 2259728 90.584053126 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=5110904 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 2259730 90.584065843 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=5176387 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 2259731 90.584241087 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=4979938 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 2259732 90.584079950 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=5241870 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 2259733 90.584093151 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=5307353 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 2259750 90.584343160 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Spurious Retransmission] http(80) → 7944 [ACK] Seq=5503802 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 [TCP segment of a reassembled PDU] 2259755 90.584384778 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Spurious Retransmission] http(80) → 7944 [ACK] Seq=5569285 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 [TCP segment of a reassembled PDU] 2259758 90.584424175 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Spurious Retransmission] http(80) → 7944 [PSH, ACK] Seq=5634768 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 [TCP segment of a reassembled PDU] 2274340 90.975959919 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → 8918 [FIN, ACK] Seq=452058 Ack=716 Win=45568 Len=0 TSval=46105791 TSecr=46105787 2277080 91.035957316 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → 8414 [FIN, ACK] Seq=2110508 Ack=489 Win=45056 Len=0 TSval=46105806 TSecr=46105800
We could try rebooting the Thumbor hosts to the kernel version with the SACK fixes, they are currently running with SACKs disabled.
Mentioned in SAL (#wikimedia-operations) [2019-07-18T14:46:44Z] <godog> roll-restart thumbor in codfw - T228086
Still seeing the same retransmits in codfw after kernel upgrade and roll restart of thumbor, I'll have to stall looking at this at least for the time being though