Page MenuHomePhabricator

Swift TCP retransmits increase
Open, HighPublic0 Story Points

Description

See https://grafana.wikimedia.org/d/000000366/network-performances-global?panelId=18&fullscreen&orgId=1&from=now-30d&to=now and screenshot bellow:

Since 18/06, the Swift servers are showing a significant increase of TCP retransmits, from < 0.1% to >3%.

Even though it's not a high %, it could be the symptoms of a larger issue and have a performance impact.

Event Timeline

ayounsi triaged this task as High priority.Jul 15 2019, 6:20 PM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 15 2019, 6:20 PM

If we narrow it down to ms-fe* hosts they regularly spike between 5% and 15% which is a bit more worrying.
https://grafana.wikimedia.org/d/SxmTH3IZk/arzhels-playground?orgId=1&panelId=3&fullscreen&from=now-30d&to=now (temporary dashboard)

Another theory is that Selective ACK has been disabled on that day on the ms-fe hosts (and is still disabled). /cc @MoritzMuehlenhoff

The same thing started to happen around the same time for labstore1007: https://grafana.wikimedia.org/d/SxmTH3IZk/arzhels-playground?orgId=1&panelId=2&fullscreen&from=now-30d&to=now (temporary dashboard)

Looking at packet capture all retransmits look like regular https traffic, that's what made me think of SACK.
net.ipv4.tcp_sack = 0 on that host as well.

Mentioned in SAL (#wikimedia-operations) [2019-07-16T10:29:37Z] <moritzm> rebooting ms-fe1005 to pick up kernel with SACK fixed (T228086)

jijiki added a subscriber: jijiki.Jul 16 2019, 10:31 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-16T11:12:16Z] <moritzm> rebooting remaining swift frontends in eqiad to pick up a kernel with SACK fixed (T228086)

I've also rebooted the remaining frontends, but with some more data it doesn't actually seem as if this is caused by the disabled SACKs, if e.g. one limits the dashboard to "stat1005" (the blank period is where the server was depooled for the reboot), it seems as spiky as before: https://grafana.wikimedia.org/d/SxmTH3IZk/arzhels-playground?orgId=1&panelId=3&fullscreen&from=now-3h&to=now

Not sure if relevant or not, but cluster wmcs also shows elevated retransmits around the same period:

From a cursory look on both ms-fe2005 and ms-fe1005 with tshark -NNnt -i any -Y 'tcp.analysis.retransmission' it looks like burst of retransmissions when talking to thumbor hosts, and sometimes on localhost too

2041469 81.000654601 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 3441 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39410214 Ack=9351 Win=65536 Len=3373 TSval=46103297 TSecr=1208757373
2041470 81.000659299 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 4412 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39413587 Ack=9351 Win=65536 Len=4344 TSval=46103297 TSecr=1208757373
2041474 81.000718263 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 2964 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39417931 Ack=9351 Win=65536 Len=2896 TSval=46103297 TSecr=1208757373
2041479 81.000793941 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 9233 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39420827 Ack=9351 Win=65536 Len=9165 TSval=46103297 TSecr=1208757373
2041480 81.000802213 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 4412 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39429992 Ack=9351 Win=65536 Len=4344 TSval=46103297 TSecr=1208757373
2041484 81.000870533 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 12129 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39434336 Ack=9351 Win=65536 Len=12061 TSval=46103297 TSecr=1208757373
2041485 81.000877614 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 1516 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39446397 Ack=9351 Win=65536 Len=1448 TSval=46103297 TSecr=1208757373
2041488 81.000904565 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 5860 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39447845 Ack=9351 Win=65536 Len=5792 TSval=46103297 TSecr=1208757373
2041491 81.000929362 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 8756 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39453637 Ack=9351 Win=65536 Len=8688 TSval=46103297 TSecr=1208757373
2041498 81.000990643 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 545 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39462325 Ack=9351 Win=65536 Len=477 TSval=46103297 TSecr=1208757373
2041499 81.000995002 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 13100 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39462802 Ack=9351 Win=65536 Len=13032 TSval=46103297 TSecr=1208757373
2041505 81.001025151 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 3441 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39475834 Ack=9351 Win=65536 Len=3373 TSval=46103297 TSecr=1208757373
2041506 81.001030769 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 7308 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39479207 Ack=9351 Win=65536 Len=7240 TSval=46103297 TSecr=1208757373
2041511 81.001050929 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 2964 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39486447 Ack=9351 Win=65536 Len=2896 TSval=46103297 TSecr=1208757373
2041514 81.001097907 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 6337 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39489343 Ack=9351 Win=65536 Len=6269 TSval=46103297 TSecr=1208757373
2041515 81.001104096 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 7308 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39495612 Ack=9351 Win=65536 Len=7240 TSval=46103297 TSecr=1208757373
2041521 81.001163191 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 9233 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39502852 Ack=9351 Win=65536 Len=9165 TSval=46103297 TSecr=1208757373
2041522 81.001170656 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 4412 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39512017 Ack=9351 Win=65536 Len=4344 TSval=46103297 TSecr=1208757373
2041527 81.001207300 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 2964 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39516361 Ack=9351 Win=65536 Len=2896 TSval=46103297 TSecr=1208757373
2041532 81.001230835 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 9233 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39519257 Ack=9351 Win=65536 Len=9165 TSval=46103297 TSecr=1208757373
2041533 81.001239246 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 1516 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39528422 Ack=9351 Win=65536 Len=1448 TSval=46103297 TSecr=1208757373
2041537 81.001277182 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 14548 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39529870 Ack=9351 Win=65536 Len=14480 TSval=46103297 TSecr=1208757373
2041542 81.001335141 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 8756 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39544350 Ack=9351 Win=65536 Len=8688 TSval=46103297 TSecr=1208757373
2041544 81.001352615 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 5860 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39553038 Ack=9351 Win=65536 Len=5792 TSval=46103297 TSecr=1208757373
2041547 81.001408853 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 2470 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39558830 Ack=9351 Win=65536 Len=2402 TSval=46103297 TSecr=1208757373
2041548 81.001414313 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 11652 [TCP Retransmission] https(443) → 60440 [ACK] Seq=39561232 Ack=9351 Win=65536 Len=11584 TSval=46103297 TSecr=1208757373
2041550 81.001443148 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 4889 [TCP Retransmission] https(443) → 60440 [PSH, ACK] Seq=39572816 Ack=9351 Win=65536 Len=4821 TSval=46103297 TSecr=1208757374
2176710 86.483963787 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 5343 [TCP Retransmission] 64628 → http(80) [PSH, ACK] Seq=197830 Ack=1 Win=44032 Len=5275 TSval=46104668 TSecr=46104663
2195481 87.107952128 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → 65522 [FIN, ACK] Seq=271932 Ack=806 Win=45568 Len=0 TSval=46104824 TSecr=46104820
2204157 87.395970158 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 42962 [TCP Retransmission] egs(1926) → http(80) [PSH, ACK] Seq=263366 Ack=1 Win=44032 Len=42894 TSval=46104896 TSecr=46104890
2208077 87.619953408 ms-fe1005.eqiad.wmnet → vl1017-eth1.lvs1015.eqiad.wmnet TCP 76 [TCP Retransmission] https(443) → 52238 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=46104952 TSecr=331036462 WS=512
2208928 87.647952967 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → pc-mta-addrmap(2246) [FIN, ACK] Seq=170066 Ack=666 Win=45056 Len=0 TSval=46104959 TSecr=46104954
2209780 87.679964990 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → 65348 [FIN, ACK] Seq=189472 Ack=764 Win=45568 Len=0 TSval=46104967 TSecr=46104963
2210294 87.691947624 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → ms-olap2(2394) [FIN, ACK] Seq=1049041 Ack=768 Win=45568 Len=0 TSval=46104970 TSecr=46104966
2232323 88.339524690 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TLSv1.2 1516 [TCP Fast Retransmission] , Ignored Unknown Record
2233000 88.352140394 ms-fe.svc.eqiad.wmnet → thumbor1003.eqiad.wmnet TCP 1516 [TCP Retransmission] https(443) → 60440 [ACK] Seq=42687545 Ack=9982 Win=67584 Len=1448 TSval=46105135 TSecr=1208759211
2234850 88.395963485 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 121 [TCP Retransmission] http(80) → banyan-net(2708) [PSH, ACK] Seq=393606 Ack=521 Win=45056 Len=53 TSval=46105146 TSecr=46105142
2235445 88.415952399 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 121 [TCP Window Full] [TCP Retransmission] http(80) → banyan-net(2708) [PSH, ACK] Seq=786822 Ack=521 Win=45056 Len=53 TSval=46105151 TSecr=46105147
2237490 88.463945175 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 121 [TCP Retransmission] http(80) → banyan-net(2708) [PSH, ACK] Seq=917894 Ack=521 Win=45056 Len=53 TSval=46105163 TSecr=46105158
2259727 90.584037957 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=5045421 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693
2259728 90.584053126 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=5110904 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693
2259730 90.584065843 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=5176387 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693
2259731 90.584241087 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=4979938 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693
2259732 90.584079950 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=5241870 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693
2259733 90.584093151 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Retransmission] http(80) → 7944 [ACK] Seq=5307353 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693
2259750 90.584343160 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Spurious Retransmission] http(80) → 7944 [ACK] Seq=5503802 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 [TCP segment of a reassembled PDU]
2259755 90.584384778 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Spurious Retransmission] http(80) → 7944 [ACK] Seq=5569285 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 [TCP segment of a reassembled PDU]
2259758 90.584424175 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 65551 [TCP Spurious Retransmission] http(80) → 7944 [PSH, ACK] Seq=5634768 Ack=793 Win=45568 Len=65483 TSval=46105693 TSecr=46105693 [TCP segment of a reassembled PDU]
2274340 90.975959919 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → 8918 [FIN, ACK] Seq=452058 Ack=716 Win=45568 Len=0 TSval=46105791 TSecr=46105787
2277080 91.035957316 ms-fe1005.eqiad.wmnet → ms-fe1005.eqiad.wmnet TCP 68 [TCP Retransmission] http(80) → 8414 [FIN, ACK] Seq=2110508 Ack=489 Win=45056 Len=0 TSval=46105806 TSecr=46105800

We could try rebooting the Thumbor hosts to the kernel version with the SACK fixes, they are currently running with SACKs disabled.

Mentioned in SAL (#wikimedia-operations) [2019-07-18T14:46:44Z] <godog> roll-restart thumbor in codfw - T228086

Not sure if relevant or not, but cluster wmcs also shows elevated retransmits around the same period:

Yup, T228086#5334741

Still seeing the same retransmits in codfw after kernel upgrade and roll restart of thumbor, I'll have to stall looking at this at least for the time being though