Page MenuHomePhabricator

Memory leak on ats-tls 8.0.6
Open, HighPublic

Description

Initial symptoms:

vgutierrez@lvs1013:~$ sudo -i journalctl -u pybal --since=today |grep "not pooled" |grep cp1075 |wc -l
135
vgutierrez@lvs1013:~$ sudo -i journalctl -u pybal --since=today |grep "not pooled" |grep cp1081 |wc -l
129

Flapping for unknown reasons

Memory leak reported to upstream in https://github.com/apache/trafficserver/issues/6614

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -1
operations/puppetproduction+397 -0
operations/puppetproduction+1 -12
operations/puppetproduction+15 -0
operations/puppetproduction+6 -0
operations/debs/trafficservermaster+202 -38
operations/puppetproduction+2 -2
operations/debs/trafficservermaster+87 -0
operations/debs/trafficservermaster+239 -0
operations/puppetproduction+2 -8
operations/puppetproduction+6 -6
operations/puppetproduction+6 -0
operations/puppetproduction+14 -0
operations/debs/trafficservermaster+11 -3
operations/puppetproduction+8 -0
operations/debs/trafficservermaster+20 -0
operations/debs/trafficservermaster+2 K -0
operations/puppetproduction+15 -0
operations/debs/trafficservermaster+52 -0
operations/puppetproduction+4 -5
operations/puppetproduction+5 -4
operations/puppetproduction+12 -1
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

image.png (626×689 px, 57 KB)
it looks like the leak is still there

Mentioned in SAL (#wikimedia-operations) [2020-04-10T15:03:08Z] <vgutierrez> restart ats-tls on cp1083 and cp1085 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-11T07:01:40Z] <vgutierrez> restart ats-tls on cp[1079,1081,1083,1085].eqiad.wmnet- T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-11T15:39:45Z] <vgutierrez> restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-12T05:50:15Z] <vgutierrez> restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-12T11:11:34Z] <vgutierrez> restart ats-tls on cp5008.eqsin.wmnet - T249335

Change 588366 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Enable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/588366

Change 588366 merged by Vgutierrez:
[operations/puppet@production] ATS: Enable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/588366

Mentioned in SAL (#wikimedia-operations) [2020-04-13T07:50:06Z] <vgutierrez> enable memory tracking in ats-tls on cp1085 - T249335

res_memory_tracking should help a lot, this is an example report @ cp1085 after restarting ats-tls:

     Allocated      |        In-Use      | Type Size  |   Free List Name
--------------------|--------------------|------------|----------------------------------
                  0 |                  0 |    2097152 | memory/ioBufAllocator[14]
                  0 |                  0 |    1048576 | memory/ioBufAllocator[13]
                  0 |                  0 |     524288 | memory/ioBufAllocator[12]
                  0 |                  0 |     262144 | memory/ioBufAllocator[11]
                  0 |                  0 |     131072 | memory/ioBufAllocator[10]
                  0 |                  0 |      65536 | memory/ioBufAllocator[9]
           18874368 |            8585216 |      32768 | memory/ioBufAllocator[8]
           59244544 |           58851328 |      16384 | memory/ioBufAllocator[7]
            9175040 |            7225344 |       8192 | memory/ioBufAllocator[6]
           42467328 |           41648128 |       4096 | memory/ioBufAllocator[5]
             262144 |                  0 |       2048 | memory/ioBufAllocator[4]
             131072 |                  0 |       1024 | memory/ioBufAllocator[3]
              65536 |                  0 |        512 | memory/ioBufAllocator[2]
            6717440 |            6702592 |        256 | memory/ioBufAllocator[1]
             131072 |             117248 |        128 | memory/ioBufAllocator[0]
             122880 |              73728 |         96 | memory/eventAllocator
            3280320 |            3270480 |         80 | memory/mutexAllocator
            1368064 |            1361152 |         64 | memory/ioBlockAllocator
            2121600 |            2114496 |         48 | memory/ioDataAllocator
            2807040 |            2797920 |        240 | memory/ioAllocator
                  0 |                  0 |        432 | memory/socksAllocator
                  0 |                  0 |        128 | memory/udpReadContAllocator
                  0 |                  0 |        160 | memory/udpPacketAllocator
            1327104 |            1327104 |        864 | memory/netVCAllocator
                  0 |                  0 |        128 | memory/UDPIOEventAllocator
            5898240 |            5795840 |       1024 | memory/sslNetVCAllocator
                  0 |                  0 |         64 | memory/RamCacheLRUEntry
                  0 |                  0 |         96 | memory/RamCacheCLFUSEntry
                  0 |                  0 |        160 | memory/openDirEntry
                  0 |                  0 |         48 | memory/evacuationKey
                  0 |                  0 |         64 | memory/cacheRemoveCont
                  0 |                  0 |         96 | memory/evacuationBlock
                  0 |                  0 |        944 | memory/cacheVConnection
                  0 |                  0 |         16 | memory/DNSRequestDataAllocator
                  0 |                  0 |      29584 | memory/dnsBufAllocator
                  0 |                  0 |       1280 | memory/dnsEntryAllocator
               4096 |                 16 |         16 | memory/expiryQueueEntry
               8192 |                384 |         64 | memory/refCountCacheHashingValueAllocator
                  0 |                  0 |         96 | memory/hostDBFileContAllocator
             294912 |               2304 |       2304 | memory/hostDBContAllocator
                  0 |                  0 |        128 | memory/OneWayTunnelAllocator
           21233664 |           21139456 |       2048 | memory/hdrStrHeap
           29097984 |           28913664 |       2048 | memory/hdrHeap
                  0 |                  0 |        256 | memory/httpCacheAltAllocator
            6045696 |            5982720 |       1312 | memory/http2ClientSessionAllocator
            1382880 |            1357152 |       1072 | memory/http2StreamAllocator
                  0 |                  0 |        128 | memory/RemapPluginsAlloc
             294912 |             294912 |        192 | memory/httpServerSessionAllocator
            2273280 |             737040 |       8880 | memory/httpSMAllocator
            1395680 |            1348832 |        976 | memory/http1ClientSessionAllocator
                  0 |                  0 |        128 | memory/socksProxyAllocator
               4096 |                  0 |         32 | memory/MIMEFieldSDKHandle
                  0 |                  0 |        288 | memory/INKVConnAllocator
              16384 |                640 |        128 | memory/INKContAllocator
               4096 |                128 |         32 | memory/apiHookAllocator
                  0 |                  0 |        512 | memory/FetchSMAllocator
             262144 |              83968 |       1024 | memory/ArenaBlock
          216311808 |          199731792 |            | TOTAL
-----------------------------------------------------------------------------------------
Allocs     | Frees      | Size In-use          | Avg Size   | Location
-----------|------------|----------------------|------------|--------------------------------------------------------------------
     34073 |      34049 |                98304 |       4096 | memory/IOBuffer/UNKNOWN-LOCATION
        64 |         64 |                    0 |          0 | memory/IOBuffer/traffic_server/InkIOCoreAPI.cc:554
      9964 |       9793 |               700416 |       4096 | memory/IOBuffer/Http1ClientSession.cc:208
    280538 |     280255 |              8671232 |      30640 | memory/IOBuffer/HttpSM.cc:6532
      2775 |       2775 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:6258
     91516 |      91515 |                 4096 |       4096 | memory/IOBuffer/HttpSM.cc:5975
      2299 |       2299 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:5611
        69 |         69 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:745
     92817 |      91935 |              7225344 |       8192 | memory/IOBuffer/HttpServerSession.cc:88
        80 |         80 |                    0 |          0 | memory/IOBuffer/HttpTunnel.cc:95
     12178 |      12178 |                    0 |          0 | memory/IOBuffer/HttpTunnel.cc:100
    167600 |     158594 |             36888576 |       4096 | memory/IOBuffer/Http2ClientSession.cc:198
     54660 |      54659 |                 4096 |       4096 | memory/IOBuffer/Http2ClientSession.cc:204
    163612 |     159110 |             58967808 |      13098 | memory/IOBuffer/Http2ClientSession.h:103
         8 |          8 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1048
         8 |          8 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1051
     26049 |      25116 |              3821568 |       4096 | memory/IOBuffer/ProtocolProbeSessionAccept.cc:59
     13091 |          0 |              3351296 |        256 | memory/IOBuffer/SSLSessionCache.cc:161
     13091 |          0 |              3351296 |        256 | memory/IOBuffer/SSLSessionCache.cc:165
     51996 |      51985 |                45056 |       4096 | memory/IOBuffer/P_SSLNetVConnection.h:208
                                     123129088 |            | TOTAL
----------------------------------------------------------------------------------------------------------------------------------
Total Allocated      | Total Freed          | Currently Allocated  | Type
---------------------|----------------------|----------------------|----------------------
                   0 |                    0 |                    0 | SSL Allocated Memory
---------------------|----------------------|----------------------|----------------------

Change 588399 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm2

https://gerrit.wikimedia.org/r/588399

Since yesterday, memory increased up to 10Gb in cp1085, current snapshot:

     Allocated      |        In-Use      | Type Size  |   Free List Name
--------------------|--------------------|------------|----------------------------------
                  0 |                  0 |    2097152 | memory/ioBufAllocator[14]
                  0 |                  0 |    1048576 | memory/ioBufAllocator[13]
                  0 |                  0 |     524288 | memory/ioBufAllocator[12]
                  0 |                  0 |     262144 | memory/ioBufAllocator[11]
                  0 |                  0 |     131072 | memory/ioBufAllocator[10]
                  0 |                  0 |      65536 | memory/ioBufAllocator[9]
         2656043008 |            9732096 |      32768 | memory/ioBufAllocator[8]
         1774714880 |          299696128 |      16384 | memory/ioBufAllocator[7]
           26214400 |            8290304 |       8192 | memory/ioBufAllocator[6]
          321912832 |          192004096 |       4096 | memory/ioBufAllocator[5]
             262144 |                  0 |       2048 | memory/ioBufAllocator[4]
             131072 |                  0 |       1024 | memory/ioBufAllocator[3]
              65536 |                  0 |        512 | memory/ioBufAllocator[2]
          828899328 |          268435712 |        256 | memory/ioBufAllocator[1]
            1458176 |             539776 |        128 | memory/ioBufAllocator[0]
            5136384 |              81888 |         96 | memory/eventAllocator
           11211840 |            4780160 |         80 | memory/mutexAllocator
          295149568 |            5891456 |         64 | memory/ioBlockAllocator
          168903840 |           54765024 |         48 | memory/ioDataAllocator
           21444480 |           14791920 |        240 | memory/ioAllocator
                  0 |                  0 |        432 | memory/socksAllocator
                  0 |                  0 |        128 | memory/udpReadContAllocator
                  0 |                  0 |        160 | memory/udpPacketAllocator
            8294400 |            7720704 |        864 | memory/netVCAllocator
                  0 |                  0 |        128 | memory/UDPIOEventAllocator
          102891520 |           25023488 |       1024 | memory/sslNetVCAllocator
                  0 |                  0 |         64 | memory/RamCacheLRUEntry
                  0 |                  0 |         96 | memory/RamCacheCLFUSEntry
                  0 |                  0 |        160 | memory/openDirEntry
                  0 |                  0 |         48 | memory/evacuationKey
                  0 |                  0 |         64 | memory/cacheRemoveCont
                  0 |                  0 |         96 | memory/evacuationBlock
                  0 |                  0 |        944 | memory/cacheVConnection
                  0 |                  0 |         16 | memory/DNSRequestDataAllocator
                  0 |                  0 |      29584 | memory/dnsBufAllocator
                  0 |                  0 |       1280 | memory/dnsEntryAllocator
               4096 |                 96 |         16 | memory/expiryQueueEntry
               8192 |                384 |         64 | memory/refCountCacheHashingValueAllocator
                  0 |                  0 |         96 | memory/hostDBFileContAllocator
             294912 |               2304 |       2304 | memory/hostDBContAllocator
                  0 |                  0 |        128 | memory/OneWayTunnelAllocator
          164626432 |          123990016 |       2048 | memory/hdrStrHeap
          206045184 |          143296512 |       2048 | memory/hdrHeap
                  0 |                  0 |        256 | memory/httpCacheAltAllocator
           52396032 |           52373728 |       1312 | memory/http2ClientSessionAllocator
            9403584 |            9346768 |       1072 | memory/http2StreamAllocator
                  0 |                  0 |        128 | memory/RemapPluginsAlloc
            1744896 |            1631040 |        192 | memory/httpServerSessionAllocator
           34099200 |             879120 |       8880 | memory/httpSMAllocator
            9262240 |            8718608 |        976 | memory/http1ClientSessionAllocator
                  0 |                  0 |        128 | memory/socksProxyAllocator
               4096 |                  0 |         32 | memory/MIMEFieldSDKHandle
                  0 |                  0 |        288 | memory/INKVConnAllocator
              98304 |                512 |        128 | memory/INKContAllocator
               4096 |                128 |         32 | memory/apiHookAllocator
                  0 |                  0 |        512 | memory/FetchSMAllocator
            1441792 |             100352 |       1024 | memory/ArenaBlock
         6702166464 |         1232092320 |            | TOTAL
-----------------------------------------------------------------------------------------
Allocs     | Frees      | Size In-use          | Avg Size   | Location
-----------|------------|----------------------|------------|--------------------------------------------------------------------
 171501590 |  171501545 |               184320 |       4096 | memory/IOBuffer/UNKNOWN-LOCATION
     44232 |      44232 |                    0 |          0 | memory/IOBuffer/traffic_server/InkIOCoreAPI.cc:554
   7814507 |    7814112 |              1617920 |       4096 | memory/IOBuffer/Http1ClientSession.cc:208
 656163824 |  656163513 |              9588736 |      30831 | memory/IOBuffer/HttpSM.cc:6532
       208 |        205 |                98304 |      32768 | memory/IOBuffer/HttpSM.cc:6661
       242 |        239 |                98304 |      32768 | memory/IOBuffer/HttpSM.cc:6662
  16924807 |   16924807 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:6258
 223251636 |  223251636 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:5975
   4887381 |    4887380 |                32768 |      32768 | memory/IOBuffer/HttpSM.cc:5611
     42335 |      42335 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:745
 210440602 |  210439590 |              8290304 |       8192 | memory/IOBuffer/HttpServerSession.cc:88
     57296 |      57296 |                    0 |          0 | memory/IOBuffer/HttpTunnel.cc:95
  52404347 |   52404346 |                  256 |        256 | memory/IOBuffer/HttpTunnel.cc:100
 786300291 |  786255300 |            184283136 |       4096 | memory/IOBuffer/Http2ClientSession.cc:198
 121784814 |  121784812 |                 8192 |       4096 | memory/IOBuffer/Http2ClientSession.cc:204
 660893610 |  660871107 |            300235136 |      13342 | memory/IOBuffer/Http2ClientSession.h:103
      5529 |       5529 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1048
      5529 |       5529 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1051
  25215852 |   25214542 |              5365760 |       4096 | memory/IOBuffer/ProtocolProbeSessionAccept.cc:59
  13964635 |   13440347 |            134217728 |        256 | memory/IOBuffer/SSLSessionCache.cc:161
  13964635 |   13440347 |            134217728 |        256 | memory/IOBuffer/SSLSessionCache.cc:165
  78465776 |   78465663 |               462848 |       4096 | memory/IOBuffer/P_SSLNetVConnection.h:208
                                     778701440 |            | TOTAL
----------------------------------------------------------------------------------------------------------------------------------
Total Allocated      | Total Freed          | Currently Allocated  | Type
---------------------|----------------------|----------------------|----------------------
                   0 |                    0 |                    0 | SSL Allocated Memory
---------------------|----------------------|----------------------|----------------------

Change 588399 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm2

https://gerrit.wikimedia.org/r/588399

Mentioned in SAL (#wikimedia-operations) [2020-04-14T15:13:10Z] <vgutierrez> upload trafficserver 8.0.7-rc0-1wm2 to apt.wm.o (buster) - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-14T15:15:27Z] <vgutierrez> update to ats 8.0.7-rc0-1wm2 on cp[4026,4032] - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-15T06:04:05Z] <vgutierrez> update to ats 8.0.7-rc0-1wm2 on cp[5006,5012] - T249335

Change 588954 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Revert "ATS: Re-enable parent proxies on ats-tls"

https://gerrit.wikimedia.org/r/588954

Mentioned in SAL (#wikimedia-operations) [2020-04-15T12:41:30Z] <vgutierrez> rolling upgrade to ATS 8.0.7-rc0-1wm2 in ulsfo and eqsin - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-15T14:36:13Z] <vgutierrez> rolling upgrade to ATS 8.0.7-rc0-1wm2 on cp[3064,3065,2042,2041,1090,1089] - T249335

Change 589182 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm3

https://gerrit.wikimedia.org/r/589182

Change 589182 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm3

https://gerrit.wikimedia.org/r/589182

Mentioned in SAL (#wikimedia-operations) [2020-04-16T07:49:18Z] <vgutierrez> upload trafficserver 8.0.7-rc0-1wm3 to apt.wm.o (buster) - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-16T07:50:39Z] <vgutierrez> rolling update ats to version 8.0.7-rc0-1wm3 in cp[4026,4032,5006,5012] - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-16T10:44:59Z] <vgutierrez> upgrading ATS to version 8.0.7-rc0-1wm3 - T249335

Change 589473 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Track current client|server transactions

https://gerrit.wikimedia.org/r/589473

Change 589473 merged by Vgutierrez:
[operations/puppet@production] ATS: Track current client and server transactions

https://gerrit.wikimedia.org/r/589473

Mentioned in SAL (#wikimedia-operations) [2020-04-16T10:44:59Z] <vgutierrez> upgrading ATS to version 8.0.7-rc0-1wm3 - T249335

This, and/or the TLS 1.3 deployment, appears to have caused an extra regression in response time:

https://grafana.wikimedia.org/d/000000143/navigation-timing?orgId=1&var-source=navtiming2&var-metric=responseStart&var-percentile=p50&from=1587009985039&to=1587111099058

Screenshot 2020-04-17 at 10.15.46.png (378×1 px, 78 KB)

Still a 6.5% regression on the week-over-week p75 22 hours after this deployment.

Mentioned in SAL (#wikimedia-operations) [2020-04-19T16:05:07Z] <vgutierrez> rolling restart of ats-tls in text@esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-20T05:54:50Z] <vgutierrez> rolling restart of ats-tls in cp[3052,3054,3056,3058,3060,4028,4029,4030,4031,4032] - T249335

Change 590993 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm3asan

https://gerrit.wikimedia.org/r/590993

Mentioned in SAL (#wikimedia-operations) [2020-04-20T08:46:49Z] <vgutierrez> restart ats-tls in cp3064 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-27T05:47:39Z] <vgutierrez> rolling restart ats-tls in cp[1085,1089] and text@esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-27T13:53:23Z] <vgutierrez> restart ats-tls on cp3056 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-27T13:55:48Z] <vgutierrez> depool cp4026 and upgrade to ATS 8.1.0 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-27T14:46:47Z] <vgutierrez> pool cp4026 running ATS 8.1.0 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-28T13:53:17Z] <vgutierrez> update ATS 8.1 on cp4026 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-28T15:34:36Z] <vgutierrez> rolling restart of ats-tls on cp[3050,3052,3054,3056] - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-29T06:07:32Z] <vgutierrez> ats-tls restart on cp3064 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-29T06:17:28Z] <vgutierrez> ats-tls restart on cp[3050,3058] - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-29T08:32:19Z] <vgutierrez> upgrade to ATS 8.1 on cp4032 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-30T14:11:46Z] <vgutierrez> rolling restart of ats-tls on text@esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-01T13:01:12Z] <vgutierrez> rolling restart of ats-tls in text@esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-07T15:27:57Z] <vgutierrez> rolling restart of ats-tls on text@esams - T249335

Change 595088 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Puppetize max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595088

Change 595088 merged by Vgutierrez:
[operations/puppet@production] ATS: Puppetize max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595088

Change 595089 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Tune max_connections_in and max_connections_active_in on cp3050

https://gerrit.wikimedia.org/r/595089

Change 595089 merged by Vgutierrez:
[operations/puppet@production] ATS: Tune max_connections_in and max_connections_active_in on cp3050

https://gerrit.wikimedia.org/r/595089

Mentioned in SAL (#wikimedia-operations) [2020-05-08T07:19:17Z] <vgutierrez> ats-tls restart on cp3050 and cp3052 (max_connections_active_in experiment) - T249335

Change 595134 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in on esams

https://gerrit.wikimedia.org/r/595134

Change 595134 merged by Vgutierrez:
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in on esams

https://gerrit.wikimedia.org/r/595134

Mentioned in SAL (#wikimedia-operations) [2020-05-08T08:20:23Z] <vgutierrez> rolling restart of ats-tls on esams - T249335

Change 595172 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in globally

https://gerrit.wikimedia.org/r/595172

Change 595172 merged by Vgutierrez:
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in globally

https://gerrit.wikimedia.org/r/595172

Mentioned in SAL (#wikimedia-operations) [2020-05-08T13:22:33Z] <vgutierrez> rolling restart of ats-tls on eqiad, codfw, ulsfo and eqsin - T249335

Change 595484 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-1wm3

https://gerrit.wikimedia.org/r/595484

Change 595484 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-1wm3

https://gerrit.wikimedia.org/r/595484

Mentioned in SAL (#wikimedia-operations) [2020-05-11T10:15:10Z] <vgutierrez> upload trafficserver 8.0.7-1wm3 to apt.wm.o (buster) - T242767 T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-11T13:44:15Z] <vgutierrez> upgrade ATS to 8.0.7-1wm4 in cp4032 - T249335

Change 595533 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Relese 8.0.7-1wm4

https://gerrit.wikimedia.org/r/595533

Change 595533 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-1wm4

https://gerrit.wikimedia.org/r/595533

Mentioned in SAL (#wikimedia-operations) [2020-05-12T06:56:54Z] <vgutierrez> upload trafficserver 8.0.7-1wm4 to apt.wm.o (buster) - T242767 T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-12T10:30:17Z] <vgutierrez> upgrade trafficserver to version 8.0.7-1wm5 on cp4032 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-12T10:55:24Z] <vgutierrez> upgrade trafficserver to version 8.0.7-1wm5 on cp5011 - T249335

Change 595919 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595919

Change 595919 merged by Vgutierrez:
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595919

Mentioned in SAL (#wikimedia-operations) [2020-05-12T13:33:39Z] <vgutierrez> rolling upgrade of ATS to version 8.0.7-1wm5 - T249335

Change 596179 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-1wm6

https://gerrit.wikimedia.org/r/596179

Change 596179 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-1wm6

https://gerrit.wikimedia.org/r/596179

Change 596238 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] trafficserver_exporter: Track throttled active connections

https://gerrit.wikimedia.org/r/596238

Change 596238 merged by Vgutierrez:
[operations/puppet@production] trafficserver_exporter: Track throttled active connections

https://gerrit.wikimedia.org/r/596238

Change 597473 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable KA for POST requests on esams

https://gerrit.wikimedia.org/r/597473

Change 597473 merged by Vgutierrez:
[operations/puppet@production] ATS: Disable KA for POST requests on esams

https://gerrit.wikimedia.org/r/597473

Mentioned in SAL (#wikimedia-operations) [2020-05-20T09:43:55Z] <vgutierrez> disable KA for POST/PUT requests on esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-20T10:27:59Z] <vgutierrez> rolling restart of ats-tls in text@esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-06-05T07:52:46Z] <vgutierrez> rolling restart of ats-tls - T249335

Change 588954 merged by Vgutierrez:
[operations/puppet@production] Revert "ATS: Re-enable parent proxies on ats-tls"

https://gerrit.wikimedia.org/r/588954

Change 587516 abandoned by Vgutierrez:
ATS: Increase to 2 the number of accept_threads in cp3052 and cp3056

https://gerrit.wikimedia.org/r/587516

Change 610764 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/610764

Change 610764 merged by Vgutierrez:
[operations/puppet@production] ATS: Disable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/610764

BBlack added a subscriber: BBlack.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!