Page MenuHomePhabricator

Memory leak on ats-tls 8.0.6
Open, HighPublic

Description

Initial symptoms:

vgutierrez@lvs1013:~$ sudo -i journalctl -u pybal --since=today |grep "not pooled" |grep cp1075 |wc -l
135
vgutierrez@lvs1013:~$ sudo -i journalctl -u pybal --since=today |grep "not pooled" |grep cp1081 |wc -l
129

Flapping for unknown reasons

Memory leak reported to upstream in https://github.com/apache/trafficserver/issues/6614

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -1
operations/puppetproduction+397 -0
operations/puppetproduction+1 -12
operations/puppetproduction+15 -0
operations/puppetproduction+6 -0
operations/debs/trafficservermaster+202 -38
operations/puppetproduction+2 -2
operations/debs/trafficservermaster+87 -0
operations/debs/trafficservermaster+239 -0
operations/puppetproduction+2 -8
operations/puppetproduction+6 -6
operations/puppetproduction+6 -0
operations/puppetproduction+14 -0
operations/debs/trafficservermaster+11 -3
operations/puppetproduction+8 -0
operations/debs/trafficservermaster+20 -0
operations/debs/trafficservermaster+2 K -0
operations/puppetproduction+15 -0
operations/debs/trafficservermaster+52 -0
operations/puppetproduction+4 -5
operations/puppetproduction+5 -4
operations/puppetproduction+12 -1
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2020-04-09T05:40:40Z] <vgutierrez> upgrade ats to version 8.0.6-1wm6 in cp[4025,4031,5005,5011] - T249335

Memory consumption has improved a lot after backporting two HTTP/2 fixes:

This also has reduced the issues with pybal healthchecks almost completely, I'll continue to monitor this over the next few days.

Vgutierrez lowered the priority of this task from High to Medium.Apr 10 2020, 5:15 AM
Vgutierrez raised the priority of this task from Medium to High.Apr 10 2020, 1:01 PM

it looks like the leak is still there

Mentioned in SAL (#wikimedia-operations) [2020-04-10T15:03:08Z] <vgutierrez> restart ats-tls on cp1083 and cp1085 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-11T07:01:40Z] <vgutierrez> restart ats-tls on cp[1079,1081,1083,1085].eqiad.wmnet- T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-11T15:39:45Z] <vgutierrez> restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-12T05:50:15Z] <vgutierrez> restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-12T11:11:34Z] <vgutierrez> restart ats-tls on cp5008.eqsin.wmnet - T249335

Change 588366 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Enable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/588366

Change 588366 merged by Vgutierrez:
[operations/puppet@production] ATS: Enable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/588366

Mentioned in SAL (#wikimedia-operations) [2020-04-13T07:50:06Z] <vgutierrez> enable memory tracking in ats-tls on cp1085 - T249335

res_memory_tracking should help a lot, this is an example report @ cp1085 after restarting ats-tls:

     Allocated      |        In-Use      | Type Size  |   Free List Name
--------------------|--------------------|------------|----------------------------------
                  0 |                  0 |    2097152 | memory/ioBufAllocator[14]
                  0 |                  0 |    1048576 | memory/ioBufAllocator[13]
                  0 |                  0 |     524288 | memory/ioBufAllocator[12]
                  0 |                  0 |     262144 | memory/ioBufAllocator[11]
                  0 |                  0 |     131072 | memory/ioBufAllocator[10]
                  0 |                  0 |      65536 | memory/ioBufAllocator[9]
           18874368 |            8585216 |      32768 | memory/ioBufAllocator[8]
           59244544 |           58851328 |      16384 | memory/ioBufAllocator[7]
            9175040 |            7225344 |       8192 | memory/ioBufAllocator[6]
           42467328 |           41648128 |       4096 | memory/ioBufAllocator[5]
             262144 |                  0 |       2048 | memory/ioBufAllocator[4]
             131072 |                  0 |       1024 | memory/ioBufAllocator[3]
              65536 |                  0 |        512 | memory/ioBufAllocator[2]
            6717440 |            6702592 |        256 | memory/ioBufAllocator[1]
             131072 |             117248 |        128 | memory/ioBufAllocator[0]
             122880 |              73728 |         96 | memory/eventAllocator
            3280320 |            3270480 |         80 | memory/mutexAllocator
            1368064 |            1361152 |         64 | memory/ioBlockAllocator
            2121600 |            2114496 |         48 | memory/ioDataAllocator
            2807040 |            2797920 |        240 | memory/ioAllocator
                  0 |                  0 |        432 | memory/socksAllocator
                  0 |                  0 |        128 | memory/udpReadContAllocator
                  0 |                  0 |        160 | memory/udpPacketAllocator
            1327104 |            1327104 |        864 | memory/netVCAllocator
                  0 |                  0 |        128 | memory/UDPIOEventAllocator
            5898240 |            5795840 |       1024 | memory/sslNetVCAllocator
                  0 |                  0 |         64 | memory/RamCacheLRUEntry
                  0 |                  0 |         96 | memory/RamCacheCLFUSEntry
                  0 |                  0 |        160 | memory/openDirEntry
                  0 |                  0 |         48 | memory/evacuationKey
                  0 |                  0 |         64 | memory/cacheRemoveCont
                  0 |                  0 |         96 | memory/evacuationBlock
                  0 |                  0 |        944 | memory/cacheVConnection
                  0 |                  0 |         16 | memory/DNSRequestDataAllocator
                  0 |                  0 |      29584 | memory/dnsBufAllocator
                  0 |                  0 |       1280 | memory/dnsEntryAllocator
               4096 |                 16 |         16 | memory/expiryQueueEntry
               8192 |                384 |         64 | memory/refCountCacheHashingValueAllocator
                  0 |                  0 |         96 | memory/hostDBFileContAllocator
             294912 |               2304 |       2304 | memory/hostDBContAllocator
                  0 |                  0 |        128 | memory/OneWayTunnelAllocator
           21233664 |           21139456 |       2048 | memory/hdrStrHeap
           29097984 |           28913664 |       2048 | memory/hdrHeap
                  0 |                  0 |        256 | memory/httpCacheAltAllocator
            6045696 |            5982720 |       1312 | memory/http2ClientSessionAllocator
            1382880 |            1357152 |       1072 | memory/http2StreamAllocator
                  0 |                  0 |        128 | memory/RemapPluginsAlloc
             294912 |             294912 |        192 | memory/httpServerSessionAllocator
            2273280 |             737040 |       8880 | memory/httpSMAllocator
            1395680 |            1348832 |        976 | memory/http1ClientSessionAllocator
                  0 |                  0 |        128 | memory/socksProxyAllocator
               4096 |                  0 |         32 | memory/MIMEFieldSDKHandle
                  0 |                  0 |        288 | memory/INKVConnAllocator
              16384 |                640 |        128 | memory/INKContAllocator
               4096 |                128 |         32 | memory/apiHookAllocator
                  0 |                  0 |        512 | memory/FetchSMAllocator
             262144 |              83968 |       1024 | memory/ArenaBlock
          216311808 |          199731792 |            | TOTAL
-----------------------------------------------------------------------------------------
Allocs     | Frees      | Size In-use          | Avg Size   | Location
-----------|------------|----------------------|------------|--------------------------------------------------------------------
     34073 |      34049 |                98304 |       4096 | memory/IOBuffer/UNKNOWN-LOCATION
        64 |         64 |                    0 |          0 | memory/IOBuffer/traffic_server/InkIOCoreAPI.cc:554
      9964 |       9793 |               700416 |       4096 | memory/IOBuffer/Http1ClientSession.cc:208
    280538 |     280255 |              8671232 |      30640 | memory/IOBuffer/HttpSM.cc:6532
      2775 |       2775 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:6258
     91516 |      91515 |                 4096 |       4096 | memory/IOBuffer/HttpSM.cc:5975
      2299 |       2299 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:5611
        69 |         69 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:745
     92817 |      91935 |              7225344 |       8192 | memory/IOBuffer/HttpServerSession.cc:88
        80 |         80 |                    0 |          0 | memory/IOBuffer/HttpTunnel.cc:95
     12178 |      12178 |                    0 |          0 | memory/IOBuffer/HttpTunnel.cc:100
    167600 |     158594 |             36888576 |       4096 | memory/IOBuffer/Http2ClientSession.cc:198
     54660 |      54659 |                 4096 |       4096 | memory/IOBuffer/Http2ClientSession.cc:204
    163612 |     159110 |             58967808 |      13098 | memory/IOBuffer/Http2ClientSession.h:103
         8 |          8 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1048
         8 |          8 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1051
     26049 |      25116 |              3821568 |       4096 | memory/IOBuffer/ProtocolProbeSessionAccept.cc:59
     13091 |          0 |              3351296 |        256 | memory/IOBuffer/SSLSessionCache.cc:161
     13091 |          0 |              3351296 |        256 | memory/IOBuffer/SSLSessionCache.cc:165
     51996 |      51985 |                45056 |       4096 | memory/IOBuffer/P_SSLNetVConnection.h:208
                                     123129088 |            | TOTAL
----------------------------------------------------------------------------------------------------------------------------------
Total Allocated      | Total Freed          | Currently Allocated  | Type
---------------------|----------------------|----------------------|----------------------
                   0 |                    0 |                    0 | SSL Allocated Memory
---------------------|----------------------|----------------------|----------------------

Change 588399 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm2

https://gerrit.wikimedia.org/r/588399

Since yesterday, memory increased up to 10Gb in cp1085, current snapshot:

     Allocated      |        In-Use      | Type Size  |   Free List Name
--------------------|--------------------|------------|----------------------------------
                  0 |                  0 |    2097152 | memory/ioBufAllocator[14]
                  0 |                  0 |    1048576 | memory/ioBufAllocator[13]
                  0 |                  0 |     524288 | memory/ioBufAllocator[12]
                  0 |                  0 |     262144 | memory/ioBufAllocator[11]
                  0 |                  0 |     131072 | memory/ioBufAllocator[10]
                  0 |                  0 |      65536 | memory/ioBufAllocator[9]
         2656043008 |            9732096 |      32768 | memory/ioBufAllocator[8]
         1774714880 |          299696128 |      16384 | memory/ioBufAllocator[7]
           26214400 |            8290304 |       8192 | memory/ioBufAllocator[6]
          321912832 |          192004096 |       4096 | memory/ioBufAllocator[5]
             262144 |                  0 |       2048 | memory/ioBufAllocator[4]
             131072 |                  0 |       1024 | memory/ioBufAllocator[3]
              65536 |                  0 |        512 | memory/ioBufAllocator[2]
          828899328 |          268435712 |        256 | memory/ioBufAllocator[1]
            1458176 |             539776 |        128 | memory/ioBufAllocator[0]
            5136384 |              81888 |         96 | memory/eventAllocator
           11211840 |            4780160 |         80 | memory/mutexAllocator
          295149568 |            5891456 |         64 | memory/ioBlockAllocator
          168903840 |           54765024 |         48 | memory/ioDataAllocator
           21444480 |           14791920 |        240 | memory/ioAllocator
                  0 |                  0 |        432 | memory/socksAllocator
                  0 |                  0 |        128 | memory/udpReadContAllocator
                  0 |                  0 |        160 | memory/udpPacketAllocator
            8294400 |            7720704 |        864 | memory/netVCAllocator
                  0 |                  0 |        128 | memory/UDPIOEventAllocator
          102891520 |           25023488 |       1024 | memory/sslNetVCAllocator
                  0 |                  0 |         64 | memory/RamCacheLRUEntry
                  0 |                  0 |         96 | memory/RamCacheCLFUSEntry
                  0 |                  0 |        160 | memory/openDirEntry
                  0 |                  0 |         48 | memory/evacuationKey
                  0 |                  0 |         64 | memory/cacheRemoveCont
                  0 |                  0 |         96 | memory/evacuationBlock
                  0 |                  0 |        944 | memory/cacheVConnection
                  0 |                  0 |         16 | memory/DNSRequestDataAllocator
                  0 |                  0 |      29584 | memory/dnsBufAllocator
                  0 |                  0 |       1280 | memory/dnsEntryAllocator
               4096 |                 96 |         16 | memory/expiryQueueEntry
               8192 |                384 |         64 | memory/refCountCacheHashingValueAllocator
                  0 |                  0 |         96 | memory/hostDBFileContAllocator
             294912 |               2304 |       2304 | memory/hostDBContAllocator
                  0 |                  0 |        128 | memory/OneWayTunnelAllocator
          164626432 |          123990016 |       2048 | memory/hdrStrHeap
          206045184 |          143296512 |       2048 | memory/hdrHeap
                  0 |                  0 |        256 | memory/httpCacheAltAllocator
           52396032 |           52373728 |       1312 | memory/http2ClientSessionAllocator
            9403584 |            9346768 |       1072 | memory/http2StreamAllocator
                  0 |                  0 |        128 | memory/RemapPluginsAlloc
            1744896 |            1631040 |        192 | memory/httpServerSessionAllocator
           34099200 |             879120 |       8880 | memory/httpSMAllocator
            9262240 |            8718608 |        976 | memory/http1ClientSessionAllocator
                  0 |                  0 |        128 | memory/socksProxyAllocator
               4096 |                  0 |         32 | memory/MIMEFieldSDKHandle
                  0 |                  0 |        288 | memory/INKVConnAllocator
              98304 |                512 |        128 | memory/INKContAllocator
               4096 |                128 |         32 | memory/apiHookAllocator
                  0 |                  0 |        512 | memory/FetchSMAllocator
            1441792 |             100352 |       1024 | memory/ArenaBlock
         6702166464 |         1232092320 |            | TOTAL
-----------------------------------------------------------------------------------------
Allocs     | Frees      | Size In-use          | Avg Size   | Location
-----------|------------|----------------------|------------|--------------------------------------------------------------------
 171501590 |  171501545 |               184320 |       4096 | memory/IOBuffer/UNKNOWN-LOCATION
     44232 |      44232 |                    0 |          0 | memory/IOBuffer/traffic_server/InkIOCoreAPI.cc:554
   7814507 |    7814112 |              1617920 |       4096 | memory/IOBuffer/Http1ClientSession.cc:208
 656163824 |  656163513 |              9588736 |      30831 | memory/IOBuffer/HttpSM.cc:6532
       208 |        205 |                98304 |      32768 | memory/IOBuffer/HttpSM.cc:6661
       242 |        239 |                98304 |      32768 | memory/IOBuffer/HttpSM.cc:6662
  16924807 |   16924807 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:6258
 223251636 |  223251636 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:5975
   4887381 |    4887380 |                32768 |      32768 | memory/IOBuffer/HttpSM.cc:5611
     42335 |      42335 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:745
 210440602 |  210439590 |              8290304 |       8192 | memory/IOBuffer/HttpServerSession.cc:88
     57296 |      57296 |                    0 |          0 | memory/IOBuffer/HttpTunnel.cc:95
  52404347 |   52404346 |                  256 |        256 | memory/IOBuffer/HttpTunnel.cc:100
 786300291 |  786255300 |            184283136 |       4096 | memory/IOBuffer/Http2ClientSession.cc:198
 121784814 |  121784812 |                 8192 |       4096 | memory/IOBuffer/Http2ClientSession.cc:204
 660893610 |  660871107 |            300235136 |      13342 | memory/IOBuffer/Http2ClientSession.h:103
      5529 |       5529 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1048
      5529 |       5529 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1051
  25215852 |   25214542 |              5365760 |       4096 | memory/IOBuffer/ProtocolProbeSessionAccept.cc:59
  13964635 |   13440347 |            134217728 |        256 | memory/IOBuffer/SSLSessionCache.cc:161
  13964635 |   13440347 |            134217728 |        256 | memory/IOBuffer/SSLSessionCache.cc:165
  78465776 |   78465663 |               462848 |       4096 | memory/IOBuffer/P_SSLNetVConnection.h:208
                                     778701440 |            | TOTAL
----------------------------------------------------------------------------------------------------------------------------------
Total Allocated      | Total Freed          | Currently Allocated  | Type
---------------------|----------------------|----------------------|----------------------
                   0 |                    0 |                    0 | SSL Allocated Memory
---------------------|----------------------|----------------------|----------------------

Change 588399 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm2

https://gerrit.wikimedia.org/r/588399

Mentioned in SAL (#wikimedia-operations) [2020-04-14T15:13:10Z] <vgutierrez> upload trafficserver 8.0.7-rc0-1wm2 to apt.wm.o (buster) - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-14T15:15:27Z] <vgutierrez> update to ats 8.0.7-rc0-1wm2 on cp[4026,4032] - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-15T06:04:05Z] <vgutierrez> update to ats 8.0.7-rc0-1wm2 on cp[5006,5012] - T249335

Change 588954 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Revert "ATS: Re-enable parent proxies on ats-tls"

https://gerrit.wikimedia.org/r/588954

Mentioned in SAL (#wikimedia-operations) [2020-04-15T12:41:30Z] <vgutierrez> rolling upgrade to ATS 8.0.7-rc0-1wm2 in ulsfo and eqsin - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-15T14:36:13Z] <vgutierrez> rolling upgrade to ATS 8.0.7-rc0-1wm2 on cp[3064,3065,2042,2041,1090,1089] - T249335

Change 589182 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm3

https://gerrit.wikimedia.org/r/589182

Change 589182 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm3

https://gerrit.wikimedia.org/r/589182

Mentioned in SAL (#wikimedia-operations) [2020-04-16T07:49:18Z] <vgutierrez> upload trafficserver 8.0.7-rc0-1wm3 to apt.wm.o (buster) - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-16T07:50:39Z] <vgutierrez> rolling update ats to version 8.0.7-rc0-1wm3 in cp[4026,4032,5006,5012] - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-16T10:44:59Z] <vgutierrez> upgrading ATS to version 8.0.7-rc0-1wm3 - T249335

Change 589473 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Track current client|server transactions

https://gerrit.wikimedia.org/r/589473

Change 589473 merged by Vgutierrez:
[operations/puppet@production] ATS: Track current client and server transactions

https://gerrit.wikimedia.org/r/589473

Gilles added a subscriber: Gilles.Apr 17 2020, 8:18 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-16T10:44:59Z] <vgutierrez> upgrading ATS to version 8.0.7-rc0-1wm3 - T249335

This, and/or the TLS 1.3 deployment, appears to have caused an extra regression in response time:

https://grafana.wikimedia.org/d/000000143/navigation-timing?orgId=1&var-source=navtiming2&var-metric=responseStart&var-percentile=p50&from=1587009985039&to=1587111099058

Still a 6.5% regression on the week-over-week p75 22 hours after this deployment.

Mentioned in SAL (#wikimedia-operations) [2020-04-19T16:05:07Z] <vgutierrez> rolling restart of ats-tls in text@esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-20T05:54:50Z] <vgutierrez> rolling restart of ats-tls in cp[3052,3054,3056,3058,3060,4028,4029,4030,4031,4032] - T249335

Change 590993 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm3asan

https://gerrit.wikimedia.org/r/590993

Mentioned in SAL (#wikimedia-operations) [2020-04-20T08:46:49Z] <vgutierrez> restart ats-tls in cp3064 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-27T05:47:39Z] <vgutierrez> rolling restart ats-tls in cp[1085,1089] and text@esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-27T13:53:23Z] <vgutierrez> restart ats-tls on cp3056 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-27T13:55:48Z] <vgutierrez> depool cp4026 and upgrade to ATS 8.1.0 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-27T14:46:47Z] <vgutierrez> pool cp4026 running ATS 8.1.0 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-28T13:53:17Z] <vgutierrez> update ATS 8.1 on cp4026 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-28T15:34:36Z] <vgutierrez> rolling restart of ats-tls on cp[3050,3052,3054,3056] - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-29T06:07:32Z] <vgutierrez> ats-tls restart on cp3064 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-29T06:17:28Z] <vgutierrez> ats-tls restart on cp[3050,3058] - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-29T08:32:19Z] <vgutierrez> upgrade to ATS 8.1 on cp4032 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-04-30T14:11:46Z] <vgutierrez> rolling restart of ats-tls on text@esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-01T13:01:12Z] <vgutierrez> rolling restart of ats-tls in text@esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-07T15:27:57Z] <vgutierrez> rolling restart of ats-tls on text@esams - T249335

Change 595088 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Puppetize max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595088

Change 595088 merged by Vgutierrez:
[operations/puppet@production] ATS: Puppetize max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595088

Change 595089 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Tune max_connections_in and max_connections_active_in on cp3050

https://gerrit.wikimedia.org/r/595089

Change 595089 merged by Vgutierrez:
[operations/puppet@production] ATS: Tune max_connections_in and max_connections_active_in on cp3050

https://gerrit.wikimedia.org/r/595089

Mentioned in SAL (#wikimedia-operations) [2020-05-08T07:19:17Z] <vgutierrez> ats-tls restart on cp3050 and cp3052 (max_connections_active_in experiment) - T249335

Change 595134 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in on esams

https://gerrit.wikimedia.org/r/595134

Change 595134 merged by Vgutierrez:
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in on esams

https://gerrit.wikimedia.org/r/595134

Mentioned in SAL (#wikimedia-operations) [2020-05-08T08:20:23Z] <vgutierrez> rolling restart of ats-tls on esams - T249335

Change 595172 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in globally

https://gerrit.wikimedia.org/r/595172

Change 595172 merged by Vgutierrez:
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in globally

https://gerrit.wikimedia.org/r/595172

Mentioned in SAL (#wikimedia-operations) [2020-05-08T13:22:33Z] <vgutierrez> rolling restart of ats-tls on eqiad, codfw, ulsfo and eqsin - T249335

Change 595484 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-1wm3

https://gerrit.wikimedia.org/r/595484

Change 595484 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-1wm3

https://gerrit.wikimedia.org/r/595484

Mentioned in SAL (#wikimedia-operations) [2020-05-11T10:15:10Z] <vgutierrez> upload trafficserver 8.0.7-1wm3 to apt.wm.o (buster) - T242767 T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-11T13:44:15Z] <vgutierrez> upgrade ATS to 8.0.7-1wm4 in cp4032 - T249335

Change 595533 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Relese 8.0.7-1wm4

https://gerrit.wikimedia.org/r/595533

Change 595533 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-1wm4

https://gerrit.wikimedia.org/r/595533

Mentioned in SAL (#wikimedia-operations) [2020-05-12T06:56:54Z] <vgutierrez> upload trafficserver 8.0.7-1wm4 to apt.wm.o (buster) - T242767 T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-12T10:30:17Z] <vgutierrez> upgrade trafficserver to version 8.0.7-1wm5 on cp4032 - T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-12T10:55:24Z] <vgutierrez> upgrade trafficserver to version 8.0.7-1wm5 on cp5011 - T249335

Change 595919 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595919

Change 595919 merged by Vgutierrez:
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595919

Mentioned in SAL (#wikimedia-operations) [2020-05-12T13:33:39Z] <vgutierrez> rolling upgrade of ATS to version 8.0.7-1wm5 - T249335

Change 596179 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-1wm6

https://gerrit.wikimedia.org/r/596179

Change 596179 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-1wm6

https://gerrit.wikimedia.org/r/596179

Change 596238 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] trafficserver_exporter: Track throttled active connections

https://gerrit.wikimedia.org/r/596238

Change 596238 merged by Vgutierrez:
[operations/puppet@production] trafficserver_exporter: Track throttled active connections

https://gerrit.wikimedia.org/r/596238

Change 597473 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable KA for POST requests on esams

https://gerrit.wikimedia.org/r/597473

Change 597473 merged by Vgutierrez:
[operations/puppet@production] ATS: Disable KA for POST requests on esams

https://gerrit.wikimedia.org/r/597473

Mentioned in SAL (#wikimedia-operations) [2020-05-20T09:43:55Z] <vgutierrez> disable KA for POST/PUT requests on esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-05-20T10:27:59Z] <vgutierrez> rolling restart of ats-tls in text@esams - T249335

Mentioned in SAL (#wikimedia-operations) [2020-06-05T07:52:46Z] <vgutierrez> rolling restart of ats-tls - T249335

Change 588954 merged by Vgutierrez:
[operations/puppet@production] Revert "ATS: Re-enable parent proxies on ats-tls"

https://gerrit.wikimedia.org/r/588954

Change 587516 abandoned by Vgutierrez:
ATS: Increase to 2 the number of accept_threads in cp3052 and cp3056

https://gerrit.wikimedia.org/r/587516

Change 610764 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/610764

Change 610764 merged by Vgutierrez:
[operations/puppet@production] ATS: Disable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/610764