Page MenuHomePhabricator
Create Task
Maniphest T249335

Memory leak on ats-tls 8.0.6
Closed, InvalidPublic
Actions

Description

Initial symptoms:

vgutierrez@lvs1013:~$ sudo -i journalctl -u pybal --since=today |grep "not pooled" |grep cp1075 |wc -l
135
vgutierrez@lvs1013:~$ sudo -i journalctl -u pybal --since=today |grep "not pooled" |grep cp1081 |wc -l
129

Flapping for unknown reasons

Memory leak reported to upstream in https://github.com/apache/trafficserver/issues/6614

Details

SubjectRepoBranchLines +/-
Release 8.0.7-rc0-1wm3asanoperations/debs/trafficservermaster+11 -3
ATS: Disable res_track_memory in cp1085operations/puppetproduction+0 -1
ATS: Increase to 2 the number of accept_threads in cp3052 and cp3056operations/puppetproduction+397 -0
Revert "ATS: Re-enable parent proxies on ats-tls"operations/puppetproduction+1 -12
ATS: Disable KA for POST requests on esamsoperations/puppetproduction+15 -0
trafficserver_exporter: Track throttled active connectionsoperations/puppetproduction+6 -0
Release 8.0.7-1wm6operations/debs/trafficservermaster+202 -38
ATS: Increase max_connections_in and max_connections_active_inoperations/puppetproduction+2 -2
Release 8.0.7-1wm4operations/debs/trafficservermaster+87 -0
Release 8.0.7-1wm3operations/debs/trafficservermaster+239 -0
ATS: Increase max_connections_in and max_connections_active_in globallyoperations/puppetproduction+2 -8
ATS: Increase max_connections_in and max_connections_active_in on esamsoperations/puppetproduction+6 -6
ATS: Tune max_connections_in and max_connections_active_in on cp3050operations/puppetproduction+6 -0
ATS: Puppetize max_connections_in and max_connections_active_inoperations/puppetproduction+14 -0
ATS: Track current client and server transactionsoperations/puppetproduction+8 -0
Release 8.0.7-rc0-1wm3operations/debs/trafficservermaster+20 -0
Release 8.0.7-rc0-1wm2operations/debs/trafficservermaster+2 K -0
ATS: Enable res_track_memory in cp1085operations/puppetproduction+15 -0
Release 8.0.6-1wm6operations/debs/trafficservermaster+52 -0
Revert "ATS: Disable wmf-analytics log"operations/puppetproduction+4 -5
ATS: Disable wmf-analytics logoperations/puppetproduction+5 -4
ATS: Re-enable parent proxies on ats-tlsoperations/puppetproduction+12 -1
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Apr 12 2020, 11:11 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-12T11:11:34Z] <vgutierrez> restart ats-tls on cp5008.eqsin.wmnet - T249335

• Mholloway subscribed.Apr 12 2020, 6:48 PM
gerritbot added a comment.Apr 13 2020, 7:23 AM

Change 588366 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Enable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/588366

gerritbot added a comment.Apr 13 2020, 7:44 AM

Change 588366 merged by Vgutierrez:
[operations/puppet@production] ATS: Enable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/588366

Stashbot added a comment.Apr 13 2020, 7:50 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-13T07:50:06Z] <vgutierrez> enable memory tracking in ats-tls on cp1085 - T249335

Vgutierrez added a comment.Apr 13 2020, 7:59 AM

res_memory_tracking should help a lot, this is an example report @ cp1085 after restarting ats-tls:

     Allocated      |        In-Use      | Type Size  |   Free List Name
--------------------|--------------------|------------|----------------------------------
                  0 |                  0 |    2097152 | memory/ioBufAllocator[14]
                  0 |                  0 |    1048576 | memory/ioBufAllocator[13]
                  0 |                  0 |     524288 | memory/ioBufAllocator[12]
                  0 |                  0 |     262144 | memory/ioBufAllocator[11]
                  0 |                  0 |     131072 | memory/ioBufAllocator[10]
                  0 |                  0 |      65536 | memory/ioBufAllocator[9]
           18874368 |            8585216 |      32768 | memory/ioBufAllocator[8]
           59244544 |           58851328 |      16384 | memory/ioBufAllocator[7]
            9175040 |            7225344 |       8192 | memory/ioBufAllocator[6]
           42467328 |           41648128 |       4096 | memory/ioBufAllocator[5]
             262144 |                  0 |       2048 | memory/ioBufAllocator[4]
             131072 |                  0 |       1024 | memory/ioBufAllocator[3]
              65536 |                  0 |        512 | memory/ioBufAllocator[2]
            6717440 |            6702592 |        256 | memory/ioBufAllocator[1]
             131072 |             117248 |        128 | memory/ioBufAllocator[0]
             122880 |              73728 |         96 | memory/eventAllocator
            3280320 |            3270480 |         80 | memory/mutexAllocator
            1368064 |            1361152 |         64 | memory/ioBlockAllocator
            2121600 |            2114496 |         48 | memory/ioDataAllocator
            2807040 |            2797920 |        240 | memory/ioAllocator
                  0 |                  0 |        432 | memory/socksAllocator
                  0 |                  0 |        128 | memory/udpReadContAllocator
                  0 |                  0 |        160 | memory/udpPacketAllocator
            1327104 |            1327104 |        864 | memory/netVCAllocator
                  0 |                  0 |        128 | memory/UDPIOEventAllocator
            5898240 |            5795840 |       1024 | memory/sslNetVCAllocator
                  0 |                  0 |         64 | memory/RamCacheLRUEntry
                  0 |                  0 |         96 | memory/RamCacheCLFUSEntry
                  0 |                  0 |        160 | memory/openDirEntry
                  0 |                  0 |         48 | memory/evacuationKey
                  0 |                  0 |         64 | memory/cacheRemoveCont
                  0 |                  0 |         96 | memory/evacuationBlock
                  0 |                  0 |        944 | memory/cacheVConnection
                  0 |                  0 |         16 | memory/DNSRequestDataAllocator
                  0 |                  0 |      29584 | memory/dnsBufAllocator
                  0 |                  0 |       1280 | memory/dnsEntryAllocator
               4096 |                 16 |         16 | memory/expiryQueueEntry
               8192 |                384 |         64 | memory/refCountCacheHashingValueAllocator
                  0 |                  0 |         96 | memory/hostDBFileContAllocator
             294912 |               2304 |       2304 | memory/hostDBContAllocator
                  0 |                  0 |        128 | memory/OneWayTunnelAllocator
           21233664 |           21139456 |       2048 | memory/hdrStrHeap
           29097984 |           28913664 |       2048 | memory/hdrHeap
                  0 |                  0 |        256 | memory/httpCacheAltAllocator
            6045696 |            5982720 |       1312 | memory/http2ClientSessionAllocator
            1382880 |            1357152 |       1072 | memory/http2StreamAllocator
                  0 |                  0 |        128 | memory/RemapPluginsAlloc
             294912 |             294912 |        192 | memory/httpServerSessionAllocator
            2273280 |             737040 |       8880 | memory/httpSMAllocator
            1395680 |            1348832 |        976 | memory/http1ClientSessionAllocator
                  0 |                  0 |        128 | memory/socksProxyAllocator
               4096 |                  0 |         32 | memory/MIMEFieldSDKHandle
                  0 |                  0 |        288 | memory/INKVConnAllocator
              16384 |                640 |        128 | memory/INKContAllocator
               4096 |                128 |         32 | memory/apiHookAllocator
                  0 |                  0 |        512 | memory/FetchSMAllocator
             262144 |              83968 |       1024 | memory/ArenaBlock
          216311808 |          199731792 |            | TOTAL
-----------------------------------------------------------------------------------------
Allocs     | Frees      | Size In-use          | Avg Size   | Location
-----------|------------|----------------------|------------|--------------------------------------------------------------------
     34073 |      34049 |                98304 |       4096 | memory/IOBuffer/UNKNOWN-LOCATION
        64 |         64 |                    0 |          0 | memory/IOBuffer/traffic_server/InkIOCoreAPI.cc:554
      9964 |       9793 |               700416 |       4096 | memory/IOBuffer/Http1ClientSession.cc:208
    280538 |     280255 |              8671232 |      30640 | memory/IOBuffer/HttpSM.cc:6532
      2775 |       2775 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:6258
     91516 |      91515 |                 4096 |       4096 | memory/IOBuffer/HttpSM.cc:5975
      2299 |       2299 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:5611
        69 |         69 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:745
     92817 |      91935 |              7225344 |       8192 | memory/IOBuffer/HttpServerSession.cc:88
        80 |         80 |                    0 |          0 | memory/IOBuffer/HttpTunnel.cc:95
     12178 |      12178 |                    0 |          0 | memory/IOBuffer/HttpTunnel.cc:100
    167600 |     158594 |             36888576 |       4096 | memory/IOBuffer/Http2ClientSession.cc:198
     54660 |      54659 |                 4096 |       4096 | memory/IOBuffer/Http2ClientSession.cc:204
    163612 |     159110 |             58967808 |      13098 | memory/IOBuffer/Http2ClientSession.h:103
         8 |          8 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1048
         8 |          8 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1051
     26049 |      25116 |              3821568 |       4096 | memory/IOBuffer/ProtocolProbeSessionAccept.cc:59
     13091 |          0 |              3351296 |        256 | memory/IOBuffer/SSLSessionCache.cc:161
     13091 |          0 |              3351296 |        256 | memory/IOBuffer/SSLSessionCache.cc:165
     51996 |      51985 |                45056 |       4096 | memory/IOBuffer/P_SSLNetVConnection.h:208
                                     123129088 |            | TOTAL
----------------------------------------------------------------------------------------------------------------------------------
Total Allocated      | Total Freed          | Currently Allocated  | Type
---------------------|----------------------|----------------------|----------------------
                   0 |                    0 |                    0 | SSL Allocated Memory
---------------------|----------------------|----------------------|----------------------
gerritbot added a comment.Apr 13 2020, 1:04 PM

Change 588399 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm2

https://gerrit.wikimedia.org/r/588399

Vgutierrez added a comment.Apr 14 2020, 5:27 AM

Since yesterday, memory increased up to 10Gb in cp1085, current snapshot:

     Allocated      |        In-Use      | Type Size  |   Free List Name
--------------------|--------------------|------------|----------------------------------
                  0 |                  0 |    2097152 | memory/ioBufAllocator[14]
                  0 |                  0 |    1048576 | memory/ioBufAllocator[13]
                  0 |                  0 |     524288 | memory/ioBufAllocator[12]
                  0 |                  0 |     262144 | memory/ioBufAllocator[11]
                  0 |                  0 |     131072 | memory/ioBufAllocator[10]
                  0 |                  0 |      65536 | memory/ioBufAllocator[9]
         2656043008 |            9732096 |      32768 | memory/ioBufAllocator[8]
         1774714880 |          299696128 |      16384 | memory/ioBufAllocator[7]
           26214400 |            8290304 |       8192 | memory/ioBufAllocator[6]
          321912832 |          192004096 |       4096 | memory/ioBufAllocator[5]
             262144 |                  0 |       2048 | memory/ioBufAllocator[4]
             131072 |                  0 |       1024 | memory/ioBufAllocator[3]
              65536 |                  0 |        512 | memory/ioBufAllocator[2]
          828899328 |          268435712 |        256 | memory/ioBufAllocator[1]
            1458176 |             539776 |        128 | memory/ioBufAllocator[0]
            5136384 |              81888 |         96 | memory/eventAllocator
           11211840 |            4780160 |         80 | memory/mutexAllocator
          295149568 |            5891456 |         64 | memory/ioBlockAllocator
          168903840 |           54765024 |         48 | memory/ioDataAllocator
           21444480 |           14791920 |        240 | memory/ioAllocator
                  0 |                  0 |        432 | memory/socksAllocator
                  0 |                  0 |        128 | memory/udpReadContAllocator
                  0 |                  0 |        160 | memory/udpPacketAllocator
            8294400 |            7720704 |        864 | memory/netVCAllocator
                  0 |                  0 |        128 | memory/UDPIOEventAllocator
          102891520 |           25023488 |       1024 | memory/sslNetVCAllocator
                  0 |                  0 |         64 | memory/RamCacheLRUEntry
                  0 |                  0 |         96 | memory/RamCacheCLFUSEntry
                  0 |                  0 |        160 | memory/openDirEntry
                  0 |                  0 |         48 | memory/evacuationKey
                  0 |                  0 |         64 | memory/cacheRemoveCont
                  0 |                  0 |         96 | memory/evacuationBlock
                  0 |                  0 |        944 | memory/cacheVConnection
                  0 |                  0 |         16 | memory/DNSRequestDataAllocator
                  0 |                  0 |      29584 | memory/dnsBufAllocator
                  0 |                  0 |       1280 | memory/dnsEntryAllocator
               4096 |                 96 |         16 | memory/expiryQueueEntry
               8192 |                384 |         64 | memory/refCountCacheHashingValueAllocator
                  0 |                  0 |         96 | memory/hostDBFileContAllocator
             294912 |               2304 |       2304 | memory/hostDBContAllocator
                  0 |                  0 |        128 | memory/OneWayTunnelAllocator
          164626432 |          123990016 |       2048 | memory/hdrStrHeap
          206045184 |          143296512 |       2048 | memory/hdrHeap
                  0 |                  0 |        256 | memory/httpCacheAltAllocator
           52396032 |           52373728 |       1312 | memory/http2ClientSessionAllocator
            9403584 |            9346768 |       1072 | memory/http2StreamAllocator
                  0 |                  0 |        128 | memory/RemapPluginsAlloc
            1744896 |            1631040 |        192 | memory/httpServerSessionAllocator
           34099200 |             879120 |       8880 | memory/httpSMAllocator
            9262240 |            8718608 |        976 | memory/http1ClientSessionAllocator
                  0 |                  0 |        128 | memory/socksProxyAllocator
               4096 |                  0 |         32 | memory/MIMEFieldSDKHandle
                  0 |                  0 |        288 | memory/INKVConnAllocator
              98304 |                512 |        128 | memory/INKContAllocator
               4096 |                128 |         32 | memory/apiHookAllocator
                  0 |                  0 |        512 | memory/FetchSMAllocator
            1441792 |             100352 |       1024 | memory/ArenaBlock
         6702166464 |         1232092320 |            | TOTAL
-----------------------------------------------------------------------------------------
Allocs     | Frees      | Size In-use          | Avg Size   | Location
-----------|------------|----------------------|------------|--------------------------------------------------------------------
 171501590 |  171501545 |               184320 |       4096 | memory/IOBuffer/UNKNOWN-LOCATION
     44232 |      44232 |                    0 |          0 | memory/IOBuffer/traffic_server/InkIOCoreAPI.cc:554
   7814507 |    7814112 |              1617920 |       4096 | memory/IOBuffer/Http1ClientSession.cc:208
 656163824 |  656163513 |              9588736 |      30831 | memory/IOBuffer/HttpSM.cc:6532
       208 |        205 |                98304 |      32768 | memory/IOBuffer/HttpSM.cc:6661
       242 |        239 |                98304 |      32768 | memory/IOBuffer/HttpSM.cc:6662
  16924807 |   16924807 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:6258
 223251636 |  223251636 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:5975
   4887381 |    4887380 |                32768 |      32768 | memory/IOBuffer/HttpSM.cc:5611
     42335 |      42335 |                    0 |          0 | memory/IOBuffer/HttpSM.cc:745
 210440602 |  210439590 |              8290304 |       8192 | memory/IOBuffer/HttpServerSession.cc:88
     57296 |      57296 |                    0 |          0 | memory/IOBuffer/HttpTunnel.cc:95
  52404347 |   52404346 |                  256 |        256 | memory/IOBuffer/HttpTunnel.cc:100
 786300291 |  786255300 |            184283136 |       4096 | memory/IOBuffer/Http2ClientSession.cc:198
 121784814 |  121784812 |                 8192 |       4096 | memory/IOBuffer/Http2ClientSession.cc:204
 660893610 |  660871107 |            300235136 |      13342 | memory/IOBuffer/Http2ClientSession.h:103
      5529 |       5529 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1048
      5529 |       5529 |                    0 |          0 | memory/IOBuffer/PluginVC.cc:1051
  25215852 |   25214542 |              5365760 |       4096 | memory/IOBuffer/ProtocolProbeSessionAccept.cc:59
  13964635 |   13440347 |            134217728 |        256 | memory/IOBuffer/SSLSessionCache.cc:161
  13964635 |   13440347 |            134217728 |        256 | memory/IOBuffer/SSLSessionCache.cc:165
  78465776 |   78465663 |               462848 |       4096 | memory/IOBuffer/P_SSLNetVConnection.h:208
                                     778701440 |            | TOTAL
----------------------------------------------------------------------------------------------------------------------------------
Total Allocated      | Total Freed          | Currently Allocated  | Type
---------------------|----------------------|----------------------|----------------------
                   0 |                    0 |                    0 | SSL Allocated Memory
---------------------|----------------------|----------------------|----------------------
gerritbot added a comment.Apr 14 2020, 2:52 PM

Change 588399 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm2

https://gerrit.wikimedia.org/r/588399

Stashbot added a comment.Apr 14 2020, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-14T15:13:10Z] <vgutierrez> upload trafficserver 8.0.7-rc0-1wm2 to apt.wm.o (buster) - T249335

Stashbot added a comment.Apr 14 2020, 3:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-14T15:15:27Z] <vgutierrez> update to ats 8.0.7-rc0-1wm2 on cp[4026,4032] - T249335

Stashbot added a comment.Apr 15 2020, 6:04 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-15T06:04:05Z] <vgutierrez> update to ats 8.0.7-rc0-1wm2 on cp[5006,5012] - T249335

hashar mentioned this in T250015: Unable to find image 'docker-registry.wikimedia.org/releng/quibble-stretch-php73:0.0.41-s1'.Apr 15 2020, 6:08 AM
gerritbot added a comment.Apr 15 2020, 8:36 AM

Change 588954 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Revert "ATS: Re-enable parent proxies on ats-tls"

https://gerrit.wikimedia.org/r/588954

Stashbot added a comment.Apr 15 2020, 12:41 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-15T12:41:30Z] <vgutierrez> rolling upgrade to ATS 8.0.7-rc0-1wm2 in ulsfo and eqsin - T249335

Stashbot added a comment.Apr 15 2020, 2:36 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-15T14:36:13Z] <vgutierrez> rolling upgrade to ATS 8.0.7-rc0-1wm2 on cp[3064,3065,2042,2041,1090,1089] - T249335

gerritbot added a comment.Apr 16 2020, 4:46 AM

Change 589182 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm3

https://gerrit.wikimedia.org/r/589182

gerritbot added a comment.Apr 16 2020, 7:31 AM

Change 589182 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm3

https://gerrit.wikimedia.org/r/589182

Stashbot added a comment.Apr 16 2020, 7:49 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-16T07:49:18Z] <vgutierrez> upload trafficserver 8.0.7-rc0-1wm3 to apt.wm.o (buster) - T249335

Stashbot added a comment.Apr 16 2020, 7:50 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-16T07:50:39Z] <vgutierrez> rolling update ats to version 8.0.7-rc0-1wm3 in cp[4026,4032,5006,5012] - T249335

Stashbot added a comment.Apr 16 2020, 10:45 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-16T10:44:59Z] <vgutierrez> upgrading ATS to version 8.0.7-rc0-1wm3 - T249335

gerritbot added a comment.Apr 17 2020, 6:40 AM

Change 589473 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Track current client|server transactions

https://gerrit.wikimedia.org/r/589473

gerritbot added a comment.Apr 17 2020, 7:40 AM

Change 589473 merged by Vgutierrez:
[operations/puppet@production] ATS: Track current client and server transactions

https://gerrit.wikimedia.org/r/589473

• Gilles subscribed.Apr 17 2020, 8:18 AM
In T249335#6062023, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2020-04-16T10:44:59Z] <vgutierrez> upgrading ATS to version 8.0.7-rc0-1wm3 - T249335

This, and/or the TLS 1.3 deployment, appears to have caused an extra regression in response time:

https://grafana.wikimedia.org/d/000000143/navigation-timing?orgId=1&var-source=navtiming2&var-metric=responseStart&var-percentile=p50&from=1587009985039&to=1587111099058

Screenshot 2020-04-17 at 10.15.46.png (378×1 px, 78 KB)

Still a 6.5% regression on the week-over-week p75 22 hours after this deployment.

Stashbot added a comment.Apr 19 2020, 4:05 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-19T16:05:07Z] <vgutierrez> rolling restart of ats-tls in text@esams - T249335

Krinkle added a project: Wikimedia-Incident.Apr 20 2020, 12:55 AM
Stashbot added a comment.Apr 20 2020, 5:54 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-20T05:54:50Z] <vgutierrez> rolling restart of ats-tls in cp[3052,3054,3056,3058,3060,4028,4029,4030,4031,4032] - T249335

gerritbot added a comment.Apr 20 2020, 7:29 AM

Change 590993 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm3asan

https://gerrit.wikimedia.org/r/590993

Stashbot added a comment.Apr 20 2020, 8:46 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-20T08:46:49Z] <vgutierrez> restart ats-tls in cp3064 - T249335

Stashbot added a comment.Apr 27 2020, 5:47 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-27T05:47:39Z] <vgutierrez> rolling restart ats-tls in cp[1085,1089] and text@esams - T249335

Stashbot added a comment.Apr 27 2020, 1:53 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-27T13:53:23Z] <vgutierrez> restart ats-tls on cp3056 - T249335

Stashbot added a comment.Apr 27 2020, 1:55 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-27T13:55:48Z] <vgutierrez> depool cp4026 and upgrade to ATS 8.1.0 - T249335

Stashbot added a comment.Apr 27 2020, 2:46 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-27T14:46:47Z] <vgutierrez> pool cp4026 running ATS 8.1.0 - T249335

• Mholloway unsubscribed.Apr 27 2020, 2:50 PM
Stashbot added a comment.Apr 28 2020, 1:53 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-28T13:53:17Z] <vgutierrez> update ATS 8.1 on cp4026 - T249335

Stashbot added a comment.Apr 28 2020, 3:34 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-28T15:34:36Z] <vgutierrez> rolling restart of ats-tls on cp[3050,3052,3054,3056] - T249335

Stashbot added a comment.Apr 29 2020, 6:07 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-29T06:07:32Z] <vgutierrez> ats-tls restart on cp3064 - T249335

Stashbot added a comment.Apr 29 2020, 6:17 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-29T06:17:28Z] <vgutierrez> ats-tls restart on cp[3050,3058] - T249335

Stashbot added a comment.Apr 29 2020, 8:32 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-29T08:32:19Z] <vgutierrez> upgrade to ATS 8.1 on cp4032 - T249335

Stashbot added a comment.Apr 30 2020, 2:11 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-30T14:11:46Z] <vgutierrez> rolling restart of ats-tls on text@esams - T249335

Stashbot added a comment.May 1 2020, 1:01 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-01T13:01:12Z] <vgutierrez> rolling restart of ats-tls in text@esams - T249335

Stashbot added a comment.May 7 2020, 3:27 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-07T15:27:57Z] <vgutierrez> rolling restart of ats-tls on text@esams - T249335

gerritbot added a comment.May 8 2020, 6:58 AM

Change 595088 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Puppetize max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595088

gerritbot added a comment.May 8 2020, 7:05 AM

Change 595088 merged by Vgutierrez:
[operations/puppet@production] ATS: Puppetize max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595088

gerritbot added a comment.May 8 2020, 7:12 AM

Change 595089 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Tune max_connections_in and max_connections_active_in on cp3050

https://gerrit.wikimedia.org/r/595089

gerritbot added a comment.May 8 2020, 7:16 AM

Change 595089 merged by Vgutierrez:
[operations/puppet@production] ATS: Tune max_connections_in and max_connections_active_in on cp3050

https://gerrit.wikimedia.org/r/595089

Stashbot added a comment.May 8 2020, 7:19 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-08T07:19:17Z] <vgutierrez> ats-tls restart on cp3050 and cp3052 (max_connections_active_in experiment) - T249335

gerritbot added a comment.May 8 2020, 8:12 AM

Change 595134 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in on esams

https://gerrit.wikimedia.org/r/595134

gerritbot added a comment.May 8 2020, 8:15 AM

Change 595134 merged by Vgutierrez:
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in on esams

https://gerrit.wikimedia.org/r/595134

Stashbot added a comment.May 8 2020, 8:20 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-08T08:20:23Z] <vgutierrez> rolling restart of ats-tls on esams - T249335

gerritbot added a comment.May 8 2020, 1:01 PM

Change 595172 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in globally

https://gerrit.wikimedia.org/r/595172

gerritbot added a comment.May 8 2020, 1:19 PM

Change 595172 merged by Vgutierrez:
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in globally

https://gerrit.wikimedia.org/r/595172

Stashbot added a comment.May 8 2020, 1:22 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-08T13:22:33Z] <vgutierrez> rolling restart of ats-tls on eqiad, codfw, ulsfo and eqsin - T249335

gerritbot added a comment.May 11 2020, 8:55 AM

Change 595484 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-1wm3

https://gerrit.wikimedia.org/r/595484

gerritbot added a comment.May 11 2020, 9:42 AM

Change 595484 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-1wm3

https://gerrit.wikimedia.org/r/595484

Stashbot added a comment.May 11 2020, 10:15 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-11T10:15:10Z] <vgutierrez> upload trafficserver 8.0.7-1wm3 to apt.wm.o (buster) - T242767 T249335

Stashbot mentioned this in T242767: EventStreams drops the connection after 15 minutes, which makes it unreliable.May 11 2020, 10:15 AM
Stashbot added a comment.May 11 2020, 1:44 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-11T13:44:15Z] <vgutierrez> upgrade ATS to 8.0.7-1wm4 in cp4032 - T249335

gerritbot added a comment.May 11 2020, 2:24 PM

Change 595533 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Relese 8.0.7-1wm4

https://gerrit.wikimedia.org/r/595533

gerritbot added a comment.May 12 2020, 6:33 AM

Change 595533 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-1wm4

https://gerrit.wikimedia.org/r/595533

Stashbot added a comment.May 12 2020, 6:56 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-12T06:56:54Z] <vgutierrez> upload trafficserver 8.0.7-1wm4 to apt.wm.o (buster) - T242767 T249335

Stashbot added a comment.May 12 2020, 10:30 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-12T10:30:17Z] <vgutierrez> upgrade trafficserver to version 8.0.7-1wm5 on cp4032 - T249335

Stashbot added a comment.May 12 2020, 10:55 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-12T10:55:24Z] <vgutierrez> upgrade trafficserver to version 8.0.7-1wm5 on cp5011 - T249335

gerritbot added a comment.May 12 2020, 12:35 PM

Change 595919 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595919

gerritbot added a comment.May 12 2020, 1:25 PM

Change 595919 merged by Vgutierrez:
[operations/puppet@production] ATS: Increase max_connections_in and max_connections_active_in

https://gerrit.wikimedia.org/r/595919

Stashbot added a comment.May 12 2020, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-12T13:33:39Z] <vgutierrez> rolling upgrade of ATS to version 8.0.7-1wm5 - T249335

gerritbot added a comment.May 13 2020, 11:05 AM

Change 596179 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.7-1wm6

https://gerrit.wikimedia.org/r/596179

gerritbot added a comment.May 13 2020, 1:26 PM

Change 596179 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.7-1wm6

https://gerrit.wikimedia.org/r/596179

Stashbot mentioned this in T251537: ATS: Add the ability to check if origin server responses can be cached and their lifetime to the Lua plugin.May 13 2020, 2:35 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-13T14:35:19Z] <vgutierrez> upload trafficserver 8.0.7-1wm6 to apt.wm.o (buster) - T249335 T251537

gerritbot added a comment.May 13 2020, 3:45 PM

Change 596238 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] trafficserver_exporter: Track throttled active connections

https://gerrit.wikimedia.org/r/596238

gerritbot added a comment.May 14 2020, 1:08 PM

Change 596238 merged by Vgutierrez:
[operations/puppet@production] trafficserver_exporter: Track throttled active connections

https://gerrit.wikimedia.org/r/596238

gerritbot added a comment.May 20 2020, 8:39 AM

Change 597473 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable KA for POST requests on esams

https://gerrit.wikimedia.org/r/597473

gerritbot added a comment.May 20 2020, 9:43 AM

Change 597473 merged by Vgutierrez:
[operations/puppet@production] ATS: Disable KA for POST requests on esams

https://gerrit.wikimedia.org/r/597473

Stashbot added a comment.May 20 2020, 9:43 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-20T09:43:55Z] <vgutierrez> disable KA for POST/PUT requests on esams - T249335

Stashbot added a comment.May 20 2020, 10:28 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-20T10:27:59Z] <vgutierrez> rolling restart of ats-tls in text@esams - T249335

Stashbot added a comment.Jun 5 2020, 7:52 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-05T07:52:46Z] <vgutierrez> rolling restart of ats-tls - T249335

gerritbot added a comment.Jun 9 2020, 9:54 AM

Change 588954 merged by Vgutierrez:
[operations/puppet@production] Revert "ATS: Re-enable parent proxies on ats-tls"

https://gerrit.wikimedia.org/r/588954

gerritbot added a comment.Jun 12 2020, 1:19 PM

Change 587516 abandoned by Vgutierrez:
ATS: Increase to 2 the number of accept_threads in cp3052 and cp3056

https://gerrit.wikimedia.org/r/587516

gerritbot added a comment.Jul 9 2020, 10:51 AM

Change 610764 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/610764

gerritbot added a comment.Jul 9 2020, 10:59 AM

Change 610764 merged by Vgutierrez:
[operations/puppet@production] ATS: Disable res_track_memory in cp1085

https://gerrit.wikimedia.org/r/610764

ArielGlenn removed a project: Patch-For-Review.Sep 28 2020, 12:33 PM
Krinkle moved this task from Active investigation to Awaiting report on the Wikimedia-Incident board.Feb 17 2021, 4:51 AM
BBlack edited projects, added Traffic-Icebox; removed Traffic.Sep 1 2021, 11:21 PM
BBlack subscribed.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

BBlack moved this task from Backlog to Close or Untag? on the Traffic-Icebox board.Apr 13 2022, 8:28 PM
Krinkle assigned this task to Vgutierrez.EditedApr 21 2022, 7:19 PM
Krinkle subscribed.

I believe this was resolved since and/or obsoleted by HAProxy, is that right?

BCornwall closed this task as Invalid.Sep 19 2022, 8:54 PM
BCornwall subscribed.

I believe this to be the case. From https://wikitech.wikimedia.org/wiki/Caching_overview#2022:

In April 2022, we replaced ATS with HAProxy for TLS termination and HTTP2 (T290005). This changed the stack to: HAProxy for TLS termination, Varnish frontend, and ATS backend.

Since ats-tls is no longer part of the stack and this ticket has become quite stale, I'm closing.

gerritbot added a comment.Oct 28 2022, 8:27 AM

Change 590993 abandoned by Vgutierrez:

[operations/debs/trafficserver@master] Release 8.0.7-rc0-1wm3asan

Reason:

https://gerrit.wikimedia.org/r/590993

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL