Page MenuHomePhabricator
Create Task
Maniphest T244464

Investigate side-effects of enabling KA between ats-tls and varnish-fe
Closed, ResolvedPublic
Actions

Description

Right now ats-tls doesn't try to reuse connections between itself and varnish-fe, resulting on a big amount of connections between these two layers, this has been like this for a very long time and we are not longer sure of the side effects.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+13 -0ATS: Enable parent proxies in cp5006
operations/puppetproduction+1 -14ATS: Disable parent proxies globally
operations/puppetproduction+3 -2ATS: Disable parent proxies on ulsfo
operations/puppetproduction+5 -22ATS: Enable KA between ats-tls and varnish-fe globally
operations/puppetproduction+0 -12cache: revert Connection:KA probe experiment on cp4026
operations/puppetproduction+16 -0cache: use Connection:KA for varnish-ats checks
operations/debs/trafficserver8.0.6+41 -0Release 8.0.5-1wm16
operations/puppetproduction+4 -4ATS: Extend KA experiment between ats-tls and varnish-fe to all ulsfo
operations/debs/trafficservermaster+41 -0Release 8.0.5-1wm16
operations/puppetproduction+3 -3ATS: Test KA between ats-tls and varnish-fe on cp4031 + ATS 8.0.5-1wm16
operations/puppetproduction+3 -4Revert "ATS: Test KA on cp4031 whilst parent proxies are disabled"
operations/puppetproduction+4 -3ATS: Test KA on cp4031 whilst parent proxies are disabled
operations/puppetproduction+3 -3Revert "Revert "ATS: Disable KA on cp4031""
operations/puppetproduction+3 -3Revert "ATS: Disable KA on cp4031"
operations/puppetproduction+1 -1ATS: Don't assume that http_settings is mandatory on ats-tls profile
operations/puppetproduction+3 -3ATS: Disable KA on cp4031
operations/puppetproduction+4 -0varnish: Sync idle timeout with ats-tls on cp4031
operations/puppetproduction+26 -4ATS: Avoid hardcoding Connection: Close on ats-tls when keepalive is ON
operations/puppetproduction+0 -0Revert "ATS: Enable KeepAlive for the whole caching cluster in ulsfo"
operations/puppetproduction+1 -1ATS: Log ConnReuse on ats-tls
operations/puppetproduction+0 -0ATS: Enable KeepAlive for the whole caching cluster in ulsfo
operations/puppetproduction+1 -1ATS: Allow server session sharing by ip on ats-tls in cp4031
operations/puppetproduction+28 -22ATS: Allow configuring via hiera server session sharing settings
operations/puppetproduction+12 -0ATS: Enable KA between ats-tls and varnish-fe on cp4031
operations/puppetproduction+33 -28ATS: Allow configuring via hiera KA against origin servers
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Feb 6 2020, 9:43 AM

Change 570594 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Allow configuring via hiera KA against origin servers

https://gerrit.wikimedia.org/r/570594

gerritbot added a project: Patch-For-Review.Feb 6 2020, 9:43 AM
Vgutierrez triaged this task as Medium priority.Feb 6 2020, 9:58 AM
Vgutierrez moved this task from Triage to Caching on the Traffic board.
gerritbot added a comment.Feb 6 2020, 10:01 AM

Change 570594 merged by Vgutierrez:
[operations/puppet@production] ATS: Allow configuring via hiera KA against origin servers

https://gerrit.wikimedia.org/r/570594

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2020, 10:10 AM
gerritbot added a comment.Feb 6 2020, 10:14 AM

Change 570599 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Enable KA between ats-tls and varnish-fe on cp4031

https://gerrit.wikimedia.org/r/570599

gerritbot added a project: Patch-For-Review.Feb 6 2020, 10:14 AM
gerritbot added a comment.Feb 6 2020, 10:18 AM

Change 570599 merged by Vgutierrez:
[operations/puppet@production] ATS: Enable KA between ats-tls and varnish-fe on cp4031

https://gerrit.wikimedia.org/r/570599

Stashbot added a comment.Feb 6 2020, 10:19 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-06T10:19:20Z] <vgutierrez> Enabling HTTP keepalive between ats-tls and varnish-frontend on cp4031 - T244464

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2020, 11:10 AM
gerritbot added a comment.Feb 6 2020, 12:39 PM

Change 570622 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Allow configuring via hiera server session sharing settings

https://gerrit.wikimedia.org/r/570622

gerritbot added a project: Patch-For-Review.Feb 6 2020, 12:39 PM
ayounsi removed a subscriber: ayounsi.Feb 6 2020, 12:46 PM
gerritbot added a comment.Feb 6 2020, 1:09 PM

Change 570622 merged by Vgutierrez:
[operations/puppet@production] ATS: Allow configuring via hiera server session sharing settings

https://gerrit.wikimedia.org/r/570622

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2020, 1:10 PM
gerritbot added a comment.Feb 6 2020, 1:12 PM

Change 570633 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Allow server session sharing by ip on ats-tls in cp4031

https://gerrit.wikimedia.org/r/570633

gerritbot added a project: Patch-For-Review.Feb 6 2020, 1:12 PM
gerritbot added a comment.Feb 6 2020, 1:17 PM

Change 570633 merged by Vgutierrez:
[operations/puppet@production] ATS: Allow server session sharing by ip on ats-tls in cp4031

https://gerrit.wikimedia.org/r/570633

Stashbot added a comment.Feb 6 2020, 1:22 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-06T13:22:32Z] <vgutierrez> Enable server session sharing on ats-tls in cp4031 - T244464

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2020, 2:10 PM
gerritbot added a comment.Feb 10 2020, 2:25 PM

Change 571291 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Enable KeepAlive for the whole caching cluster in ulsfo

https://gerrit.wikimedia.org/r/571291

gerritbot added a project: Patch-For-Review.Feb 10 2020, 2:25 PM
gerritbot added a comment.Feb 10 2020, 2:31 PM

Change 571294 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Log ConnReuse on ats-tls

https://gerrit.wikimedia.org/r/571294

gerritbot added a comment.Feb 10 2020, 2:50 PM

Change 571291 merged by Vgutierrez:
[operations/puppet@production] ATS: Enable KeepAlive for the whole caching cluster in ulsfo

https://gerrit.wikimedia.org/r/571291

gerritbot added a comment.Feb 10 2020, 2:51 PM

Change 571294 merged by Vgutierrez:
[operations/puppet@production] ATS: Log ConnReuse on ats-tls

https://gerrit.wikimedia.org/r/571294

Stashbot added a comment.Feb 10 2020, 2:52 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T14:52:13Z] <vgutierrez> rolling restart of ats-tls in ulsfo - T244464

Maintenance_bot removed a project: Patch-For-Review.Feb 10 2020, 3:11 PM
gerritbot added a comment.Feb 10 2020, 3:54 PM

Change 571311 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Avoid hardcoding Connection: Close on ats-tls when keepalive is ON

https://gerrit.wikimedia.org/r/571311

gerritbot added a project: Patch-For-Review.Feb 10 2020, 3:54 PM
gerritbot added a comment.Feb 11 2020, 7:28 AM

Change 571459 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Revert "ATS: Enable KeepAlive for the whole caching cluster in ulsfo"

https://gerrit.wikimedia.org/r/571459

gerritbot added a comment.Feb 11 2020, 10:04 AM

Change 571459 merged by Vgutierrez:
[operations/puppet@production] Revert "ATS: Enable KeepAlive for the whole caching cluster in ulsfo"

https://gerrit.wikimedia.org/r/571459

Stashbot added a comment.Feb 11 2020, 10:07 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-11T10:07:54Z] <vgutierrez> rolling restart of ats-tls in ulsfo - T244464

gerritbot added a comment.Feb 11 2020, 11:06 AM

Change 571472 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: use Connection:KA for varnish-ats checks

https://gerrit.wikimedia.org/r/571472

gerritbot added a comment.Feb 11 2020, 11:14 AM

Change 571311 merged by Vgutierrez:
[operations/puppet@production] ATS: Avoid hardcoding Connection: Close on ats-tls when keepalive is ON

https://gerrit.wikimedia.org/r/571311

Stashbot added a comment.Feb 11 2020, 11:20 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-11T11:20:20Z] <vgutierrez> ats-tls effectively reusing connections between ats-tls and varnish-fe on cp4031 - T244464

gerritbot added a comment.Feb 11 2020, 1:41 PM

Change 571483 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] varnish: Sync idle timeout with ats-tls on cp4031

https://gerrit.wikimedia.org/r/571483

gerritbot added a comment.Feb 11 2020, 2:16 PM

Change 571483 merged by Vgutierrez:
[operations/puppet@production] varnish: Sync idle timeout with ats-tls on cp4031

https://gerrit.wikimedia.org/r/571483

Stashbot added a comment.Feb 11 2020, 2:20 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-11T14:20:31Z] <vgutierrez> restart varnish-fe on cp4031 - T244464

gerritbot added a comment.Feb 11 2020, 5:05 PM

Change 571540 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable KA on cp4031

https://gerrit.wikimedia.org/r/571540

gerritbot added a comment.Feb 11 2020, 5:11 PM

Change 571540 merged by Vgutierrez:
[operations/puppet@production] ATS: Disable KA on cp4031

https://gerrit.wikimedia.org/r/571540

Stashbot added a comment.Feb 11 2020, 5:13 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-11T17:13:15Z] <vgutierrez> Disable KA on cp4031 - T244464

gerritbot added a comment.Feb 12 2020, 10:25 AM

Change 571688 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Revert "ATS: Disable KA on cp4031"

https://gerrit.wikimedia.org/r/571688

gerritbot added a comment.Feb 12 2020, 10:28 AM

Change 571688 merged by Vgutierrez:
[operations/puppet@production] Revert "ATS: Disable KA on cp4031"

https://gerrit.wikimedia.org/r/571688

Stashbot added a comment.Feb 12 2020, 10:32 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-12T10:32:21Z] <vgutierrez> Enable KA between ats-tls and varnish-fe on cp4031 - T244464

gerritbot added a comment.Feb 12 2020, 10:35 AM

Change 571690 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Don't assume that http_settings is mandatory on ats-tls profile

https://gerrit.wikimedia.org/r/571690

gerritbot added a comment.Feb 12 2020, 10:44 AM

Change 571690 merged by Vgutierrez:
[operations/puppet@production] ATS: Don't assume that http_settings is mandatory on ats-tls profile

https://gerrit.wikimedia.org/r/571690

gerritbot added a comment.Feb 12 2020, 1:11 PM

Change 571719 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Revert "Revert "ATS: Disable KA on cp4031""

https://gerrit.wikimedia.org/r/571719

gerritbot added a comment.Feb 12 2020, 1:13 PM

Change 571719 merged by Vgutierrez:
[operations/puppet@production] Revert "Revert "ATS: Disable KA on cp4031""

https://gerrit.wikimedia.org/r/571719

Stashbot added a comment.Feb 12 2020, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-12T13:15:52Z] <vgutierrez> disabling KA between ats-tls and varnish-fe on cp4031 - T244464

Vgutierrez changed the task status from Open to Stalled.Feb 12 2020, 2:03 PM

ATS is having issues handling properly the connect and the TTFB timeout when KA is enabled and parent proxies are being used (ats-tls). This has been reported to upstream as https://github.com/apache/trafficserver/issues/6415

gerritbot added a comment.Feb 12 2020, 3:15 PM

Change 571736 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Test KA on cp4031 whilst disable parent proxies are disabled

https://gerrit.wikimedia.org/r/571736

gerritbot added a comment.Feb 12 2020, 3:51 PM

Change 571736 merged by Vgutierrez:
[operations/puppet@production] ATS: Test KA on cp4031 whilst parent proxies are disabled

https://gerrit.wikimedia.org/r/571736

Stashbot added a comment.Feb 12 2020, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-12T15:56:16Z] <vgutierrez> Enable KA and disable parent proxies on cp4031 - T244464

gerritbot added a comment.Feb 12 2020, 5:07 PM

Change 571763 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Revert "ATS: Test KA on cp4031 whilst parent proxies are disabled"

https://gerrit.wikimedia.org/r/571763

gerritbot added a comment.Feb 12 2020, 5:09 PM

Change 571763 merged by Vgutierrez:
[operations/puppet@production] Revert "ATS: Test KA on cp4031 whilst parent proxies are disabled"

https://gerrit.wikimedia.org/r/571763

Stashbot added a comment.Feb 12 2020, 5:09 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-12T17:09:39Z] <vgutierrez> disabling KA between ats-tls and varnish-fe on cp4031 - T244464

gerritbot added a comment.Feb 13 2020, 7:00 AM

Change 571869 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.5-1wm16

https://gerrit.wikimedia.org/r/571869

Vgutierrez changed the task status from Stalled to Open.Feb 13 2020, 7:13 AM

After some investigations, it looks like PR 5811 from upstream could fix the issue, I've backported it as part of 8.0.5-1wm16: https://gerrit.wikimedia.org/r/c/operations/debs/trafficserver/+/571869

gerritbot added a comment.Feb 13 2020, 7:42 AM

Change 571876 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Test KA between ats-tls and varnish-fe on cp4031 + ATS 8.0.5-1wm16

https://gerrit.wikimedia.org/r/571876

gerritbot added a comment.Feb 13 2020, 7:46 AM

Change 571876 merged by Vgutierrez:
[operations/puppet@production] ATS: Test KA between ats-tls and varnish-fe on cp4031 + ATS 8.0.5-1wm16

https://gerrit.wikimedia.org/r/571876

Stashbot added a comment.Feb 13 2020, 7:49 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-13T07:49:18Z] <vgutierrez> testing ATS 8.0.5-1wm16 + KA between ats-tls and varnish-fe in cp4031 - T244464

gerritbot added a comment.Feb 13 2020, 10:35 AM

Change 571869 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.5-1wm16

https://gerrit.wikimedia.org/r/571869

Stashbot added a comment.Feb 13 2020, 11:08 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-13T11:08:29Z] <vgutierrez> upload trafficserver 8.0.5-1wm16 to apt.wm.o (buster) - T244464

Vgutierrez added a comment.Feb 13 2020, 11:15 AM

The issue with timeouts and KeepAlive can be easily understood with a small environment using curl + ATS + httpbin.

  1. curl requests /delay/20, the request returns successfully after 20 seconds
  2. curl requests (again) /delay/20, the request returns successfully after 23 seconds
  3. httpbin sees the following requests:
time="2020-02-12T16:37:41.5787" status=200 method="GET" uri="/delay/20" size_bytes=507 duration_ms=20002.73
time="2020-02-12T16:37:52.2979" status=200 method="GET" uri="/delay/20" size_bytes=0 duration_ms=3717.91
time="2020-02-12T16:38:12.2721" status=200 method="GET" uri="/delay/20" size_bytes=507 duration_ms=20002.50

What's happening? on the second curl request, a timeout is triggered on ATS after ~3 seconds (the connect timeout in the test scenario). In our production environment this means that ats-be has been unnecessarily retrying every request that takes longer than 10 seconds (the connect timeout for ats-be). This has been happening since January 14th

Stashbot added a comment.Feb 13 2020, 11:18 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-13T11:18:21Z] <vgutierrez> rolling upgrade of ATS to version 8.0.5-1wm16 fleet wide - T244464

Vgutierrez added a comment.Feb 13 2020, 1:43 PM

the issue described above it should be fixed almost everywhere:

===== NODE GROUP =====
(76) cp[2001-2002,2004-2008,2010-2014,2016-2020,2022-2026].codfw.wmnet,cp[1075-1090].eqiad.wmnet,cp[5001-5012].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4021-4025,4027-4031].ulsfo.wmnet
----- OUTPUT of 'apt-cache policy...r|grep Installed' -----
  Installed: 8.0.5-1wm16
===== NODE GROUP =====

only 2 nodes on ulsfo that are running 8.0.6-rc0 don't have the patch applied

gerritbot added a comment.Feb 17 2020, 6:51 AM

Change 572515 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Extend KA experiment between ats-tls and varnish-fe to all ulsfo

https://gerrit.wikimedia.org/r/572515

gerritbot added a comment.Feb 17 2020, 10:15 AM

Change 572515 merged by Vgutierrez:
[operations/puppet@production] ATS: Extend KA experiment between ats-tls and varnish-fe to all ulsfo

https://gerrit.wikimedia.org/r/572515

Stashbot added a comment.Feb 17 2020, 10:20 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-17T10:20:41Z] <vgutierrez> rolling restart of ats-tls and varnish-fe on ulsfo to enable KA between them - T244464

gerritbot added a comment.Feb 19 2020, 8:05 AM

Change 573220 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@8.0.6] Release 8.0.5-1wm16

https://gerrit.wikimedia.org/r/573220

gerritbot added a comment.Feb 19 2020, 8:06 AM

Change 573220 merged by Vgutierrez:
[operations/debs/trafficserver@8.0.6] Release 8.0.5-1wm16

https://gerrit.wikimedia.org/r/573220

gerritbot added a comment.Feb 19 2020, 4:27 PM

Change 571472 merged by Ema:
[operations/puppet@production] cache: use Connection:KA for varnish-ats checks

https://gerrit.wikimedia.org/r/571472

gerritbot added a comment.Feb 19 2020, 5:07 PM

Change 573337 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: revert Connection:KA probe experiment on cp4026

https://gerrit.wikimedia.org/r/573337

gerritbot added a comment.Feb 19 2020, 5:10 PM

Change 573337 merged by Ema:
[operations/puppet@production] cache: revert Connection:KA probe experiment on cp4026

https://gerrit.wikimedia.org/r/573337

gerritbot added a comment.Mar 5 2020, 9:27 AM

Change 577198 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Enable KA between ats-tls and varnish-fe globally

https://gerrit.wikimedia.org/r/577198

gerritbot added a comment.Mar 5 2020, 10:04 AM

Change 577198 merged by Vgutierrez:
[operations/puppet@production] ATS: Enable KA between ats-tls and varnish-fe globally

https://gerrit.wikimedia.org/r/577198

Stashbot added a comment.Mar 5 2020, 10:14 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-05T10:14:30Z] <vgutierrez> Enable keep alive between ats-tls and varnish-fe globally - T244464

gerritbot added a comment.Mar 5 2020, 10:37 AM

Change 577210 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable parent proxies on ulsfo

https://gerrit.wikimedia.org/r/577210

gerritbot added a comment.Mar 5 2020, 11:06 AM

Change 577210 merged by Vgutierrez:
[operations/puppet@production] ATS: Disable parent proxies on ulsfo

https://gerrit.wikimedia.org/r/577210

Stashbot added a comment.Mar 5 2020, 11:10 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-05T11:10:33Z] <vgutierrez> Disable parent proxies on ats-tls in ulsfo - T244464

gerritbot added a comment.Mar 9 2020, 7:25 AM

Change 578182 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable parent proxies globally

https://gerrit.wikimedia.org/r/578182

gerritbot added a comment.Mar 9 2020, 10:03 AM

Change 578182 merged by Vgutierrez:
[operations/puppet@production] ATS: Disable parent proxies globally

https://gerrit.wikimedia.org/r/578182

Stashbot added a comment.Mar 9 2020, 10:04 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-09T10:04:55Z] <vgutierrez> disable parent proxies globally on ats-tls - T244464

Vgutierrez closed this task as Resolved.Mar 9 2020, 10:26 AM
Vgutierrez claimed this task.

KA between ats-tls and varnish-fe is working successfully and enabled globally in the caching cluster for text & upload

• Gilles mentioned this in T238494: 15% response start regression as of 2019-11-11 (Varnish->ATS).Apr 6 2020, 10:08 AM
gerritbot added a comment.Feb 25 2021, 8:06 AM

Change 666838 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Enable parent proxies in cp5006

https://gerrit.wikimedia.org/r/666838

gerritbot added a comment.Feb 25 2021, 8:11 AM

Change 666838 merged by Vgutierrez:
[operations/puppet@production] ATS: Enable parent proxies in cp5006

https://gerrit.wikimedia.org/r/666838

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL