Page MenuHomePhabricator
Create Task
Maniphest T231287

Investigate HTTP/2 limits on trafficserver
Closed, ResolvedPublic
Actions

Description

ATS TLS termination instance is showing the following errors:

Aug 27 09:45:10 cp5001 traffic_manager[29456]: [Aug 27 09:45:10.996] {0x2afc108ce700} ERROR: HTTP/2 connection error client_ip=REDACTED session_id=287952 stream_id=627 recv priority too frequent priority changes

As an initial attempt to fix the issue, https://github.com/apache/trafficserver/commit/aa319a461c8326724ea10a42ae474fcdac7fc849 has been backported but the errors are still showing up after upgrading ATS to 8.0.5-1wm3.

It looks like the default HTTP/2 limits might need some tuning

Details

SubjectRepoBranchLines +/-
ATS: Disable HTTP/2 max priority frames limit on cp5001operations/puppetproduction+8 -0
ATS: Allow configuring HTTP/2 settingsoperations/puppetproduction+51 -0
Release 8.0.5-1wm4operations/debs/trafficservermaster+82 -32
Customize query in gerrit

Event Timeline

Vgutierrez created this task.Aug 27 2019, 9:48 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 27 2019, 9:48 AM
• ema moved this task from Backlog to TLS on the Traffic board.Aug 27 2019, 10:06 AM
Vgutierrez added a comment.Aug 27 2019, 11:25 AM

Triggering the issue is relatively easy browsing https://maps.wikimedia.org with Chrome 76:

t=264968 [st=29427]  HTTP2_SESSION_RECV_GOAWAY
                     --> active_streams = 22
                     --> debug_data = ""
                     --> error_code = "11 (ENHANCE_YOUR_CALM)"
                     --> last_accepted_stream_id = 1207
                     --> unclaimed_streams = 0
Vgutierrez updated the task description. (Show Details)Aug 27 2019, 11:29 AM
Stashbot added a comment.Aug 27 2019, 1:43 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-27T13:43:33Z] <vgutierrez> repool cp5001 - T231287

Stashbot added a comment.Aug 27 2019, 2:38 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-27T14:38:33Z] <vgutierrez> depool cp5001 - T231287

Vgutierrez added a comment.Aug 27 2019, 2:43 PM

Further analysis shows that actually ATS is rate limiting PRIORITY frames even when they are disabled:

proxy.config.http2.stream_priority_enabled: 0
proxy.config.http2.max_priority_frames_per_minute: 120

taking into account that while stream_priority_enabled is disabled, we could safely avoid rate limiting priority frames without being concerned with CVE-2019-9513, I think we could patch ATS to avoid this issue

gerritbot added a comment.Aug 27 2019, 3:19 PM

Change 532723 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/debs/trafficserver@master] Release 8.0.5-1wm4

https://gerrit.wikimedia.org/r/532723

gerritbot added a project: Patch-For-Review.Aug 27 2019, 3:19 PM
Vgutierrez added a comment.Aug 28 2019, 5:34 AM

After some discussion with upstream developers, https://github.com/apache/trafficserver/pull/5888 has been submitted and it's been included in https://gerrit.wikimedia.org/r/532723

Vgutierrez triaged this task as Medium priority.Aug 28 2019, 5:35 AM
gerritbot added a comment.Aug 28 2019, 6:02 AM

Change 532852 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Allow configuring HTTP/2 limits

https://gerrit.wikimedia.org/r/532852

gerritbot added a comment.Aug 28 2019, 6:02 AM

Change 532853 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Disable HTTP/2 max priority frames limit

https://gerrit.wikimedia.org/r/532853

gerritbot added a comment.Aug 28 2019, 7:33 AM

Change 532723 merged by Vgutierrez:
[operations/debs/trafficserver@master] Release 8.0.5-1wm4

https://gerrit.wikimedia.org/r/532723

Stashbot added a comment.Aug 28 2019, 8:10 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-28T08:10:25Z] <vgutierrez> uploaded trafficserver-8.0.5-1wm4 to apt.wikimedia.org (stretch) - T231287

Stashbot added a comment.Aug 28 2019, 9:56 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-28T09:56:26Z] <vgutierrez> upgrading trafficserver on cp5001 to version 8.0.5-1wm4 - T231287

Stashbot added a comment.Aug 28 2019, 9:58 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-28T09:58:08Z] <vgutierrez> repooling cp5001 - T231287

gerritbot added a comment.Aug 28 2019, 11:24 AM

Change 532852 merged by Vgutierrez:
[operations/puppet@production] ATS: Allow configuring HTTP/2 settings

https://gerrit.wikimedia.org/r/532852

gerritbot added a comment.Aug 28 2019, 11:28 AM

Change 532853 merged by Vgutierrez:
[operations/puppet@production] ATS: Disable HTTP/2 max priority frames limit on cp5001

https://gerrit.wikimedia.org/r/532853

Vgutierrez closed this task as Resolved.Aug 28 2019, 11:28 AM
Vgutierrez claimed this task.
DannyS712 removed a project: Patch-For-Review.Nov 18 2019, 11:37 AM
DannyS712 subscribed.

[batch] remove patch for review tag from resolved tasks

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits