Maniphest T198152

Size of headers processed by varnish?
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Nuria
	Jun 25 2018, 11:05 PM

Description

Recent instances of too-long-user-agents made me wonder if we have a limit to the length of the headers processed via varnish, seems like we should .

See for example a recent UA (see parent task also for counts of how many where these long on a given hour)

Mozilla/5.0 (X11; Linux x86_64_128) AppleWebKit/11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111.111111111111111111111111111111111111111111111111111111111111111111111111 (KHTML, like Gecko) Linux/222222222222222222222222222222222222222222222222222222222222222222222222....

continues to 2035 characters

Details

	Subject	Repo	Branch	Lines +/-
	Revert "Increase client header buffer size, to allow for larger sets of cookies."	operations/puppet	production	+0 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		JAllemandou	T197281 Fix failing webrequest hours (upload and text 2018-06-14-11)
		Resolved		• ema	T198152 Size of headers processed by varnish?

Event Timeline

• Nuria created this task.Jun 25 2018, 11:05 PM

Restricted Application added a project: SRE. · View Herald TranscriptJun 25 2018, 11:05 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• Nuria edited subscribers, added: • ema, BBlack; removed: Aklapper, MusikAnimal.Jun 25 2018, 11:05 PM

elukey subscribed.Jun 26 2018, 6:21 AM

• ema triaged this task as Medium priority.Jun 29 2018, 8:18 AM

• ema moved this task from Backlog to Caching on the Traffic board.

Both varnish and nginx limit the maximum request header length to 8k by default. We have set nginx's limit to 16k, while leaving the default on varnish untouched.

It seems like a good idea to ensure that the two settings match, either by increasing http_req_hdr_len on varnish if we do indeed need to accept request headers up to 16k in length, or by decreasing large_client_header_buffers on nginx if that is not the case.

Ya, 8k seems quite a bit, not sure why would we need more than that in either end.

• Vvjjkkii renamed this task from Size of headers processed by varnish? to 3aaaaaaaaa.Jul 1 2018, 1:01 AM

• Vvjjkkii removed JAllemandou as the assignee of this task.

• Vvjjkkii raised the priority of this task from Medium to High.

• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).

• Vvjjkkii updated the task description. (Show Details)

• Vvjjkkii added a subscriber: JAllemandou.

JJMC89 renamed this task from 3aaaaaaaaa to Size of headers processed by varnish?.Jul 1 2018, 3:10 AM

JJMC89 assigned this task to JAllemandou.

JJMC89 lowered the priority of this task from High to Medium.

JJMC89 removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.

JJMC89 updated the task description. (Show Details)

JJMC89 edited subscribers, added: Aklapper; removed: JAllemandou.

elukey removed JAllemandou as the assignee of this task.Jul 2 2018, 5:56 AM

elukey added a subscriber: JAllemandou.

• fdans moved this task from Incoming to Radar on the Analytics board.Jul 2 2018, 4:18 PM

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:07 PM

In T198152#4324161, @ema wrote:

Both varnish and nginx limit the maximum request header length to 8k by default. We have set nginx's limit to 16k, while leaving the default on varnish untouched.

So I've tried increasing http_req_hdr_len on Varnish, but then found that also Apache at the application layer (Mediawiki) imposes a limit of 8192 on the maximum request header length.

We changed the nginx setting 8 years ago: to "allow for larger sets of cookies": https://phabricator.wikimedia.org/rOPUPf738b5ea1db6a1608cc0dbd41a8d887002fdba49. I'm not sure if the change ever worked across the whole stack, it definitely does not work now though. I think we can safely revert that change at this point and have 8192 as the maximum length for a single request header line (name+value).

Change 520235 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Revert "Increase client header buffer size, to allow for larger sets of cookies."

https://gerrit.wikimedia.org/r/520235

gerritbot added a project: Patch-For-Review.Jul 2 2019, 1:20 PM

Change 520235 merged by Ema:
[operations/puppet@production] Revert "Increase client header buffer size, to allow for larger sets of cookies."

https://gerrit.wikimedia.org/r/520235

Maintenance_bot removed a project: Patch-For-Review.Jul 2 2019, 3:10 PM