Very long response headers cause a 502 response from the Toolforge front proxy
Open, Needs TriagePublicBUG REPORT
Actions

Assigned To

None

Authored By

	bd808
	Feb 2 2024, 5:26 PM

Description

In T356195#9509296, @So9q wrote:

To see the error in action try this url: https://topic-curator.toolforge.org/results?lang=en&subgraph=&qid=Q7889&limit=&prefix=&affix=&terms=videogame&terms=computer+games&terms=video+game&terms=computer+game&terms=videogames
Select all
Click submit
->

There are currently unknown/undocumented limits to the length of a header that can be sent back to a client via a Toolforge tool. There are at least two layers of Nginx between a Toolforge webservice and it's users: Internet → Toolforge front proxy → Kubernetes ingress → tool's webservice. The "from the Toolforge front proxy" in the title may only be that the front proxy is relaying a 502 response that is actually generated at the tool's Kubernetes ingress.

Memory is finite, so there does technically need to be some limit on buffered response sizes, but a) we should know what those limits are, and b) we should consider if we can raise them from the software's compiled defaults.

Related Objects

Mentioned Here: T356195: Request creation of topic-curator VPS project

Event Timeline

bd808 created this task.Feb 2 2024, 5:26 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 2 2024, 5:26 PM

bd808 updated the task description. (Show Details)Feb 2 2024, 5:27 PM

bd808 updated the task description. (Show Details)Feb 2 2024, 9:04 PM

I tested on my VPS in OVH and got:

upstream sent too big header while reading response header from upstream

with a default proxy configuration like so:

server {
    listen 8080;
    server_name ip-adress;

    location / {
        proxy_pass http://localhost:5000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

after adjusting to:

server {
    listen 8080;
    server_name ip-adress;

    location / {
        proxy_busy_buffers_size   512k;
        proxy_buffers   4 512k;
        proxy_buffer_size   256k;
        proxy_pass http://localhost:5000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

everything worked fine with ~130 QIDs in the header

see https://www.cyberciti.biz/faq/nginx-upstream-sent-too-big-header-while-reading-response-header-from-upstream/

	F41947001: image.png
	Feb 17 2024, 2:58 PM

Very long response headers cause a 502 response from the Toolforge front proxyOpen, Needs TriagePublicBUG REPORTActions

Description

Related Objects

Event Timeline

Very long response headers cause a 502 response from the Toolforge front proxy
Open, Needs TriagePublicBUG REPORT
Actions