Page MenuHomePhabricator

Loading https://graphite.wikimedia.org/ throws an envoy error
Closed, ResolvedPublic

Description

Earlier today I tried to load https://graphite.wikimedia.org/ (I needed to know some key names from there). It redirected me to the idp, and then upstream connect error or disconnect/reset before headers. reset reason: connection termination appeared:

image.png (298×884 px, 18 KB)

@RLazarus investigated this in -operations, and advised to create a task (saying they've some logs).

Event Timeline

That Envoy error, in this case from graphite1004's TLS proxy, means that Graphite hung up on Envoy before sending a response.

I found this in journalctl -u uwsgi-graphite-web.service on graphite1004:

Oct 08 20:51:49 graphite1004 uwsgi-graphite-web[150868]: [pid: 150931|app: 0|req: 36707/98870] 10.64.16.149 (Urbanecm) {94 vars in 4095 bytes} [Fri Oct  8 20:51:49 2021] GET / => generated 962 bytes in 8 msecs (HTTP/1.1 200) 3 headers in 111 bytes (2 switches on core 0)
Oct 08 20:51:49 graphite1004 uwsgi-graphite-web[150868]: invalid request block size: 4205 (max 4096)...skip
Oct 08 20:51:49 graphite1004 uwsgi-graphite-web[150868]: invalid request block size: 4195 (max 4096)...skip
Oct 08 20:51:49 graphite1004 uwsgi-graphite-web[150868]: invalid request block size: 4205 (max 4096)...skip
Oct 08 20:51:50 graphite1004 uwsgi-graphite-web[150868]: invalid request block size: 4195 (max 4096)...skip

From https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html (ctrl-f "invalid request block size") it looks like we need to increase the buffer-size option for uWSGI, but I'll leave that for o11y folks to confirm.

I can't reproduce this at the moment on https://graphite.wikimedia.org, can you @Urbanecm ?

Yes. Loading https://graphite.wikimedia.org/ still shows upstream connect error or disconnect/reset before headers. reset reason: connection termination on my end (after logging in via IDP).

Change 729903 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: bump uwsgi buffer space

https://gerrit.wikimedia.org/r/729903

Change 729903 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: bump uwsgi buffer space

https://gerrit.wikimedia.org/r/729903

Mentioned in SAL (#wikimedia-operations) [2021-10-11T08:06:54Z] <godog> bounce uwsgi on graphite hosts to bump request size limit - T292877

Change 729904 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: fix buffer-size option name

https://gerrit.wikimedia.org/r/729904

Change 729904 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: fix buffer-size option name

https://gerrit.wikimedia.org/r/729904

fgiunchedi claimed this task.

Confirmed working by @Urbanecm