| Urbanecm | |
| Oct 8 2021, 9:05 PM |
| F34679485: image.png | |
| Oct 8 2021, 9:05 PM |
Earlier today I tried to load https://graphite.wikimedia.org/ (I needed to know some key names from there). It redirected me to the idp, and then upstream connect error or disconnect/reset before headers. reset reason: connection termination appeared:
@RLazarus investigated this in -operations, and advised to create a task (saying they've some logs).
That Envoy error, in this case from graphite1004's TLS proxy, means that Graphite hung up on Envoy before sending a response.
I found this in journalctl -u uwsgi-graphite-web.service on graphite1004:
Oct 08 20:51:49 graphite1004 uwsgi-graphite-web[150868]: [pid: 150931|app: 0|req: 36707/98870] 10.64.16.149 (Urbanecm) {94 vars in 4095 bytes} [Fri Oct 8 20:51:49 2021] GET / => generated 962 bytes in 8 msecs (HTTP/1.1 200) 3 headers in 111 bytes (2 switches on core 0)
Oct 08 20:51:49 graphite1004 uwsgi-graphite-web[150868]: invalid request block size: 4205 (max 4096)...skip
Oct 08 20:51:49 graphite1004 uwsgi-graphite-web[150868]: invalid request block size: 4195 (max 4096)...skip
Oct 08 20:51:49 graphite1004 uwsgi-graphite-web[150868]: invalid request block size: 4205 (max 4096)...skip
Oct 08 20:51:50 graphite1004 uwsgi-graphite-web[150868]: invalid request block size: 4195 (max 4096)...skipFrom https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html (ctrl-f "invalid request block size") it looks like we need to increase the buffer-size option for uWSGI, but I'll leave that for o11y folks to confirm.
I can't reproduce this at the moment on https://graphite.wikimedia.org, can you @Urbanecm ?
Yes. Loading https://graphite.wikimedia.org/ still shows upstream connect error or disconnect/reset before headers. reset reason: connection termination on my end (after logging in via IDP).
Change 729903 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] graphite: bump uwsgi buffer space
Change 729903 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: bump uwsgi buffer space
Mentioned in SAL (#wikimedia-operations) [2021-10-11T08:06:54Z] <godog> bounce uwsgi on graphite hosts to bump request size limit - T292877
Change 729904 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] graphite: fix buffer-size option name
Change 729904 merged by Filippo Giunchedi:
[operations/puppet@production] graphite: fix buffer-size option name