Page MenuHomePhabricator

Toolforge nginx/openresty timeout is too short
Closed, ResolvedPublic

Description

Reproduce:
Open a Listeria query that will take more than 15 seconds to run, e.g. https://listeria.toolforge.org/index.php?action=update&project=wikidata&page=Wikidata:Database_reports/to_delete/empty_category_items

Expected:

The page may load for more than 15 seconds without timeout. (The timeout should be increased to at least 120 seconds, preferably 600)

See also T249786: wikisense unusable due to "504 Gateway Time-out"

Event Timeline

The proxy_read_timeout setting at both the front proxy and the Kubernetes ingress is 3600s (1 hour).

Could that be related to nginx.ingress.kubernetes.io/proxy-send-timeout?

Could that be related to nginx.ingress.kubernetes.io/proxy-send-timeout?

Nah, that doesn't make sense.

This is unrelated to timeouts: https://listeria.toolforge.org/ also shows 504. I think it is down.

https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout

Syntax: proxy_send_timeout time;
Default:
proxy_send_timeout 60s;
Context: http, server, location

Sets a timeout for transmitting a request to the proxied server. The timeout is set only between two successive write operations, not for the transmission of the whole request. If the proxied server does not receive anything within this time, the connection is closed.

We don't have an custom tuning for that at either the Kubernetes ingress or the front proxy. That setting seems to be in-bound to the upstream service, but we could try it.

Mentioned in SAL (#wikimedia-cloud) [2020-09-30T18:25:36Z] <bstorm> restarting the webservice because it's not receiving requests from what I can tell T264219

nskaggs claimed this task.
nskaggs subscribed.

This seems to be fixed now. Please feel free to reopen if needed.