Page MenuHomePhabricator

BackendResponseTime on beta cluster is high when logged in
Open, Needs TriagePublic

Description

On production, wgBackendResponseTime is around 500ms but on beta cluster it's around 4000ms - there must be something wrong. This only happens if you are logged in to an account, and everything works fine in incognito mode.

Event Timeline

I can't see anything immediately obvious and new in the logs, and the backends seem as low on resources as normal… perhaps we should just turn it off and on again?

Today I found I even could not access beta cluster...

Request from [redacted] via deployment-cache-text07 deployment-cache-text07, Varnish XID 30920951
Error: 503, Backend fetch failed at Wed, 30 Nov 2022 06:27:49 GMT

Today I found I even could not access beta cluster...

Request from [redacted] via deployment-cache-text07 deployment-cache-text07, Varnish XID 30920951
Error: 503, Backend fetch failed at Wed, 30 Nov 2022 06:27:49 GMT

Looks good to me now.
(FYI, I heard someone else also encountered "504 Gateway Timeout" error)