Steps to replicate the issue (include links if applicable):
What happens?:
503 Service Unavailable No server is available to handle this request.
What should have happened instead?:
A normal beta wikipedia page
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-releng) [2025-03-27T16:05:24Z] <bd808> sudo systemctl restart varnish-frontend on deployment-cache-text08 (T390209)
Comment Actions
Boy oh boy do we need more runbooks for beta cluster.
- I went to https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Help looking for a CDN troubleshooting link.
- We link to https://wikitech.wikimedia.org/wiki/Caching_overview which is close to useless. It did remind me that we have a stack that is something like haproxy → varnish → ATS → backend(s) with the first 3 hops all on the deployment-cache-text* nodes.
- curl -v --insecure -H 'wikidata.beta.wmflabs.org' https://localhost gave me the 503. This was expected and checked at the outermost HAProxy layer.
- Now how do I check at the next step in, varnish? Off to https://wikitech.wikimedia.org/wiki/Varnish sniffing for clues.
- What does sudo varnishncsa -n frontend say? Nothing at all... even while I'm refreshing and getting new 503s from my browser. Seems like a problem.
- sudo systemctl restart varnish-frontend turns it off and on again as Roy and Moss taught me to try oh so many years ago.
- sudo varnishncsa -n frontend starts showing stuff!
- https://wikidata.beta.wmflabs.org redirects me to https://wikidata.beta.wmflabs.org/wiki/Wikidata:Main_Page!
[16:06] < bd808> \o/ Restarting varnish seems to have done the trick