I'm getting 503 errors from beta labs when I visit http://en.wikipedia.beta.wmflabs.org/w/index.php?title=Topic:S2kt0b7x2jrv7gw0&action=compare-postsummary-revisions&topicsummary_newRevision=s3ems8b3lyhnoafm
I can't find these errors in deployment-bastion:/data/project/logs where I would expect to find something. I notice some weirdness:
- No fatal.log since August 19 (maybe it's all in HHVM logs?)
- No hhvm log
- No apache log (but maybe this isn't relevant with hhvm?)
- The error message and its HTTP header gives no indication what server handled it, so you have to visit all the front-end servers.
I think I found the problem in deployment-mediawiki02:/var/log/hhvm/error.log . Maybe this bug more succinctly is "Server hhvm logs are not brought to deployment-bastion". FWIW there is a fluorine:/a/mw-log/hhvm.log for production logs.
Here's the complete error headers and text of error message. To be clear, I'm not reporting this particular 503 error, I'm reporting problems *logging* 503 errors.
HTTP/1.1 503 Service Unavailable^M
Content-Type: text/html; charset=utf-8^M
X-Varnish: 281336368, 1116870927^M
Via: 1.1 varnish, 1.1 varnish^M
Date: Mon, 20 Oct 2014 19:59:36 GMT^M
X-Cache: deployment-cache-text02 miss (0), deployment-cache-text02 frontend miss (0)^M
Cache-Control: private, s-maxage=0, max-age=0, must-revalidate^M
Set-Cookie: GeoIP=::::v4; Path=/; Domain=.wmflabs.org
Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.
If you report this error to the Wikimedia System Administrators, please include the details below.
Request: GET http://en.wikipedia.beta.wmflabs.org/w/index.php?title=Topic:S2kt0b7x2jrv7gw0&action=compare-postsummary-revisions&topicsummary_newRevision=s3ems8b3lyhnoafm, from 127.0.0.1 via deployment-cache-text02 deployment-cache-text02 ([127.0.0.1]:3128), Varnish XID 281335977
Forwarded for: 184.108.40.206, 127.0.0.1
Error: 503, Service Unavailable at Mon, 20 Oct 2014 19:57:01 GMT