Page MenuHomePhabricator

Beta cluster unreachable when interacting with sessions (Error: 502, Next Hop Connection Failed)
Closed, ResolvedPublic

Description

I can’t open beta cluster wikis (tested: enwiki and wikidatawiki) in my browser – after two minutes, the following message appears:

Screenshot 2021-06-04 at 10-49-46 Wikimedia Error.png (670×228 px, 22 KB)

Screenshot 2021-06-04 at 10-49-53 Wikimedia Error.png (658×139 px, 16 KB)

Curiously enough, it works in a private window, and also via the command line. However, trying to load Special:UserLogin in a private window or on the command line also times out:

$ time curl -i https://en.wikipedia.beta.wmflabs.org/wiki/Special:UserLogin
HTTP/2 503 
date: Fri, 04 Jun 2021 08:54:42 GMT
server: Varnish
content-type: text/html; charset=utf-8
age: 0
x-cache: deployment-cache-text06 int
x-cache-status: int-front
server-timing: cache;desc="int-front", host;desc="deployment-cache-text06"
report-to: { "group": "wm_nel", "max_age": 86400, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
nel: { "report_to": "wm_nel", "max_age": 86400, "failure_fraction": 0.05, "success_fraction": 0.0}
set-cookie: WMF-Last-Access=04-Jun-2021;Path=/;HttpOnly;secure;Expires=Tue, 06 Jul 2021 00:00:00 GMT
set-cookie: WMF-Last-Access-Global=04-Jun-2021;Path=/;Domain=.wikipedia.beta.wmflabs.org;HttpOnly;secure;Expires=Tue, 06 Jul 2021 00:00:00 GMT
x-client-ip: 0.0.0.0
cache-control: private, s-maxage=0, max-age=0, must-revalidate
set-cookie: GeoIP=DE:BE:Berlin:52.52:13.40:v4; Path=/; secure; Domain=.beta.wmflabs.org
content-length: 1810

<!DOCTYPE html>
<html lang="en">
<meta charset="utf-8">
<title>Wikimedia Error</title>
<style>
* { margin: 0; padding: 0; }
body { background: #fff; font: 15px/1.6 sans-serif; color: #333; }
.content { margin: 7% auto 0; padding: 2em 1em 1em; max-width: 640px; }
.footer { clear: both; margin-top: 14%; border-top: 1px solid #e5e5e5; background: #f9f9f9; padding: 2em 0; font-size: 0.8em; text-align: center; }
img { float: left; margin: 0 2em 2em 0; }
a img { border: 0; }
h1 { margin-top: 1em; font-size: 1.2em; }
.content-text { overflow: hidden; overflow-wrap: break-word; word-wrap: break-word; -webkit-hyphens: auto; -moz-hyphens: auto; -ms-hyphens: auto; hyphens: auto; }
p { margin: 0.7em 0 1em 0; }
a { color: #0645ad; text-decoration: none; }
a:hover { text-decoration: underline; }
code { font-family: sans-serif; }
.text-muted { color: #777; }
</style>
<div class="content" role="main">
<a href="https://www.wikimedia.org"><img src="https://www.wikimedia.org/static/images/wmf-logo.png" srcset="https://www.wikimedia.org/static/images/wmf-logo-2x.png 2x" alt="Wikimedia" width="135" height="101">
</a>
<h1>Error</h1>
<div class="content-text">
<p>Our servers are currently under maintenance or experiencing a technical problem.

Please <a href="" title="Reload this page" onclick="window.location.reload(false); return false">try again</a> in a few&nbsp;minutes.</p>

<p>See the error message at the bottom of this page for more&nbsp;information.</p>
</div>
</div>
<div class="footer"><p>If you report this error to the Wikimedia System Administrators, please include the details below.</p><p class="text-muted"><code>Request from 95.90.239.99 via deployment-cache-text06 frontend, Varnish XID 35262005<br>Error: 503, Backend fetch failed at Fri, 04 Jun 2021 08:54:42 GMT</code></p>
</div>
</html>

real    1m36,036s
user    0m0,022s
sys     0m0,010s

It seems like there might be an issue with sessions?