Page MenuHomePhabricator

Quarry: HTTP 500 errors for all authenticated requests
Closed, ResolvedPublic

Description

Navigating to https://quarry.wmflabs.org/ or any other Quarry page fails with a 500 error if you have an authentication cookie set.

https://quarry-dev.wmflabs.org/ seems to be working fine.

Related Objects

Duplicates Merged Here
T278233: 500 server error

Event Timeline

Urbanecm_WMF triaged this task as Unbreak Now! priority.Mar 23 2021, 2:01 PM
Urbanecm_WMF subscribed.

Boldly triaging this as an UBN. And, issue confirmed:

image.png (181×954 px, 18 KB)

Quarry web is failing to connect to mysql, looking:

Mar 23 14:47:02 quarry-web-01 uwsgi-quarry-web[441]: pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'quarry-db-01.quarry.eqiad.wmflabs' ([Errno 111] Connection refused)")

Not sure if it's expected, but maridb is running only on localhost ip on quarry-db-01:

tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      18433/mysqld

Mentioned in SAL (#wikimedia-cloud) [2021-03-23T15:08:07Z] <dcaro> systemctl restart mariadb on quarry-db-01 brought it back to life (T278230)

It is worth mentioning that, at least for me, the service has returned to normal operation.