Page MenuHomePhabricator

Citoid / zotero is unresponsive on eqiad
Closed, ResolvedPublic

Description

Requests to the citoid are repeatedly timing out with the error 504 gateway timeout.

Requests to both prod and staging servers of both citoid and zotero services, are timing out

i.e. curl -d '9791029801297' -H 'Content-Type: text/plain' https://staging.svc.eqiad.wmnet:4969/search
for zotero

https://en.wikipedia.org/api/rest_v1/data/citation/mediawiki/10.1038%2Fs41586-021-03470-x is an example of a request that timesout

For some reason this is not apparent on grafana in the 500 signal: https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1&refresh=5m&from=1708998234259&to=1709041434259

however a 503 signal is showing up on cxserver: https://grafana.wikimedia.org/d/F7rttgqmz/cxserver?orgId=1&refresh=30s&from=now-12h&to=now&viewPanel=15
500s

eqiad temporarily has been depooled