| Mvolz | |
| Dec 3 2024, 11:08 AM |
| F90543932: Screenshot From 2026-06-26 09-01-51.png | |
| Jun 26 2026, 8:27 AM |
| F90283796: Screenshot From 2026-06-24 13-35-20.png | |
| Jun 24 2026, 12:35 PM |
| F90283510: Screenshot From 2026-06-24 13-07-35.png | |
| Jun 24 2026, 12:33 PM |
As part of creating the SLO we're thinking about dependencies of citoid and was reminded that this most recent failure was not detected by monitoring.
How should monitoring change so it's detected?
Currently the probe checks health by accessing a wikipedia page, but historically this was example.com; this presumably wasn't triggered because as an internal IP citoid didn't have to go through url-downloader? Should we add back in monitoring using an external url?
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T358590 Citoid / zotero is unresponsive on eqiad due to url-downloader being down | |||
| Restricted Task | |||||
| Open | None | T381372 Improve monitoring in citoid so that url-downloader failures are detected |
Incident today!
100% of requests were getting 504s because we couldn't reach the url-downloader proxy due to a change in the IP of url-downloader. (T430053)
The citoid-sucess-ratio SLO measures this and should have alerted us, but it doesn't seem to have? So plausible more work needs to be done on this.
Hi, marking this as incident followup, feel free to change to a more scoped task if needed
Thanks, to follow up:
We had a significant decrease in citoid-success-ratio failures at 8 am (from about 90% down to ~70%, eyeballing it) coinciding when the codfw node was moved at occurred ~ 8am UTC. This was not detected by anyone.
At 11:48 UTC when the switchover for eqiad was deployed, we had a 0% success ratio (
On IRC we figured out the issue and the change was rolled back, resulting in effectively 20 minutes of downtime.
Two things:
In theory, I feel like this should have burned through the budget and triggered an alert to SREs and / or a task being created for this. This didn't happen - @elukey suggested maybe there's an issue with the SLO definition. We should follow up and make sure that 0% success rates alert SREs relatively quickly, however that ends up happening. Aside from elukey, is there anyone else we should include in this conversation?
In theory, if the initial decrease at 8 utc had been detected, we could have avoided the full downtime occuring at 11:48 utc.
Also, if I hadn't happen to have noticed the failure pretty much immediately, the downtime could have been considerably longer. So I think fixing the alerting system for these kinds of failures is fairly critical.
It seems the SLO alerting is not working for latency either, as evidenced in T430279: Large increase in citoid latency starting on June 25/ ~ 21 UTC - June 26 / ~13 UTC ?
We discussed yesterday with @RLazarus who would like to work with Editing-team and SRE-SLO working group to fix this
