SRE had two recent incidents reported where citoid was involved:
- T430279: Large increase in citoid latency starting on June 25/ ~ 21 UTC - June 26 / ~13 UTC
- {https://phabricator.wikimedia.org/T430041}
In both cases we were unable to troubleshoot properly because citoid service does not provide required observability, we could only check platform level metrics (kubernetes pods health) and had to defer to 'blind' mitigations (eg restarting citoid and zotero).
This makes mitigation slow and rootcausing very difficult. In this case, issues self resolved and we don't have any guarantee that it will not reoccur, nor that we can fix future non straightforward service problems.
We would expect citoid to provide:
- observability into latency of individual requests, either to the service itself or to the upstream services.
- access logs, showing the latency and status code of individual requests.