Alert:
[09:47 UTC] <icinga-wm> PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1004 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=eqiad&var-cache_type=All&var-status_type=5
The top failing urls at the time: https://logstash.wikimedia.org/goto/58461347c38952237e54c310e42fa8d4
GET https://es.wikipedia.org/api/rest_v1/page/pdf/Mari_(diosa_vasca) 2,160 GET https://zh.wikipedia.org/api/rest_v1/page/pdf/瓜拉克萨巴 1,029 GET https://es.wikipedia.org/api/rest_v1/page/pdf/Marianne_Jean-Baptiste 1,013 GET https://ja.wikipedia.org/api/rest_v1/page/pdf/カウナス・モスク 1,011 GET https://ru.wikipedia.org/api/rest_v1/page/pdf/Ftp_(программа) 1,005 GET https://es.wikipedia.org/api/rest_v1/page/pdf/Francisco_Herboso_España 1,003 GET https://ja.wikipedia.org/api/rest_v1/page/pdf/山口俊一 830 GET https://ru.wikipedia.org/api/rest_v1/page/pdf/Лескен_(Северная_Осетия) 748 GET https://ar.wikipedia.org/api/rest_v1/page/pdf/بريلة 610 GET https://pl.wikipedia.org/w/api.php?ucuser=Paweł Ziemian BOT&maxlag=10&uclimit=1&format=json&action=query&rawcontinue=&list=usercontribs&ucprop=ids|title|timestamp|comment|flags
There was a mix of HTTP responses with 500 and 503 codes.