Change Details

Since approximately 3am UTC, the amount of 5XX responses as monitored on Wikimedia CDNs shows an increase in rate, from 1-2 per second to 20-30 per second: https://grafana.wikimedia.org/d/000000503/varnish-http-errors?orgId=1&viewPanel=7&from=1651705335095&to=1651748535095 (please ignore the spike from 5:40-5:55, that was an unrelated outage T307647) Looking at logstash logs, the source of the errors seem to be maps: (error rates without maps included) {F35104642} (only error rates of maps) Dashboards seem worrying and not normal regarding maps performance (increased load and IO): https://grafana.wikimedia.org/d/W67hDHEGz/maps-performances-jgiannelos?orgId=1&from=1651143918627&to=1651748718627 {F35104641}

Since approximately 3am UTC, the amount of 5XX responses as monitored on Wikimedia CDNs shows an increase in rate, from 1-2 per second to 20-30 per second: https://grafana.wikimedia.org/d/000000503/varnish-http-errors?orgId=1&viewPanel=7&from=1651705335095&to=1651748535095 (please ignore the spike from 5:40-5:55, that was an unrelated outage T307647) Looking at logstash logs, the source of the errors seem to be maps: (error rates without maps included) {F35104642} (only error rates of maps) {F35104641} Dashboards seem worrying and not normal regarding maps performance (increased load and IO): https://grafana.wikimedia.org/d/W67hDHEGz/maps-performances-jgiannelos?orgId=1&from=1651143918627&to=1651748718627

Since approximately 3am UTC, the amount of 5XX responses as monitored on Wikimedia CDNs shows an increase in rate, from 1-2 per second to 20-30 per second: https://grafana.wikimedia.org/d/000000503/varnish-http-errors?orgId=1&viewPanel=7&from=1651705335095&to=1651748535095 (please ignore the spike from 5:40-5:55, that was an unrelated outage T307647) Looking at logstash logs, the source of the errors seem to be maps: (error rates without maps included) {F35104642} (only error rates of maps) {F35104641} Dashboards seem worrying and not normal regarding maps performance (increased load and IO): https://grafana.wikimedia.org/d/W67hDHEGz/maps-performances-jgiannelos?orgId=1&from=1651143918627&to=1651748718627 {F35104641}