12:04 <Urbanecm> hi, seems all webserices are broken, throws 301 Moved Permanently. https://tools.wmflabs.org/urbanecmbot/, https://tools.wmflabs.org/swviewer/, https://tools.wmflabs.org/meta/, https://tools.wmflabs.org/statistics/, https://tools.wmflabs.org/wikinity/ and anything else i tried 12:04 <Urbanecm> is this cloud-wide issue? 12:05 <Urbanecm> seems restarting the tool doesn't help
This is what happened:
- I reviewed and merged this patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 related to ongoing work on T234617: Toolforge. introduce new domain toolforge.org and T247236: Toolforge: introduce a system to preserve old tools.wmflabs.org URLs
- With the merge, I started monitoring prometheus metrics in this grafana dashboard: https://grafana-labs.wikimedia.org/d/R7BPaEbWk/toolforge-ingress?orgId=1&refresh=1m&from=now-3h&to=now and https://grafana-labs.wikimedia.org/d/toolforge-kubernetes/toolforge-kubernetes?refresh=1m&orgId=1&from=now-3h&to=now and the proxy server logs for the usual activity and errors
- Bad timing, I had a meeting scheduled at 10:30 UTC, which I joined and stopped looking at the metrics (or the IRC channel)
- One hour later, after I stopped paying attention to the meeting, I discovered this report and other reports on IRC. The grafana dashboard was showing clear signs of the change not working as expected.
- Immediately reverted the patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/579250 and things were back to normal. Approx 1h of downtime :-(
This is the grafana dashboard:
So, this is my fault for being distracted by a meeting and not paying enough attention. Should not have attended the meeting or double checked my calendar before merging.
The same thing will happen on Commons sometimes, I understand. It's just odd that aborrero responded here instead of on the first task, given that this one was already closed as duplicate and the contents up to that point were similar. So it feels like the report was "stolen" from me, though it was most probably unintentional.