12:04 <Urbanecm> hi, seems all webserices are broken, throws 301 Moved Permanently. https://tools.wmflabs.org/urbanecmbot/, https://tools.wmflabs.org/swviewer/, https://tools.wmflabs.org/meta/, https://tools.wmflabs.org/statistics/, https://tools.wmflabs.org/wikinity/ and anything else i tried 12:04 <Urbanecm> is this cloud-wide issue? 12:05 <Urbanecm> seems restarting the tool doesn't help
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
Sorry!
This is what happened:
- I reviewed and merged this patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 related to ongoing work on T234617: Toolforge. introduce new domain toolforge.org and T247236: Toolforge: introduce a system to preserve old tools.wmflabs.org URLs
- With the merge, I started monitoring prometheus metrics in this grafana dashboard: https://grafana-labs.wikimedia.org/d/R7BPaEbWk/toolforge-ingress?orgId=1&refresh=1m&from=now-3h&to=now and https://grafana-labs.wikimedia.org/d/toolforge-kubernetes/toolforge-kubernetes?refresh=1m&orgId=1&from=now-3h&to=now and the proxy server logs for the usual activity and errors
- Bad timing, I had a meeting scheduled at 10:30 UTC, which I joined and stopped looking at the metrics (or the IRC channel)
- One hour later, after I stopped paying attention to the meeting, I discovered this report and other reports on IRC. The grafana dashboard was showing clear signs of the change not working as expected.
- Immediately reverted the patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/579250 and things were back to normal. Approx 1h of downtime :-(
This is the grafana dashboard:
So, this is my fault for being distracted by a meeting and not paying enough attention. Should not have attended the meeting or double checked my calendar before merging.
Comment Actions
OT: Tasks are merged to whichever contains more info, given this has the conclusion, I've changed the closure :-).
Comment Actions
@AlexisJazz: This is not a competition who reports something first, but it's about which task has more relevant data and/or progress when someone realizes that two tasks are about the same thing.
Comment Actions
The same thing will happen on Commons sometimes, I understand. It's just odd that aborrero responded here instead of on the first task, given that this one was already closed as duplicate and the contents up to that point were similar. So it feels like the report was "stolen" from me, though it was most probably unintentional.
Comment Actions
I clicked in the first phab link I saw on IRC. I dont plan to followup on this topic.