[ ] [[ https://wikitech.wikimedia.org/wiki/Incident_response/Lightweight_report | lightweight incident report ]] for the incident
[ ] open task to migrate all graphite uses to prometheus, and also to make sure the metrics as exported make sense (e.g. on the existing console, there's some notes about naming inconsistencies making things difficult to interpret)
[ ] open task to clean up some of the common logging spam, and to add more data to error logs
[ ] open a design task to fix the "backlog cascading failure" issue. Somehow, service-runner should know that a request is too old to be worth working on, and throw it away before doing actual work on it. (If we fix this, then adding capacity back to the service would have been sufficient to bring us out of an outage. It would also mean that, in the general case, bursts of extra traffic while we're near capacity are much less likely to get into an outage from which we can't recover.)