| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Duplicate | None | T420227 Project deployment-prep instance deployment-sessionstore06 is down | |||
| Resolved | • bd808 | T415021 Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 | |||
| Duplicate | None | T420287 Project deployment-prep instance deployment-sessionstore06 is down |
Leaving open but marking as stalled on T415021: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 in the hope that we get less Phab spam about the overloads making monitoring flap for this instance.
Boo. Leaving it open did not stop T420284: Project deployment-prep instance deployment-sessionstore06 is down from being filed.
My first guess would be that maybe that might be because phalerts seems like it might currently only look for tasks with the open status, so maybe setting this task to 'stalled' - while probably more semantically correct - defeated the purpose of having it open as a separate task at all (ie., preventing duplicate filings) :/
Obviously you know what might be best in this situation better than me, but I wonder if setting a silence for this alert might be a possible idea, given how much it's flapping? (h/t @taavi, who informed me the other day that this was a thing)