Change #1132775 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/alerts@master] sessionstore-resources: add SessionStoreDiskSpaceUtilizationTooHigh

https://gerrit.wikimedia.org/r/1132775

Change #1132775 merged by jenkins-bot:

[operations/alerts@master] sessionstore-resources: add SessionStoreDiskSpaceUtilizationTooHigh

https://gerrit.wikimedia.org/r/1132775

Although there are various ways we could make an alert on disk space fairly sophisticated (e.g., extrapolate runway)...

No hurry on this part (the important bit is done), but I would be definitely be interested in hearing more about how this would look (or even a reference to examples, if we have them).

In T390630#10727602, @Eevans wrote:

No hurry on this part (the important bit is done), but I would be definitely be interested in hearing more about how this would look (or even a reference to examples, if we have them).

So, when I wrote this, the primary option I had in mind was to extrapolate runway "burn" rate - e.g., if the inferred growth rate over some window is high enough that we would run out of disk within some deadline (small number of hours) then fire. I've seen this pattern work well before in a different context, but I'm not currently aware of prior art here (I'll look around).

Where this would of course get tricky is setting the various knobs (extrapolation window, deadline, etc.) in a way that properly handles temporary excursions due to large compactions, but still gives us confidence the alert will fire when it should. Predicated upon that, whether this is a "win" vs. a basic utilization alert like we have right now is unclear.

Following up on how this alerting might evolve, there was some discussion in T392989 about how to make the alert insensitive to transient excursions during large compactions.

In T392989#10786849, @Scott_French wrote:

Possible alternative: Rather than using an aggregation over hosts, we could instead use the minimum utilization over a rolling window, calibrated the typical duration of a large compaction. IMO, that's an easier-to-reason-about mechanism for "ignoring" the compactions vs. aggregating over hosts (e.g., we frequently see correlated behavior across hosts). As @Eevans notes, we would likely want to combine that with dropping the alert threshold.

I think this might prove useful for what we're trying to do here. For example, let's say we want to use a 1h rolling window for to smooth out compactions, and use the last 6h of smoothed usage to predict that 24h in the future, alerting when we run out of runway within 24h.

In "pseudo-PromQL" that would look like: predict_linear(max_over_time(node_filesystem_avail_bytes{cluster="sessionstore", mountpoint="/srv"}[1h])[6h], 86400) < 0

To actually nest the queries this way, you'll need a subquery on the result of the max_over_time. For a 1w window covering the original incident, you can see a comparison showing the smoothed (left) and unsmoothed (right) extrapolations in https://grafana.wikimedia.org/goto/tWNAVcxHR?orgId=1. This correctly surfaces, e.g., when sessionstore2005 first passes the "24h left" mark early on the 29th, without the background false positives due to compactions.

In practice, if we were to use this in a real alert, we would probably want to introduce a recording rule for the smoothed node_filesystem_avail_bytes to avoid the use of subqueries in the alert query expression (e.g., introducing instance:node_filesystem_avail_bytes:max_over_time1h).

So, if we were to have both of:

1. an "instantaneous utilization" alert, just like we have today, to act as a backstop (possibly with a different trigger threshold, predicated in our confidence in the runway alert) and
2. an "extrapolated runway" alert, like that described above

... I think that would put us in a pretty solid poisition.

Change #1141959 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/alerts@master] sessionstore-resources: add SessionStoreDiskSpaceRunwayTooLow

https://gerrit.wikimedia.org/r/1141959

After a bit of thought and some back-testing over the last 2 months of data, https://gerrit.wikimedia.org/r/1141959 sketches out what T390630#10787491 could look like.

The two key changes are (1) using 3 day horizon for runway rather than 1 day (i.e., the idea being that this is a non-paging alert intended for business hours response, ideally routed to a phabricator task) and (2) using a 90m rolling window for the "smoothed" available-bytes recording rule (a bit more effective than 1h).

In T390630#10793749, @Scott_French wrote:

After a bit of thought and some back-testing over the last 2 months of data, https://gerrit.wikimedia.org/r/1141959 sketches out what T390630#10787491 could look like.

The two key changes are (1) using 3 day horizon for runway rather than 1 day (i.e., the idea being that this is a non-paging alert intended for business hours response, ideally routed to a phabricator task) and (2) using a 90m rolling window for the "smoothed" available-bytes recording rule (a bit more effective than 1h).

Sorry I hadn't replied to this sooner, it's been a busy week. :)

My Prometheus-fu is weak, so I probably need some time to get my Google on before I could usefully contribute a review for the Gerrit, but this fantastic! (...and I thought I should say so before letting more time pass 😀)

T394955

restbase2003 is soon running out and is alerting: T395845 if you could take a look at that one please.

imho the missing piece is just the notification method (T394955).

In T390630#10877285, @Dzahn wrote:

T394955: when servers are about to run out of disk, monitoring should notify the owners

This one is a bit different to the typical disk space notification, I think. Sessionstore utilization is expected to be stable over time, but (as we've seen) that could change in the face of an aberrant workload. If it does, the runway to disaster could be days, weeks, hours. So the first notification added as part of this ticket triggers at 70% and it pages (Data Persistence). The other (as-yet-merged) one forecasts, and it alerts (though a ticket would be grand).

In T390630#10877303, @Dzahn wrote:

restbase2003 is soon running out and is alerting: T395845 if you could take a look at that one please.

imho the missing piece is just the notification method (T394955).

I'm already looking into this one (as well as a few others). I thought that I had silenced them all earlier this morning. BTW, I got an email notification for these (from Nagios) —but yes— I would much prefer a ticket.

Thanks Eric and Daniel. +1 to Eric's articulation of how monitoring sessionstore disk utilization is a bit different from what we would want to do in the general "horizontal" (across all hosts) case, which in turn motivated https://gerrit.wikimedia.org/r/1141959.

FWIW, if we do go the route of adding a runway alert, I was planning to add an alertmanager receiver / routing rule to turn "task" severity notifications into phabricator tasks under Cassandra.

I'm hoping to close that out this week, pending discussion with Observability about the somewhat unusual idiom (i.e., using a recording rule to avoid a costly subquery in the alert expression).

Edit: After chatting with Eric, it sounds like Data-Persistence is the preferred first stop for the disk-runway ticket to ensure it's triaged promptly.

Change #1141959 merged by jenkins-bot:

[operations/alerts@master] sessionstore-resources: add SessionStoreDiskSpaceRunwayTooLow

https://gerrit.wikimedia.org/r/1141959

The SessionStoreDiskSpaceRunwayTooLow alert is now live, although in warning severity, to limit the scope of potential noise if something odd happens early on with the maths (e.g., while the history of the recording rule builds up).

In any case, letting this soak for 24-48h should (IMO) be plenty to assess whether this behaves as we expect from first principles and back-testing in terms of false positive rate, at which point we can promote this to task severity.

Change #1156837 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] alertmanager: update data-persistence-task phid

https://gerrit.wikimedia.org/r/1156837

Change #1156838 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/alerts@master] sessionstore-resources: move SessionStoreDiskSpaceRunwayTooLow to task

https://gerrit.wikimedia.org/r/1156838

Now that the new alert has been live for a couple of days without issuing false positives, I believe it makes sense to switch to task-severity.

Per discussion with @Marostegui, it should be fine from a process perspective to change the project phid associated with the data-persistence-task alertmanager receiver from DBA to the more general Data-Persistence - i.e., tasks can be triaged to more-specific sub-projects as needed, so long as the Data-Persistence project is retained (so that phalerts does not "lose" the task and create a duplicate one).

The two patches I've posted do this. At which point I think we can close this out.

Change #1156837 merged by Scott French:

[operations/puppet@production] alertmanager: update data-persistence-task phid

https://gerrit.wikimedia.org/r/1156837

Change #1156838 merged by jenkins-bot:

[operations/alerts@master] sessionstore-resources: move SessionStoreDiskSpaceRunwayTooLow to task

https://gerrit.wikimedia.org/r/1156838