At around 2023-12-14T07:39:02, the Cassandra instance restbase2029-a was killed by the kernel (OOM). It was eventually restarted by Puppet and returned to service at approximately 2023-12-14T07:56:54.
This host was recently added as part of a ([[ https://phabricator.wikimedia.org/T352468 | currently on-going ]]) refresh; It has only been online a few days.
{P54439}
Something else of interest in the `dmesg` output are errors from the driver for the SAS controller. There are several instances of these spanning days in the full output (attached), but an example of these is below.
{P54440}
----
{F41600595}
{F41600596}
### Update:
This has continued to happen —a total of ~~seven~~ nine times so far. Thus far it has only happened to the recently added Dell r450s (see: T352468), ~~and it has only happened once per instance.~~ Thus far it has only happened to the nodes in rack (row) `b`.
| host | ooms | rack |
| ---- | ---- | ---- |
| restbase2028-a | ✔✔ | b |
| restbase2028-b | ✔ | b |
| restbase2028-c | ✔ | b |
| restbase2029-a | ✔✔ | b |
| restbase2029-b | ✔ | b |
| restbase2029-c | | b |
| restbase2030-a | ✔ | b |
| restbase2030-b | | b |
| restbase2030-c | ✔ | b |
| restbase2031-a | | c |
| restbase2031-b | | c |
| restbase2031-c | | c |
| restbase2032-a | | c |
| restbase2032-b | | c |
| restbase2032-c | | c |