[WRN] DB_DEVICE_STALLED_READ_ALERT: 1 OSD(s) experiencing stalled read in db device of BlueFS
     osd.168 observed stalled read indications in DB device


root@cloudcephosd1043:~# ceph osd find osd.168
{
    "osd": 168,
    "addrs": {
        "addrvec": [
            {
                "type": "v2",
                "addr": "10.64.151.3:6807",
                "nonce": 236600425
            },
            {
                "type": "v1",
                "addr": "10.64.151.3:6811",
                "nonce": 236600425
            }
        ]
    },
    "osd_fsid": "22274bab-4500-47b0-b490-b495a7ca8b9a",
    "host": "cloudcephosd1042",
    "crush_location": {
        "host": "cloudcephosd1042",
        "rack": "C8",
        "root": "default"
    }
}

"ceph osd out 168"

hashar mentioned this in T434334: cloudcephosd1043 drives are very very busy.Aug 12 2026, 7:33 AM

Today, osd.353 observed stalled read indications in DB device, around 04:00 UTC 2026-08-17

There are some decent docs about this alert here:

https://oneuptime.com/blog/post/2026-03-31-rook-fix-db-device-stalled-read-alert-ceph/view

There's a reassuring screed about other admins getting tangled up by these new alerts, here:

https://www.spinics.net/lists/ceph-users/msg86138.html

Someone else on that thread recommends looking for kernel issues but of course the kernel logs on cloudcephosd1045 (which hosts osd.353) are completely fine.

RAM usage on OSDs is running around 50% for older servers and much, much less than that for newer servers. It would be reasonable to increase bluestore_cache_size fleet-wide.

root@cloudcephosd1045:/var/log# free -h
               total        used        free      shared  buff/cache   available
Mem:            62Gi        32Gi       400Mi       5.2Mi        29Gi        29Gi
Swap:          4.7Gi          0B       4.7Gi

root@cloudcephmon1005:~# ceph config get osd.353 bluestore_cache_size_ssd
3221225472

There are 8 OSDs on that host. So if we increased the cache size from 3G to 5G I'd expect usage to go from 32 to (32-(8x3)+(8*5)) = 48. If doubled to 6G we'd get a total usage of 56G.

56 feels like pressing our luck but 48 does not. These usage numbers are similar on all the hosts, so I think we should try this cache change everywhere.

I think cache resizing is a dead end. We already try to use autosizing with a cap of 6G per osd.

224 and 168 are back in; I am working on T429387 which I suspect is the same issue.

osd.54 observed stalled read indications in DB device
osd.334 observed stalled read indications in DB device
osd.335 observed stalled read indications in DB device

"osd.224 observed stalled read indications in DB device"Open, Needs TriagePublicActions

Description

Related Objects

Event Timeline

"osd.224 observed stalled read indications in DB device"
Open, Needs TriagePublic
Actions