Change #1329286 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] backy2: reduce ceph bandwidth throttle by 50%

https://gerrit.wikimedia.org/r/1329286

Change #1329286 merged by Andrew Bogott:

[operations/puppet@production] backy2: reduce ceph bandwidth throttle by 50%

https://gerrit.wikimedia.org/r/1329286

I'm going to see if I can do controller firmware updates on 1045-1047

current version 1+dz02 is ahead of target version 1+dl7c, use force to downgrade

so... I'm going to downgrade.

I didn't downgrade the ssd firmware but we did upgrade the raid firmware. No change in slow ops behavior.

Following this guide here:

https://oneuptime.com/blog/post/2026-03-31-rook-fix-bluestore-slow-op-alert-health-check-in-ceph/view

I'm following step 4:

"Ensure the I/O scheduler is set appropriately for your disk type."

root@cloudcephosd1045:/var/log/ceph# cat /sys/block/sdf/queue/scheduler
[none] mq-deadline 
root@cloudcephosd1045:/var/log/ceph# echo mq-deadline > /sys/block/sda/queue/scheduler
root@cloudcephosd1045:/var/log/ceph# cat /sys/block/sda/queue/scheduler
[mq-deadline] none
root@cloudcephosd1045:/var/log/ceph# echo mq-deadline > /sys/block/sdd/queue/scheduler
root@cloudcephosd1045:/var/log/ceph# echo mq-deadline > /sys/block/sde/queue/scheduler
root@cloudcephosd1045:/var/log/ceph# echo mq-deadline > /sys/block/sdf/queue/scheduler
root@cloudcephosd1045:/var/log/ceph# echo mq-deadline > /sys/block/sdg/queue/scheduler
root@cloudcephosd1045:/var/log/ceph# echo mq-deadline > /sys/block/sdh/queue/scheduler
root@cloudcephosd1045:/var/log/ceph# echo mq-deadline > /sys/block/sdi/queue/scheduler
root@cloudcephosd1045:/var/log/ceph# echo mq-deadline > /sys/block/sdj/queue/scheduler
root@cloudcephosd1045:/var/log/ceph# cat /sys/block/sdj/queue/scheduler
[mq-deadline] none

Then I rebooted cloudcephosd1045 to make sure that the change persisted.

root@cloudcephosd1045:~# cat /sys/block/sdj/queue/scheduler
[none] mq-deadline

It did not persist! I'm going to do the runtime setting for now, and if it helps we'll need a udev rule or something.

I've applied the same transient setting on 1046 and 1047.

Just got a stalled read on 1046. So the scheduler change did not resolve this.

Change #1338257 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudceph: enable osd_memory_target_autotune

https://gerrit.wikimedia.org/r/1338257

Another attempt: I added 2GB more ram to each of the OSDs on cloudvirt1045. Does not seem to make a difference; still getting slow ops.

I've had a look at this (with LLM assistance) and I think that I found something useful.
The 12 hosts that trip this warning are the 12 hosts fitted with Micron 5xxx PRO SATA SSDs.

Peak per-OSD commit latency over 7 days, grouped by host:

So I don't think this is the new Reef warning being noisy. I think it found something
that was always there and that we had no way to see before. @fgiunchedi's concern in
T429387#12040206 about papering over a real problem looks well founded.

I've got some queries to share about how I found this, but here is one to start with:
https://grafana-rw.wikimedia.org/goto/sclk9t?orgId=default

sort_desc(max by (model) (
  max_over_time(
    (rate(node_disk_write_time_seconds_total{instance=~"cloudcephosd1.*"}[2m])
     / clamp_min(rate(node_disk_writes_completed_total{instance=~"cloudcephosd1.*"}[2m]), 0.001))[2d:2m]
  )
  * on(instance,device) group_left(model) device_smart_info
))

In T429387#12329114, @BTullis wrote:

I've had a look at this (with LLM assistance) and I think that I found something useful.
The 12 hosts that trip this warning are the 12 hosts fitted with Micron 5xxx PRO SATA SSDs.

ok, that fits with the drive models that I flagged on this sheet (which apparently I forgot to link on this task before) https://docs.google.com/spreadsheets/d/1YdP6qkH2shhdDTJ96kWU7VbSZ197likPvZtRsEf57jk

But IMO that line of drive should still work for ceph! They're optimized for reading but that doesn't mean it should take multiple seconds for a write...

This is worth a read, I haven't tried all the suggestions (and certainly haven't tried all of them together):

https://www.reddit.com/r/ceph/comments/1sfnqup/improving_a_cluster_of_crappy_ssds/

These cumin queries are useful, too.

This query breaks down the hosts, by which SSDs they have.

sudo cumin 'cloudcephosd1*' 'lsblk -dno MODEL,SIZE | sort | uniq -c'

This query shows which hosts have had these slow operations:

sudo cumin 'cloudcephosd1*' "grep -ho 'slow operation observed' /var/log/ceph/ceph-osd.*.log | wc -l"

This command, which can be run on the cloudcephosd hosts is also useful.
It looks at the amount of data written, against the total power on hours, to calculate a value for the DWPD values.

btullis@cloudcephosd1042:~$ for d in $(lsblk -dno NAME,MODEL | awk '/MTFDDAK3T8TGA/ {print $1}'); do sudo smartctl -A "/dev/$d" |     awk -v d="$d" '/Power_On_Hours/{h=$10} /Total_LBAs_Written/{w=$10}
      END {printf "%s  %5.0f days  %4.0f TB written  %5.0f drive-writes  %.2f DWPD\n",
                  d, h/24, w*512/1e12, w*512/1e12/3.84, (w*512/1e12/3.84)/(h/24)}'; done
sda    440 days   676 TB written    176 drive-writes  0.40 DWPD
sdd    440 days   717 TB written    187 drive-writes  0.42 DWPD
sde    440 days   697 TB written    181 drive-writes  0.41 DWPD
sdf    440 days   693 TB written    181 drive-writes  0.41 DWPD
sdg    440 days   624 TB written    163 drive-writes  0.37 DWPD
sdh    440 days   712 TB written    185 drive-writes  0.42 DWPD
sdi    440 days   613 TB written    160 drive-writes  0.36 DWPD
sdj    440 days   716 TB written    187 drive-writes  0.42 DWPD

We can see that the drives are all well within their stated 1 DWPD value, so it doesn't seem to be an issue around the endurance, per-se.
But it's definitely these Micron drives that are affected.

• Peak write await over 2 days, all 17 drive models present in the fleet:

MTFDDAK1T9TDT              339.5 ms     <-- Micron 5300 PRO 1.92T
MTFDDAK3T8TGA-1BC1ZABDA    233.7 ms     <-- Micron 5400 PRO 3.84T
MTFDDAK1T9TDN               54.3 ms     <-- Micron 5200 1.92T
HFS1T9G32FEH-BA10A           5.1 ms
HFS480G3H2X069N              3.5 ms
SSDSC2KG019T8R               2.4 ms
SSDSC2KG240G8R               1.9 ms
MZ7KM1T9HMJP0D3              1.8 ms
MTFDDAK480TDS                1.6 ms
SSDSC2KB480GZR               1.4 ms
MTFDDAK480TGA-1BC1ZABDA      1.3 ms
MZ7LH480HBHQ0D3              1.3 ms
SSDSC2KG019TZR               1.3 ms
SSDSC2KG038TZR               1.1 ms
SSDSC2KB019T7R               0.9 ms
MZ7KH1T9HAJR0D3              0.8 ms
SSDSC2KB019T8R               0.6 ms

Note that the three Micron models at the top are the large data drives. The ones further down are some O/S drives, so not relevant.

We know that BlueStore has not been issuing TRIM commands (bdev_enable_discard is false by default), so my theory is that after 182 cumulative overwrites of the whole device, without TRIM being enabled, the drive's FTL believes essentially every LBA (logical block address) is live. Then it would only have its factory over-provisioning from which to do garbage-collection. This might also explain why enabling bdev_enable_discard in T434334 made things much worse rather than better. If the controller was suddenly handed a flood of TRIM that it couldn't cope with.

I have some suggestions for how we might test this...

In T429387#12329592, @Andrew wrote:

But IMO that line of drive should still work for ceph! They're optimized for reading but that doesn't mean it should take multiple seconds for a write...

Agreed, but it turns out Dell more or less accepts this point. I went looking for firmware fixes and found this instead:

KB000203252, "PowerFlex: Multiple SDS Decouple - Micron SSD (5200 and 5300 Series) Slow Device"
https://www.dell.com/support/kbdoc/en-us/000203252/multiple-sds-decouple-micron-ssd-5200-5300-series-slow-device

PowerFlex is Dell's own scale-out software defined storage, so the workload is similar to ours. The article describes Micron MTFDDAK drives producing "long IO", the storage daemon decoupling, and the system dropping into a DEGRADED state.
Dell is explicit that this is not a PowerFlex bug, that all PowerFlex versions are affected, and that the 5200 MTFDDAK parts were removed from the PowerFlex support matrix.

So the behaviour is known to the vendor, on a comparable product, and their resolution there was to stop supporting the drives rather than to fix them.

Firmware: there seem to be no fixes pending for the drives that matter

Dell's firmware labels map onto what the drives report, so D4DK403 is Dell's "K403" and D3DJ004 is "J004":

drive                          hosts        running   latest Dell   status
MTFDDAK3T8TGA 5400 PRO 3.84T   1042-1047    K403      K403          already current
MTFDDAK1T9TDT 5300 PRO 1.92T   1016-1020    J004      J008          4 revisions behind
MTFDDAK1T9TDN 5200 1.92T       1022         F003      F005          2 revisions behind

The 48 drives doing most of the damage are already on the newest firmware Dell ships. Only two versions exist for that part, K402 and K403, and K403 dates from December 2023.
So firmware is not a pending fix for 1042-1047, sadly, and Dell has not touched that part in nearly three years.

Correction: the July discard experiment did not make latency worse

I need to correct what I said in my previous comment. I pulled the metrics from the window when bdev_enable_discard was enabled on cloudcephosd1045 and I had this backwards. It did not make things worse. It fixed the latency.

period                      1045 peak commit   1045 mean util   1047 (control)
20-24 Jul, discard off      715-6088 ms        4.8-7.2%         1446-5552 ms
26 Jul-7 Aug, discard ON    165-390 ms         60-77%           1289-5763 ms
8-12 Aug, reverted          1852-3479 ms       5.5-6.3%         501-3097 ms

That is a 10x to 20x improvement in peak commit latency, comfortably under the 5 second warning threshold, with the control host flat throughout.
It agrees with what @Andrew saw at the time in T429387#12178808, that 15 of the 16 treated OSDs stopped displaying that warning.

What actually went wrong was the utilisation went through the roof, which is what T434334 caught. Per drive during that window:

discard rate          111 /s
discard volume       6.05 MB/s     (writes were 5.97 MB/s, so roughly 1:1, as expected)
mean TRIM service    4.76 ms       <-- should be microseconds
mean write service   3.94 ms       (normally 0.15-0.3 ms)

TRIM on these drives costs about 4.76 ms per command. So TRIM alone consumed roughly 53% of each drive's service time, and writes queued behind it. That is the "drives are very very busy" graph.

One detail that matters for what to do next: the discard rate and the utilisation were flat across the whole two weeks, 1126/s on the first day and 1175/s on the last.
So that was steady state allocation churn rather than a backlog being worked off, and letting it run longer would not have helped.

Proposed test

We now know that discard fixes the latency, and we know exactly why it was too expensive. The cost is per TRIM command rather than per byte, so the thing to try is getting BlueStore to issue far fewer and larger discards. bluestore_min_alloc_size_ssd is 4096 by default and is fixed when the OSD is created, so changing it means recreating the OSD. That is also the only way to get a full blkdiscard in, so the two go together naturally.

I suggest that we do the following on cloudcephosd1046, our worst host.

Take 4 of its 8 OSDs (osd.330-337) and leave the other 4 alone as an in-host control.

• drain and destroy the 4 OSDs
• blkdiscard the whole device on each of those 4 drives. This is the step that actually resets the FTL. ceph-volume lvm zap --destroy does not do it, which is probably why reimaging 1042 in August changed nothing
• recreate them at full size with bluestore_min_alloc_size_ssd = 16384
• ceph config set osd.N bdev_enable_discard true and bdev_async_discard_threads 1, set against the OSD id rather than by device class mask. There is a known Reef bug where a mask that cannot be evaluated at OSD boot silently leaves async discard switched off
• leave it 72 hours

If the utilisation is still unacceptable afterwards then we can come back and look at under-provisioning the drives instead, which buys the FTL headroom structurally with no TRIM traffic at all, at the cost of capacity.
What we should see if this works:

• peak commit latency on the 4 treated OSDs down to a few hundred ms, matching what July gave us
• discard rate around a quarter of July's 111/s per drive, so roughly 28/s, with TRIM dropping from about 53% of the drive's service time into the low teens
• mean utilisation well below the 60-77% that got this reverted last time

The main risk runs the other way. A 16K allocation unit means small overwrites can become read modify write, so write amplification may go up.
That is measurable, so we watch written bytes per drive alongside everything else and abandon if it climbs.

Rollback is quick and already proven: ceph config set osd.N bdev_enable_discard false took effect immediately in August.
Reverting the alloc size would need another recreate, but the 4 control OSDs stay untouched throughout either way.

Finally, worth confirming that discard is genuinely live rather than assuming it. This currently reads 0 across 1042-1047:

sum by (instance) (rate(node_disk_discards_completed_total{instance=~"cloudcephosd104[2-7]:9100",device=~"sd[a-j]"}[10m]))

The drives in our test cluster seem to be of the same cursed model:

MTFDDAK1T9TGB-1BC1ZABDA in cloudcephosd200[5-7].codfw
MTFDDAK3T8TGA-1BC1ZABDA in cloudcephosd2004.codfw

Micron 5400s, all. So that's a safer place to test, but it also doesn't get the traffic that eqiad1 gets so we might not get useful data there.

I'm going to try Ben's test, rebuilding osd.334, osd.335, osd.336 and osd.337. For starters I need to drain and destroy them and let the cluster rebalance, which will take ages.

Mentioned in SAL (#wikimedia-cloud) [2026-09-22T14:03:41Z] <andrewbogott> destroying ceph osd.334, osd.335, osd.336 and osd.337 for testing described on T429387

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-22T14:04:37Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-22T15:17:11Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-22T15:20:54Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-22T16:41:31Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-22T17:51:28Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-22T17:51:37Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-22T17:53:04Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-22T19:19:24Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-22T19:19:53Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-22T20:49:35Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

# ceph device ls | grep cloudcephosd1046
MTFDDAK3T8TGA-1BC1ZABDA_242349549459                   cloudcephosd1046:sdh  osd.332    1%                 
MTFDDAK3T8TGA-1BC1ZABDA_24234954957E                   cloudcephosd1046:sdg  osd.330    1%                 
MTFDDAK3T8TGA-1BC1ZABDA_24234954B728                   cloudcephosd1046:sdi  osd.333    1%                 
MTFDDAK3T8TGA-1BC1ZABDA_24234954B885                   cloudcephosd1046:sde  osd.331    1%

sdc and sdf are the OS drives. So, our test subjects (formerly known as osd334-337) are sda, sdb, sdd, sdj

As you can see in the messy logs above, the destroy calls failed so I'm going to start by zapping and destroying all four. That cleans up lvm &c. Now...

root@cloudcephosd1046:~# blkdiscard /dev/sda
<wait forever>
root@cloudcephosd1046:~# blkdiscard /dev/sdb
<etc>

And

$ ceph config set global bluestore_min_alloc_size_ssd 16384
$ ceph-volume lvm create --bluestore  --data /dev/sda
$ # etc
$ ceph config set global bluestore_min_alloc_size_ssd 4096

I can see that it took effect:

root@cloudcephosd1046:~# ceph config show osd.332 bluestore_min_alloc_size_ssd
4096
root@cloudcephosd1046:~# ceph config show osd.76 bluestore_min_alloc_size_ssd
16384
root@cloudcephosd1046:~# ceph config show osd.75 bluestore_min_alloc_size_ssd
16384
root@cloudcephosd1046:~# ceph config show osd.74 bluestore_min_alloc_size_ssd
16384
root@cloudcephosd1046:~# ceph config show osd.73 bluestore_min_alloc_size_ssd
16384
root@cloudcephosd1046:~# ceph config show osd.72 bluestore_min_alloc_size_ssd
4096

So now...

root@cloudcephosd1046:~# ceph config set osd.73 bdev_enable_discard true
root@cloudcephosd1046:~# ceph config set osd.74 bdev_enable_discard true
root@cloudcephosd1046:~# ceph config set osd.75 bdev_enable_discard true
root@cloudcephosd1046:~# ceph config set osd.76 bdev_enable_discard true
root@cloudcephosd1046:~# ceph config set osd.73 bdev_async_discard_threads 1
root@cloudcephosd1046:~# ceph config set osd.74 bdev_async_discard_threads 1
root@cloudcephosd1046:~# ceph config set osd.75 bdev_async_discard_threads 1
root@cloudcephosd1046:~# ceph config set osd.76 bdev_async_discard_threads 1

And now we wait!

Overnight things seem promising... the drives on 1046 are quite busy but not constantly maxing out like they were for the last trim test.

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-28T14:45:57Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-29T00:48:18Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-29T12:42:15Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

I'm making those changes (enable TRIM, 16k min alloc) on all the micron servers. Gradually...

• cloudcephosd1016
• cloudcephosd1017
• cloudcephosd1018
• cloudcephosd1019
• cloudcephosd1020
• cloudcephosd1022
• cloudcephosd1042
• cloudcephosd1043 (with min alloc 32768, for disk activity comparison)
• cloudcephosd1044
• cloudcephosd1045
• cloudcephosd1046
• cloudcephosd1047

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-29T22:04:40Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T01:28:27Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T10:59:48Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T12:42:58Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T12:43:04Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T12:45:31Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T12:45:35Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T12:46:14Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T12:46:18Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T12:48:21Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T12:48:27Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T18:06:49Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T18:06:56Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T18:07:05Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T18:07:10Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T18:21:13Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T18:21:19Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T18:21:31Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T18:21:35Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-09-30T18:47:02Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-01T04:25:12Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-01T05:30:59Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-01T13:38:48Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-01T16:15:26Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-01T18:17:08Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-01T18:22:40Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-02T03:38:39Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-02T03:39:59Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-02T04:10:06Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-02T08:13:43Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-02T12:37:28Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-02T17:18:04Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-02T17:21:02Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-02T23:45:15Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T00:27:36Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T00:27:47Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T00:28:05Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T00:28:13Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T05:06:03Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T13:47:10Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T18:39:04Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T19:19:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T19:19:53Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T19:20:18Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-03T21:09:56Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-05T03:46:08Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Moving 1043 to from 32k to 16k allocation since it doesn't seem much happier than the others.

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-05T07:20:34Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-05T13:56:26Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-05T13:56:35Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-05T13:56:58Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-05T14:27:25Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-05T15:11:58Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-10-05T15:12:04Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387)

In T429387#12394141, @Andrew wrote:

Moving 1043 to from 32k to 16k allocation since it doesn't seem much happier than the others.

• done

That's everything, I think! I've unmuted the "BLUESTORE_SLOW_OP_ALERT" alert so now we'll see if that still fires off and on.