The late_command support for switching our NVME drives to 4K block size was only set for eqiad and esams, and thus the latest codfw caches didn't get it at install time. Will upload a patch to correct the regex for future reimages, but we also need to correct the existing nodes. Without this everything "works", but uses the default 512 byte format, which isn't as efficient/performant as native 4K (same as memory page size).
The conceptually-simplest fix is just to reimage them all, but it might be easier and faster to just take a downtime of ats-be on each of them and reformat the storage before restarting, using the same commands as the late_command stuff:
/usr/sbin/nvme format /dev/nvme0n1 -l 2 echo ';' | /usr/sbin/sfdisk /dev/nvme0n1