The Swift ms cluster is running Stretch and should be upgraded to Bullseye. Running a new kernel will be a great occasion to revising the mkfs.xfs options we're using now.
Most notably we're disabling finobt (i.e. the free inode btree) because of free space reporting problems in the past (T199198: Some swift filesystems reporting negative disk usage). While disabling finobt fixes the problem, I believe it does slow down allocations/deletions especially on old-age filesystems and in general given our pattern of creation/deletion of objects (i.e. thumbnails, the most frequent case)
6/8 Frontends done by 2022-04-27 (the last two are the swiftrepl nodes, which are blocking on T299125 )
All backends done by 2022-0615 except ones which are obsolete and due to be cycled out of the cluster.
Initial host status:
mvernon@cumin1001:~$ date; sudo cumin '(A:stretch or A:buster) and A:swift'; sudo cumin 'A:bullseye and A:swift' Mon 11 Apr 2022 01:23:49 PM UTC 93 hosts will be targeted: ms-be[2028-2069].codfw.wmnet,ms-be[1028-1033,1035-1071].eqiad.wmnet,ms-fe[2009-2012].codfw.wmnet,ms-fe[1009-1012].eqiad.wmnet DRY-RUN mode enabled, aborting No hosts found that matches the query
Occasionally-updated host status
Fri 17 Jun 2022 02:32:35 PM UTC 25 hosts will be targeted: ms-be[2028-2039].codfw.wmnet,ms-be[1028-1033,1035-1039].eqiad.wmnet,ms-fe2009.codfw.wmnet,ms-fe1009.eqiad.wmnet DRY-RUN mode enabled, aborting 68 hosts will be targeted: ms-be[2040-2069].codfw.wmnet,ms-be[1040-1071].eqiad.wmnet,ms-fe[2010-2012].codfw.wmnet,ms-fe[1010-1012].eqiad.wmnet DRY-RUN mode enabled, aborting