@Ladsgroup do you have an older container where you're confident the initial deletion went to completion that I might usefully graph?

Just to confirm that's a genuine object...

root@ms-fe1009:~# swift stat wikipedia-commons-local-thumb.3f 3/3f/Popular_Science_Monthly_Volume_2.djvu/page773-1024px-Popular_Science_Monthly_Volume_2.djvu.jpg
               Account: AUTH_mw
             Container: wikipedia-commons-local-thumb.3f
                Object: 3/3f/Popular_Science_Monthly_Volume_2.djvu/page773-1024px-Popular_Science_Monthly_Volume_2.djvu.jpg
          Content Type: image/jpeg
        Content Length: 515462
         Last Modified: Mon, 15 Feb 2021 17:24:20 GMT
                  ETag: 49081a23279990cd7c73d413e93ea1a3
   Content-Disposition: inline;filename*=UTF-8''Popular_Science_Monthly_Volume_2.djvu.jpg
           X-Timestamp: 1613409859.86478
         Accept-Ranges: bytes
            X-Trans-Id: txd347d37ff8fa4dd8ace27-006a760a0a
X-Openstack-Request-Id: txd347d37ff8fa4dd8ace27-006a760a0a

I tried looking at another container - wikipedia-commons-local-thumb.3f in codfw. After clearing out objects with silly names that I'd expect the cleanup to have missed:

"thumbor/3/3f/Macroplaza,_Edificio_Latino,_Tribunal_de_Justicia_de_Nuevo_León.jpg/800px-\nMacroplaza,_Edificio_Latino,_Tribunal_de_Justicia_de_Nuevo_León.jpg"
"thumbor/3/3f/Puppies_Fighting.jpg/200px-\r\nPuppies_Fighting.jpg"
"3/3f/Macroplaza,_Edificio_Latino,_Tribunal_de_Justicia_de_Nuevo_León.jpg/800px-\nMacroplaza,_Edificio_Latino,_Tribunal_de_Justicia_de_Nuevo_León.jpg"
"3/3f/Puppies_Fighting.jpg/200px-\r\nPuppies_Fighting.jpg"

I don't think 0f has been effectively cleared out in eqiad at least - the oldest objects are from 2013, so the age plot is much less useful...

Here's a little table of number of thumbs created in the container per day (once I'd dug out the old ones with '\n' and other such delights in the name):

mvernon@ms-be1072:~$ sudo sqlite3 --readonly /srv/swift-storage/accounts0/containers/56489/594/dca97818fea217d3227cd186d32df594/dca97818fea217d3227cd186d32df594.db "SELECT CAST(round(julianday('now')-julianday(created_at,'unixepoch')) AS INT) AS age, count(*) from object WHERE deleted=0 GROUP BY age ORDER BY age DESC;"
17|2152
16|46173
15|80742
14|94689
13|99122
12|88016
11|70032
10|53087
9|47634
8|51107
7|44935
6|42651
5|38112
4|35010
3|35796
2|31655
1|30687
0|14415

In T257093#12193179, @TheDJ wrote:

There is no rush, it’s just that we have 315GB of storage of which a significant part isn’t used anymore because we switched from png to SVG. And the whole Score stack never gets rid of anything that used to once be in wikitext, but is no longer in use. That allows unbounded growth, which generally isn't liked, so it’s something someone should have on the radar.

@RobH from my point of view, the JBOD disk-swap behaviour seems good, so if @elukey is happy that we're going to be able to reliably (re-)image hosts of this sort, then I'm OK with using them as a basis for swift/ceph Config-J nodes.

@HFan-WMF note that whatever ends up happening about that patch, it won't address the problem of storing 300G of useless PNG files, so those will need handling (I would guess using the script in the extension that deletes old images?).

My uninformed view is that "we want to keep them except when we upgrade lilypond and then want to regenerate everything" might inform a deletion-based approach...?

OK, right, yes, I think that is likely worth doing. Looking at the container now:

root@ms-fe2009:/home/mvernon# swift stat global-data-score-render
             Container: global-data-score-render
               Objects: 3.6M
                 Bytes: 315G

Which would be quite a lot of unnecessary objects to keep forever! I think Score is owned by Reader Experience, so I've tagged them here to take a look. I'm also inclined to agree that adding a TTL might be sensible (though swift TTL is a rather blunt instrument), but I'll let RE take a look.

@TheDJ sorry, what do you mean by "this", in this context, please? [I don't think there's an action for swift-admin here, but I wanted to make sure]

The object expirer does run in the ms swift cluster, so if you start setting the relevant expire header on objects, that should take effect. NB if you do this it'll do nothing about even-older objects.

[ aside: we've previously talked about using TTL-based-thumb expiry in the past and decided against implementing it, some discussion was in T211661 . Which is obviously not the same as you're proposing here, but seemed worth mentioning]

Thanks. That swap went OK - the two drives just re-appeared in each other's places and could be remounted.

Hi folks, thanks for coming up with some possible solutions :)

OK, I've had a poke at this host (which uses storcli rather than perccli, which is a bit confusing, but I guess expected?).

@elukey thanks for taking a look! I had a meander through https://wikitech.wikimedia.org/wiki/Management_Interfaces#Troubleshooting_Commands myself (other than where the Supermicro command is missing) and likewise a BMC reset (which I did from being logged in) didn't help. I can ssh to the BMC just fine, though.

@elukey is it possible T433635 is related?

[counting 2 extra codfw backends which are already trixie, after T424892]

By renumbering, I meant "will the IP address change?". So yes, you have answered my question, thank you :)

I've done this for thanos and ms-{fe,be} hosts. Can I check that this is just physical moving, no renumbering?

@Ladsgroup I think you were the D-P representative in earlier discussion about this - can you give me a short summary of what D-P has agreed to apropos this work, please?

There is now an object-expirer service running on the thanos-swift cluster.

So @Eevans has kindly agreed to manage this in my absence - apus-be2004 will need putting into maintenance mode before you start (so I suggesting pinging him when you're nearly ready to go), and then bringing out of it again once the work is done.

I was coming here to ask for something similar - even if it'll reply to cortobot: help in-channel, it should have an option to respond to help over privmsg if addressed thus.

I've just deleted and re-created my OoO starting 21 July. Thanks for looking into this :)

apus-be2004 is an apus backend, which will need some care (otherwise the cluster will notice the host as failed and start rebalancing all over the place); I would suggest using maintenance mode.

In general, you don't want the same version in two suites - while it might work, different e.g. C library versions, python version (and thus python path version), etc. will mean that often it doesn't work.

Anything useful from upstream?

Thanos-swift cluster looks good, thanks.

Likely so, yes. The issue (as we'll have to get to with T427949) is to avoid ending up storing the uncompressed versions forever in swift (in deleted items); I think the bot being used there is keeping track of object/revisions that need deleting-permanently, and we'll have to do those in due course. But this can just hang off that task for now.

If I'm reading the ticket correctly, this looks like a rename on swift that didn't get correctly recorded in the database? So tagging in DBA for them to have a look.

We only keep swift logs for a few days (because of the large volume of them), so we won't have any swift logs from 22 May.

I'm glad to hear it worked this time! I'll close this ticket, then. If it starts happening again, do feel free to open a fresh ticket (rough timestamp and the name of the failing object like you provided this time lets me at least look at the swift logs).

I've gone looking in swift logs for the first of these two objects, and I find these four hits, in time order:

Jun  5 00:53:39 ms-fe1017 proxy-server: 10.67.223.8 10.64.0.188 05/Jun/2026/00/53/39 PUT /v1/AUTH_mw/wikipedia-en-local-temp.b6/b/b6/1cp9qih7a3qg.wgwy8u.20444050.webm.24 HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tkd31429ce8... 19439616 - dde7c619af0ac235d88742fb6eb489a7 tx1f2b5b340ed845af96311-006a221e13-f683416a-f311-41db-8397-31ce7b65 - 0.2584 - - 1780620819.034368515 1780620819.292721748 0
Jun  5 01:06:22 ms-fe1020 proxy-server: 10.67.207.169 10.64.179.7 05/Jun/2026/01/06/22 GET /v1/AUTH_mw/wikipedia-en-local-temp.b6/b/b6/1cp9qih7a3qg.wgwy8u.20444050.webm.24 HTTP/1.0 499 - wikimedia/multi-http-client%20v1.1 AUTH_tkd31429ce8... - 65536 - tx1caf58a560354aa499aa6-006a222108-44dccfa6-ba05-4e38-a9c6-c187f1a7 - 5.7017 - - 1780621576.440272093 1780621582.141992092 0
Jun  5 01:07:19 ms-fe1013 proxy-server: 10.67.220.5 10.64.179.6 05/Jun/2026/01/07/19 GET /v1/AUTH_mw/wikipedia-en-local-temp.b6/b/b6/1cp9qih7a3qg.wgwy8u.20444050.webm.24 HTTP/1.0 200 - wikimedia/multi-http-client%20v1.1 AUTH_tkd31429ce8... - 19439616 - txfc6c57deedc84f57af311-006a222146-44dccfa6-ba05-4e38-a9c6-c187f1a7 - 1.6357 - - 1780621638.248369932 1780621639.884046316 0
Jun  5 01:12:27 ms-fe1015 proxy-server: 10.67.220.5 10.64.158.3 05/Jun/2026/01/12/27 DELETE /v1/AUTH_mw/wikipedia-en-local-temp.b6/b/b6/1cp9qih7a3qg.wgwy8u.20444050.webm.24 HTTP/1.0 204 - wikimedia/multi-http-client%20v1.1 AUTH_tkd31429ce8... - - - tx8de76737374747b583f0a-006a22227b-44dccfa6-ba05-4e38-a9c6-c187f1a7 - 0.0813 - - 1780621947.821367741 1780621947.902695179 0

ms swift in codfw looks OK after this work, thanks.

All done! And all codfw backends moved to new-style storage.

Thanks @Pppery .

[subscribing because T428063 came to my swift pile, and I think this is the likely underlying issue]

Wearing my WMF staff hat, I'd like to note that "we should not store uncompressed TIFFs in commons" is definitely our call to make, and "upload tooling should not upload uncompressed TIFFs to commons" likely also.

TIFF compression can be done losslessly, so I see no reason to accept uncompressed TIFFs.

TIFF compression is fairly easy via tiffcp (I'm not a compression specialist, both -c lzma:p9 and -c zip:p12 were pretty good, the former slower but better). The difficulty is that we can't just change objects in swift "under the feet of" Mediawiki.

Cool, I've blocked that out in my calendar :)

Hi @VRiley-WMF I have oddly-full afternoons on other days at the moment; I could do 14:30-16:30 UTC on Wednesday 3rd if that works for you? And then afternoons of Tue 16th (until about 17:30 UTC) or Thu 18th (until about 16:30 UTC).

@VRiley-WMF sure; backend's can't be meaningfully depooled, so it'd be a case of "do one, check everything has recovered OK, move on to the next". [not sure if it's easier to track them here or in T401966.

A couple of thoughts. I am broadly in favour of handling thumbs and originals separately :)

You've captured the key bits from our conversation earlier, thanks. There is ongoing maintenance on both clusters at the moment (T421719, T354872), so it's plausible that there will be <100% dispersion and/or lengthy oldest completion responses; but all those commands should return 0 and should not log anything about connection refused/etc. If in any doubt, feel free to ping me.

@Jhancock.wm new-new drive looks good now, thanks :)

Oh, actually, it's showing up as connected by SAS not SATA, too:

@Jhancock.wm this is an odd one, but - can you pull the drive and check it doesn't have writes disabled somehow, please? I've tried rebooting, but still:

[   12.532808] sd 0:2:11:0: [sdl] Write Protect is on

And if I try and update the partition table or anything I get the read-only filesystem error. AFAICT there's nothing in software to otherwise suggest it's unhappy.