+1 for adding the CACHE_VERSION to the file path. That way, after parser cache rollsover during 30 days, all previous files can be wiped, knowing that any page view for one would be a miss and regenerate it as needed.

Score currently has its own custom way of directory sharding (using 1+1 character subdirs instead of 1+2 chars). I don't know if this still matters or not. My limited understanding of Swift suggests that maybe it doesn't matter and that any iteration over all files from a given wiki or extension would go by overall "score/" prefix and thus perhaps not matter whether the name contains slashes. The same seems true for other large wiki farms as well and other object stores. If NFS is still considered a reasonable middleground for medium-size wikis outside WMF we could keep it I guess. But maybe it's not needed?

@fgiunchedi Given an upcoming change to Score's files stored in Swift, do you have a preference for whether and how much subdirectory sharding is applied to its files? E.g. would score/cache23/abcdef0123456789/abcdef.png be fine, or would you prefer something more like score/cache23/a/ab/abcdef0123456789/abcdef.png or score/a/ab/abcdef0123456789_cache23/abcdef.png.

Thank you for reaching out @Krinkle, your understanding is correct: within a container what matters for iteration over files is the prefix.

In this case I think it makes sense to go score/<version>/a/ab/abcdef0123456789/abcdef.<type>. Both to be able to operate in batches on a version and keep symmetry with media storage with 1+2 scheme.

Not strictly in scope for this task but I thought I'd mention it, score is a single container AFAICT (global-data-score-render) with 3.8M files and 300G (~78k average file size). If performance becomes a problem we can also consider sharding the container into multiple ones like big wikis e.g. global-data-score-render.aa (for comparison commons thumbs containers have average ~128k file size, at ~7M files and ~800G per container). In that case the two letter sharding comes handy to match media storage.

So when lilypond is upgraded, files are regenerated, and the old files stay on disk forever (currently), with no easy way to know which are the old or new versions

I note this isn't currently true. I created rESCRf30683d0d935: Add maintenance script to delete old Score files as part of the issues to clear things up. I never wired it into puppet for automated runs (because Score has been disabled; so wasn't creating any more files at the time).

I'm just running it now (as of writing; but was this morning) upto 20200101000000 (which I'm sure I did before)... Will report back before doing it to a newer date too. Might be worth looking at stats again at that pooint

Ok, I have now deleted everything before 20210101000000

reedy@mwmaint1002:~$ mwscript extensions/Score/maintenance/GetLYFiles.php --wiki=enwiki --date=20210101000000
Total files (all extensions): 251527
53321 ly files created on or after 20210101000000

In T257093#7311136, @fgiunchedi wrote:

Not strictly in scope for this task but I thought I'd mention it, score is a single container AFAICT (global-data-score-render) with 3.8M files and 300G (~78k average file size). If performance becomes a problem we can also consider sharding the container into multiple ones like big wikis e.g. global-data-score-render.aa (for comparison commons thumbs containers have average ~128k file size, at ~7M files and ~800G per container). In that case the two letter sharding comes handy to match media storage.

Be curious to see some updated stats on this now

In T257093#7453291, @Reedy wrote:

Ok, I have now deleted everything before 20210101000000

reedy@mwmaint1002:~$ mwscript extensions/Score/maintenance/GetLYFiles.php --wiki=enwiki --date=20210101000000
Total files (all extensions): 251527
53321 ly files created on or after 20210101000000

In T257093#7311136, @fgiunchedi wrote:

Not strictly in scope for this task but I thought I'd mention it, score is a single container AFAICT (global-data-score-render) with 3.8M files and 300G (~78k average file size). If performance becomes a problem we can also consider sharding the container into multiple ones like big wikis e.g. global-data-score-render.aa (for comparison commons thumbs containers have average ~128k file size, at ~7M files and ~800G per container). In that case the two letter sharding comes handy to match media storage.

Be curious to see some updated stats on this now

We're down to 200k files and 15G:

$ swift stat global-data-score-render
       Container: global-data-score-render
       Objects: 251960
       Bytes: 15426769106

(available with mw swift credentials, and the swift python client)

That's quite a reduction! :)

now that we have svg... should this maybe be run again ?

See also: T427866: Add X-Delete-At header to Score files being stored

@TheDJ sorry, what do you mean by "this", in this context, please? [I don't think there's an action for swift-admin here, but I wanted to make sure]

@MatthewVernon i mean deleting old score files. Specifically their png versions, as we mostly use svg now since a couple of weeks. Left unattended we would be doubling the score storage if we never delete old stuff. (which based on the other ticket, i'm assuming we still don't do)

OK, right, yes, I think that is likely worth doing. Looking at the container now:

root@ms-fe2009:/home/mvernon# swift stat global-data-score-render
             Container: global-data-score-render
               Objects: 3.6M
                 Bytes: 315G

Which would be quite a lot of unnecessary objects to keep forever! I think Score is owned by Reader Experience, so I've tagged them here to take a look. I'm also inclined to agree that adding a TTL might be sensible (though swift TTL is a rather blunt instrument), but I'll let RE take a look.

My uninformed view is that "we want to keep them except when we upgrade lilypond and then want to regenerate everything" might inform a deletion-based approach...?

thanks so much for the tag-in here :D -- while REx is listed as the maintainer, I have to admit that we have little to no expertise on Score as we have never worked on it prior to this since inheriting the maintenance assignment. we could potentially help code review a patch, with the caveat that we have limited expertise in image infra. can you think of anyone on mediawiki platform who might have more related expertise?

in the meantime, could you help me understand what the urgency of this task is, so I can see whether we have any room in upcoming sprints? It also sounds like this script might just need to be run on a regular cadence unless we introduce an explicit TTL?

There is no rush, it’s just that we have 315GB of storage of which a significant part isn’t used anymore because we switched from png to SVG. And the whole Score stack never gets rid of anything that used to once be in wikitext, but is no longer in use. That allows unbounded growth, which generally isn't liked, so it’s something someone should have on the radar.

Ah! thanks for the info :) Looks like Timo is reviewing the patch (thank you Timo!), so I'll move this onto our team's "watching" column for now. Please do ping us if there's direct support needed from us still!!

@HFan-WMF note that whatever ends up happening about that patch, it won't address the problem of storing 300G of useless PNG files, so those will need handling (I would guess using the script in the extension that deletes old images?).

In T257093#12193179, @TheDJ wrote:

There is no rush, it’s just that we have 315GB of storage of which a significant part isn’t used anymore because we switched from png to SVG. And the whole Score stack never gets rid of anything that used to once be in wikitext, but is no longer in use. That allows unbounded growth, which generally isn't liked, so it’s something someone should have on the radar.

Also, it'd be really good to have a solution in place before it becomes urgent :)

I don't think those pngs are in fact useless. I might have the sequencing wrong here, but this is how it looks to me ...

• when a page is parsed the onParserFirstCallInit hook calls Score::render() for each <score> tag
• Score::render() -> Score::renderScore() -> Score::generateHTML()
• if the png does not exist then Score::generatePngAndMidi() gets called
• this creates both svg and png, and uses the dimensions of the png to set width and height in the image metadata
• and then that image metadata is used to set the width and height on the <img> tag for the svg

In T257093#12195665, @Cparle wrote:

I don't think those pngs are in fact useless. I might have the sequencing wrong here, but this is how it looks to me ...

Thanks for the rundown. To state the obvious: if the PNGs are used to calculate the right SVG size, they could be deleted immediately after. Or just search for a solution where the calculation is achieved free of the PNG generation.

In T257093#12195665, @Cparle wrote:

I don't think those pngs are in fact useless. I might have the sequencing wrong here, but this is how it looks to me ...

This is a very good point - now that we don't really need the PNG image generation any more (the SVG ticket took long enough that browser support for SVG became ubiquitous by the time it was deployed), I'd like to try updating the image dimension calculation to avoid PNG images altogether as suggested by @Volker_E

if the PNGs are used to calculate the right SVG size, they could be deleted immediately after

Well not really - if the png is missing when the page gets re-parsed it gets regenerated. We probably don't want this

just search for a solution where the calculation is achieved free of the PNG generation

+1 this seems like the sensible option

Change #1323783 had a related patch set uploaded (by Jonathanischoice; author: Jonathanischoice):

[mediawiki/extensions/Score@master] Use SVG by default and stop using PNG files, add tests for size detection

https://gerrit.wikimedia.org/r/1323783