It's worth mentioning that, like Extension:Score, Phonos will have a configurable backend. A custom FSFileBackend is the default if no backend is specified with the wgPhonosFileBackend configuration setting. I see in production Score is set to use global-multiwrite, which may or may not be what we'll want to use in production as well.

I don't fully understand how FSFileBackend will work here, as the files would only be stored on a single server. How are the files being served to users? I would expect they are still served via upload.wikimedia.org like other multimedia things, which means they should be in Swift.

For Score we just delete very old files every so often (https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Score/+/refs/heads/master/maintenance/DeleteOldScoreFiles.php), typically after a version bump in which case the cache is already invalidated. I would suggest something similar to Phonos. That said, there's been some interesting discussion in the past week in #wikimedia-perf on whether cleaning up old thumbs (also small files) is worth it given how cheap storage is: T211661#8110858.

That's very helpful, thanks Legoktm! I am about to find out if/how FSFileBackend would even work. I've been going off of Score's code so it seems it's doable. I can report back when I know more, but for this task I suppose it doesn't matter as we're asking about what we need to do for WMF production. It sounds like Swift is the only way to go there, but for local dev environments and other MediaWiki installations, we'll find a way with FSFileBackend (assuming I can get it to work!).

Another thing I wanted to mention as far as storage footprint / network performance: we currently are rendering WAV files. This is because I think two of the engines (Larnyx and eSpeak) only deliver WAV. However in production we're likely going to be using Google, which can deliver MP3 or OGG. These two formats I presume are preferred over the lossless WAV. So we can definitely save as MP3 or OGG if necessary, but if we do I suppose we'll have to do some conversion or special handling for Larnyx and eSpeak.

In T314789#8138964, @MusikAnimal wrote:

That's very helpful, thanks Legoktm! I am about to find out if/how FSFileBackend would even work. I've been going off of Score's code so it seems it's doable. I can report back when I know more, but for this task I suppose it doesn't matter as we're asking about what we need to do for WMF production. It sounds like Swift is the only way to go there, but for local dev environments and other MediaWiki installations, we'll find a way with FSFileBackend (assuming I can get it to work!).

I would recommend just copying the code out of Score to use a specific file backend if configured, otherwise defaulting to FSFileBackend. It should just work. I recently did that for EasyTimeline (c.f. 6627305b79cd4bf079e2c58f33033846f6c11d2a) and it cleaned up the code a lot.

Another thing I wanted to mention as far as storage footprint / network performance: we currently are rendering WAV files. This is because I think two of the engines (Larnyx and eSpeak) only deliver WAV. However in production we're likely going to be using Google, which can deliver MP3 or OGG. These two formats I presume are preferred over the lossless WAV. So we can definitely save as MP3 or OGG if necessary, but if we do I suppose we'll have to do some conversion or special handling for Larnyx and eSpeak.

IMO the most important consideration of file formats would be client support. I would suggest discussing this with the TimedMediaHandler-Transcode experts, Brion and TheDJ.

In T314789#8139432, @Legoktm wrote:

IMO the most important consideration of file formats would be client support. I would suggest discussing this with the TimedMediaHandler-Transcode experts, Brion and TheDJ.

When possible use mp3. It's the most universally supported format and no longer patent encumbered. Ogg/opus or ogg/vorbis specifically do not play natively in iOS. While WAV is lossless, it is also uncompressed, which makes it rather wasteful as a storage and/or transfer format.

You can use lame and/or ffmpeg to convert from wav to mp3 via a pipe on the fly if required.

In T314789#8140256, @TheDJ wrote:

You can use lame and/or ffmpeg to convert from wav to mp3 via a pipe on the fly if required.

I know Google at least supports returning mp3 directly.. Larynx we can fairly easily patch in support for returning mp3s too

In T314789#8139432, @Legoktm wrote:

I would recommend just copying the code out of Score to use a specific file backend if configured, otherwise defaulting to FSFileBackend. It should just work. I recently did that for EasyTimeline (c.f. 6627305b79cd4bf079e2c58f33033846f6c11d2a) and it cleaned up the code a lot.

Yeah that's basically what I'm doing (I'm only about halfway done though). Since you brought it up, I wanted to ask: should we be using the containerPaths option in FSFileBackend? The comments say "This should only be used for backwards-compatibility" which isn't the case for us as its a new extension. So I'm not doing a straight/copy paste of Score but can confirm FSFileBackend does in fact work!

Another source of confusion that I wanted to just say out loud was that 0777 (ref) is not interpreted as an integer in PHP but ends up becoming 511, which is correct, but I spent about an hour debugging this before I finally figured out what was going on! I was getting permission errors which would make sense for a directory with 511 which is -r-x--x--x. Turns out I was using $wgUploadPath instead of $wgUploadDirectory. I may make a separate patch to add a note about this in FSFileBackend::__construct().

In T314789#8140256, @TheDJ wrote:

You can use lame and/or ffmpeg to convert from wav to mp3 via a pipe on the fly if required.

And we already use lame (via shellbox) for Score so it should be straightforward to get set up.

In T314789#8143769, @MusikAnimal wrote:

Yeah that's basically what I'm doing (I'm only about halfway done though). Since you brought it up, I wanted to ask: should we be using the containerPaths option in FSFileBackend? The comments say "This should only be used for backwards-compatibility" which isn't the case for us as its a new extension. So I'm not doing a straight/copy paste of Score but can confirm FSFileBackend does in fact work!

I think you can skip it - I only skimmed the FileBackend code, but it does in fact look only needed for back-compat.

Thanks, all! I've created T315119: Use MP3 instead of WAV for all engines except Larnyx and have already started working on it. That should bring down our estimated storage considerably.