T317130: Test S3 / Swift would have caught this, apologies :/

https://www.mediawiki.org/wiki/Topic:Shsy2ih2cft90uc8 seems related — resolved for them by:

I'd say your $wgUploadDirectory is wrong. Shouldn't it be /home/spydar007/wiki/images instead of /images?

I recall a Phonos-specific setting which relates to $wgUploadDirectory, will look..

In T317195#8217591, @TheresNoTime wrote:

…
I recall a Phonos-specific setting which relates to $wgUploadDirectory, will look..

You could try setting $wgPhonosFileBackendDirectory. But to make it use Swift, I think you need to change $wgPhonosFileBackend (not sure what the Swift backend is called), which will then use whatever storage location is set by that backend, and $wgPhonosFileBackendDirectory will be ignored.

I note https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/543217 which has:

'wgScoreFileBackend' => [
	'default' => 'global-multiwrite'
],

Mentioned in SAL (#wikimedia-releng) [2022-09-07T15:05:55Z] <TheresNoTime> making hack changes to beta to test T317195 resolution

Hacking in

$wgPhonosFileBackend = 'global-multiwrite';
$wgPhonosFileBackendDirectory = '//upload.wikimedia.beta.wmflabs.org/phonos'; // copying Score

has progressed us further:

<a class="oo-ui-buttonElement-button oo-ui-pendingElement-pending" role="button" tabindex="0" href="http://deployment-ms-fe03.deployment-prep.eqiad.wmflabs/v1/AUTH_mw/global-data-phonos/4b67c6b189cdf4ad2d46a9fa6bac2813.mp3?temp_url_sig=[snip]&amp;temp_url_expires=1662649816" rel="nofollow" aria-description="Listen to the IPA pronunciation of hello."></a>

but now getting an (understandable) error when trying to play the audio file — namely that the mp3 at http://deployment-ms-fe03.deployment-prep.eqiad.wmflabs/v1/AUTH_mw/global-data-phonos/4b67c6b189cdf4ad2d46a9fa6bac2813.mp3?temp_url_sig=[snip]&temp_url_expires=1662649816 is not reachable...

deployment-ms-fe03.deployment-prep.eqiad.wmflabs is the mediaSwiftStore for beta, but why isn't upload.wikimedia.beta.wmflabs.org/phonos being used instead?

Removing $wgPhonosFileBackendDirectory entirely has the same result

$wgPhonosFileBackendDirectory isn't used when $wgPhonosFileBackend is set, and I thought Extension:Score was programmed the same way (the backend itself is supposed to set the directory, so $wgPhonosFileBackendDirectory is only for FSFileBackend). It should use whatever mwstore://global-multiwrite/phonos resolves to.

It's very possible we programmed something wrong in Phonos, though, because we never tested using anything other than FSFileBackend.

I just tried setting $wgPhonosPath = '//upload.wikimedia.beta.wmflabs.org/phonos'; too, same issue.

In T317195#8217881, @MusikAnimal wrote:

$wgPhonosFileBackendDirectory isn't used when $wgPhonosFileBackend is set, and I thought Extension:Score was programmed the same way (the backend itself is supposed to set the directory, so $wgPhonosFileBackendDirectory is only for FSFileBackend). It should use whatever mwstore://global-multiwrite/phonos resolves to.

Ah, it would make sense that mwstore://global-multiwrite/phonos would resolve to deployment-ms-fe03.deployment-prep.eqiad.wmflabs — we may need to make a change so that $wgPhonosPath can override the public file URL

Change 830657 had a related patch set uploaded (by Samtar; author: Samtar):

[operations/mediawiki-config@master] CommonSettings-labs.php: Set $wgPhonosFileBackend

https://gerrit.wikimedia.org/r/830657

I suspect the issue is at Engine.php#L119-L121 which has never been tested. FileBackend::getFileHttpUrl() sounded like what we wanted, so I just put it there... hehe. Extension:Score uses a different system. Perhaps getFileHttpUrl() isn't what we want at all. I'll try to investigate.

Or do we need to use SwiftFileBackend instead of the global-multiwrite one I copied from Score? 😬

Rather than me continuing to guess (and breaking things), perhaps someone from Data-Persistence (work done) could point me in the right direction?

Apologies for the random subscribe @Legoktm, but you're helpful and knowledgeable (always a mistake! /s) — do you have any pointers/ideas/links to a tutorial I've entirely missed?

Confirmed global-multiwrite is the correct backend here

Change 830657 merged by jenkins-bot:

[operations/mediawiki-config@master] CommonSettings-labs.php: Set $wgPhonosFileBackend

https://gerrit.wikimedia.org/r/830657

Context:

To test Phonos in a near-production env we need it to be working on the beta cluster. To get it to work on the beta cluster, we need to figure out how to get phonos to correctly use the Swift backend. To use the swift backend, swift needs to be working on the beta cluster.
The "other" issue is we don't know if we've cracked how to get phonos and swift to work properly until we can test that on the beta cluster, which also depends on swift working on the beta cluster.
Swift working on the beta cluster doesn't appear to be high on any other team's priorities as it "works enough" to serve what's already deployed.

Decisions to be made. Right now, we are currently working on and deciding between:

• Holding off until security review and testing on test wiki. This has the downside of being blocked and not being able to test for a couple of weeks.
• Finding the internal team in charge of Swift/beta integration and asking them to help unblock us and walk us through the lack of documentation.