As far as I'm aware swift is not publicly accessible, using the key for anything requires being inside the prod network already. Not sure how that plays into risk assessment.

In T296767#7538451, @EBernhardson wrote:

As far as I'm aware swift is not publicly accessible, using the key for anything requires being inside the prod network already. Not sure how that plays into risk assessment.

"we" (SRE) have rotated passwords when similar things have happened in the past (ie mysql passwords being posted), even though those services are similarly not externally accessible

Changing a Swift password is easy - update private puppet, rolling restart of swift front-ends.

The tricky bit is the timing - once we deploy the change by restarting the frontends, new connections will start to fail until clients get the new one (AIUI, established sessions have auth tokens that persist for a while); puppet will update the credentials on puppet-managed systems on its next run (which could be forced for a suitable subset of machines). AIUI, though, mw has a separate repo which has the credential baked into it?

If that's correct, then we'll need to co-ordinate such that the new credential is deployed to clients at (about) the same time as we do the rolling restart of the frontends. @EBernhardson are you the right person to know how this credential is deployed and when is going to be a good time to do its rollover?

In T296767#7540016, @MatthewVernon wrote:

AIUI, though, mw has a separate repo which has the credential baked into it?

On deploy1001 there's a private Git repository at /srv/mediawiki-staging/private, one of those files will have the credential. It will need to be synced out using scap sync-file, which will take a (literal) minute. But MediaWiki does cache the x-auth-token in APCu, in case that makes a difference.

In T296767#7540016, @MatthewVernon wrote:

If that's correct, then we'll need to co-ordinate such that the new credential is deployed to clients at (about) the same time as we do the rolling restart of the frontends. @EBernhardson are you the right person to know how this credential is deployed and when is going to be a good time to do its rollover?

I'm generally familiar with swift, but nothing particularly in depth or related to mw's usage. The timing does sound like a potential issue. Checking the history of the mw private repo, this key has been constant since the first commit to it in 2016 suggesting we wont find much local experience around this. It seems we could test if X-Auth-Token's issued prior to a key change are still valid after the key change, likely against the prod cluster. I can help testing that if needed, but don't have access to create/change accounts.

@Legoktm it sounds like we might usefully co-ordinate this (scap and frontend restart) one day next week? In particular if you wouldn't mind keeping an eye on the MW side that everything keeps behaving itself [and I'll keep a copy of the old credential in case we need to rollback]

The Security-Team triaged this at our clinic today. While we agree with the assessment that this is likely low risk for now, given how it would need to be exploited on our internal production networks, we obviously don't want to sit too long on rotating production credentials.

@sbassett we're going to update the key on Thursday (9th), ~16:30 UTC.

In T296767#7552457, @MatthewVernon wrote:

@sbassett we're going to update the key on Thursday (9th), ~16:30 UTC.

Awesome, thanks!