Page MenuHomePhabricator

Lcobucci\JWT\Signer\InvalidKeyProvided: Key cannot be empty
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   Lcobucci\JWT\Signer\InvalidKeyProvided: Key cannot be empty
exception.trace
from /srv/mediawiki/php-1.40.0-wmf.6/vendor/lcobucci/jwt/src/Signer/InvalidKeyProvided.php(34)
#0 /srv/mediawiki/php-1.40.0-wmf.6/vendor/lcobucci/jwt/src/Signer/Key/InMemory.php(25): Lcobucci\JWT\Signer\InvalidKeyProvided::cannotBeEmpty()
#1 /srv/mediawiki/php-1.40.0-wmf.6/vendor/lcobucci/jwt/src/Signer/Key/InMemory.php(44): Lcobucci\JWT\Signer\Key\InMemory->__construct(string, string)
#2 /srv/mediawiki/php-1.40.0-wmf.6/vendor/league/oauth2-server/src/Entities/Traits/AccessTokenTrait.php(51): Lcobucci\JWT\Signer\Key\InMemory::plainText(string)
#3 /srv/mediawiki/php-1.40.0-wmf.6/vendor/league/oauth2-server/src/Entities/Traits/AccessTokenTrait.php(62): MediaWiki\Extension\OAuth\Entity\AccessTokenEntity->initJwtConfiguration()
#4 /srv/mediawiki/php-1.40.0-wmf.6/vendor/league/oauth2-server/src/Entities/Traits/AccessTokenTrait.php(91): MediaWiki\Extension\OAuth\Entity\AccessTokenEntity->convertToJWT()
#5 /srv/mediawiki/php-1.40.0-wmf.6/extensions/OAuth/src/Rest/Handler/AbstractClientHandler.php(56): MediaWiki\Extension\OAuth\Entity\AccessTokenEntity->__toString()
#6 /srv/mediawiki/php-1.40.0-wmf.6/includes/Rest/Router.php(487): MediaWiki\Extension\OAuth\Rest\Handler\AbstractClientHandler->execute()
#7 /srv/mediawiki/php-1.40.0-wmf.6/includes/Rest/Router.php(406): MediaWiki\Rest\Router->executeHandler(MediaWiki\Extension\OAuth\Rest\Handler\RequestClient)
#8 /srv/mediawiki/php-1.40.0-wmf.6/includes/Rest/EntryPoint.php(170): MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#9 /srv/mediawiki/php-1.40.0-wmf.6/includes/Rest/EntryPoint.php(135): MediaWiki\Rest\EntryPoint->execute()
#10 /srv/mediawiki/php-1.40.0-wmf.6/rest.php(31): MediaWiki\Rest\EntryPoint::main()
#11 /srv/mediawiki/w/rest.php(3): require(string)
#12 {main}
Impact

No idea ;)

Notes

There are a few, apparently all occurring on https://meta.wikimedia.org

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
hashar triaged this task as Unbreak Now! priority.Oct 19 2022, 8:35 AM

That is a new error in 1.40.0-wmf.6

Change 844442 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/vendor@master] Downgrade lcobucci/jwt (4.2.1 => 4.1.5)

https://gerrit.wikimedia.org/r/844442

Change 844035 had a related patch set uploaded (by Hashar; author: Zabe):

[mediawiki/vendor@wmf/1.40.0-wmf.6] Downgrade lcobucci/jwt (4.2.1 => 4.1.5)

https://gerrit.wikimedia.org/r/844035

Change 844442 merged by jenkins-bot:

[mediawiki/vendor@master] Downgrade lcobucci/jwt (4.2.1 => 4.1.5)

https://gerrit.wikimedia.org/r/844442

Change 844035 merged by jenkins-bot:

[mediawiki/vendor@wmf/1.40.0-wmf.6] Downgrade lcobucci/jwt (4.2.1 => 4.1.5)

https://gerrit.wikimedia.org/r/844035

Mentioned in SAL (#wikimedia-operations) [2022-10-19T14:25:32Z] <hashar@deploy1002> Started scap: Backport for [[gerrit:844035|Downgrade lcobucci/jwt (4.2.1 => 4.1.5) (T321160)]]

Mentioned in SAL (#wikimedia-operations) [2022-10-19T14:25:58Z] <hashar@deploy1002> hashar and hashar: Backport for [[gerrit:844035|Downgrade lcobucci/jwt (4.2.1 => 4.1.5) (T321160)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-10-19T14:29:00Z] <hashar@deploy1002> sync-world aborted: Backport for [[gerrit:844035|Downgrade lcobucci/jwt (4.2.1 => 4.1.5) (T321160)]] (duration: 03m 27s)

Mentioned in SAL (#wikimedia-operations) [2022-10-19T14:29:35Z] <hashar@deploy1002> Started scap: Backport for [[gerrit:844035|Downgrade lcobucci/jwt (4.2.1 => 4.1.5) (T321160)]]

Mentioned in SAL (#wikimedia-operations) [2022-10-19T14:29:58Z] <hashar@deploy1002> hashar and hashar: Backport for [[gerrit:844035|Downgrade lcobucci/jwt (4.2.1 => 4.1.5) (T321160)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-10-19T14:34:00Z] <hashar@deploy1002> Finished scap: Backport for [[gerrit:844035|Downgrade lcobucci/jwt (4.2.1 => 4.1.5) (T321160)]] (duration: 04m 25s)

Should be good. Thank you @Zabe

I am guessing MediaWiki-extensions-OAuth could use an integration test to cover the 4.2.1 incompatibility, I imagine the update might be attempted again and there is no test covering it.

hashar lowered the priority of this task from Unbreak Now! to High.Oct 20 2022, 3:11 PM

That is no more an unbreak now priority since lcobucci/jwt got downgraded. We would still need a test in MediaWiki-extensions-OATHAuth to cover the 4.1.5 requirement and prevent a future unattended upgrade to 4.2.1.

It looks like upstream league/oauth2-server fixed this in Fix compatibility with lcobucci/jwt ^4.2, included in 8.3.5, but Extension:OAuth currently uses a custom 9.0.0 branch.

Change 860005 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/OAuth@master] Pin lcobucci/jwt to compatible version (4.1.5)

https://gerrit.wikimedia.org/r/860005

Uploaded a patch to pin the version, e.g. for local dev environments (since I just ran into the same error).

Change 860005 merged by jenkins-bot:

[mediawiki/extensions/OAuth@master] Pin lcobucci/jwt to compatible version (4.1.5)

https://gerrit.wikimedia.org/r/860005

Change 881860 had a related patch set uploaded (by Reedy; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/OAuth@REL1_39] Pin lcobucci/jwt to compatible version (4.1.5)

https://gerrit.wikimedia.org/r/881860

Change 881860 merged by jenkins-bot:

[mediawiki/extensions/OAuth@REL1_39] Pin lcobucci/jwt to compatible version (4.1.5)

https://gerrit.wikimedia.org/r/881860

Change 882176 had a related patch set uploaded (by Gergő Tisza; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/OAuth@REL1_35] Pin lcobucci/jwt to compatible version (4.1.5)

https://gerrit.wikimedia.org/r/882176

Change 882177 had a related patch set uploaded (by Gergő Tisza; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/OAuth@REL1_38] Pin lcobucci/jwt to compatible version (4.1.5)

https://gerrit.wikimedia.org/r/882177

Change 882177 merged by jenkins-bot:

[mediawiki/extensions/OAuth@REL1_38] Pin lcobucci/jwt to compatible version (4.1.5)

https://gerrit.wikimedia.org/r/882177

Change 882176 merged by jenkins-bot:

[mediawiki/extensions/OAuth@REL1_35] Pin lcobucci/jwt to compatible version (4.1.5)

https://gerrit.wikimedia.org/r/882176

Tgr assigned this task to Zabe.

Nothing left to do here, I think; there are a bunch of other tasks for figuring out the oauth2-server versioning mess in general (T279837, T261462, T319685).

Thank you to have followed up on that ask with all the releases backports!

Change 897363 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/OAuth@master] build: Remove pinning of indirect lcobucci/jwt dependency

https://gerrit.wikimedia.org/r/897363

Change 880491 had a related patch set uploaded (by Krinkle; author: Reedy):

[mediawiki/vendor@master] Upgrading lcobucci/jwt (4.1.5 => 4.3.0)

https://gerrit.wikimedia.org/r/880491

Change 897363 merged by jenkins-bot:

[mediawiki/extensions/OAuth@master] build: Remove pinning of indirect lcobucci/jwt dependency

https://gerrit.wikimedia.org/r/897363

Change 880491 merged by jenkins-bot:

[mediawiki/vendor@master] Upgrading lcobucci/jwt (4.1.5 => 4.3.0)

https://gerrit.wikimedia.org/r/880491

jeena raised the priority of this task from High to Unbreak Now!.Mar 15 2023, 9:31 PM

So we have three options:

The second seems the most straightforward, but then I have no idea why we were pinning in the first place. @Reedy @Krinkle any thoughts?

It's pinned to the exact hash in wikimedia/oauth2-server (rEOAUaf26a29b05e2: composer.json: Pin league/oauth2-server to commit) because when I tried to update it after a rebase, it failed with some errors that looking at the vendor patch, I tagged in T302757: OAuth test failures. https://github.com/thephpleague/oauth2-server/issues/1266 as the upstream which was closed without anything really being resolved, even months later...

The simplest fix was then to just pin it to the last known good working hash.

Should be testable using the OAuth extension commit in gerrit, depending upon a newer commit for the forked library in vendor.

But it doesn't look like upstream have actually fixed the underlying error; at least not on master.

It sounds like it's worth rebasing against the upstream to get the commits that it is currently ahead -
https://github.com/wikimedia/oauth2-server/compare/v9.0.0-alpha...thephpleague:oauth2-server:master (lazily done into https://github.com/wikimedia/oauth2-server/pull/17) but needs some conflicts resolving.

It may help clean up the lot for a bit.

But the immediate issue here is not v9.0.0-alpha being outdated, it's that currently our composer.json (both vendor and the extension) pin it to a specific commit that was the state of v9.0.0-alpha circa 2021 and is way behind current v9.0.0-alpha. Using v9.0.0-alpha without commit-level pinning would be fine.

I'd lean toward using current jwt (i.e. no pinning), and using or making a version of oauth2 that works with it - if feasible.

But the immediate issue here is not v9.0.0-alpha being outdated, it's that currently our composer.json (both vendor and the extension) pin it to a specific commit that was the state of v9.0.0-alpha circa 2021 and is way behind current v9.0.0-alpha. Using v9.0.0-alpha without commit-level pinning would be fine.

And as I said above...

It's pinned to the exact hash in wikimedia/oauth2-server (rEOAUaf26a29b05e2: composer.json: Pin league/oauth2-server to commit) because when I tried to update it after a rebase, it failed with some errors that looking at the vendor patch, I tagged in T302757: OAuth test failures. https://github.com/thephpleague/oauth2-server/issues/1266 as the upstream which was closed without anything really being resolved, even months later...

Removing the hash level pinning might fix this exact issue, but it's not going to pass CI due to the aforementioned issues.

Change 900140 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/OAuth@master] Revert "build: Remove pinning of indirect lcobucci/jwt dependency"

https://gerrit.wikimedia.org/r/900140

Change 900409 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/vendor@master] Revert "Upgrading lcobucci/jwt (4.1.5 => 4.3.0)"

https://gerrit.wikimedia.org/r/900409

In hindsight this should probably have been higher priority. OAuth 2 is now broken entirely. It does not affect group 2 yet, but it's pretty common to use meta or mw.org for remote login. See e.g. T332286: Login to Diff blog with Wikimedia account ends with SSO error Sorry for not realizing that in time.

We can roll back if necessary, but looks like @Jdforrester-WMF's patch above should be ready shortly...

Change 900425 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/vendor@REL1_40] Revert "Upgrading lcobucci/jwt (4.1.5 => 4.3.0)"

https://gerrit.wikimedia.org/r/900425

Change 900425 merged by Jforrester:

[mediawiki/vendor@REL1_40] Revert "Upgrading lcobucci/jwt (4.1.5 => 4.3.0)"

https://gerrit.wikimedia.org/r/900425

Change 900143 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/OAuth@REL1_40] Revert "build: Remove pinning of indirect lcobucci/jwt dependency"

https://gerrit.wikimedia.org/r/900143

Change 900427 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/vendor@wmf/1.40.0-wmf.27] Revert "Upgrading lcobucci/jwt (4.1.5 => 4.3.0)"

https://gerrit.wikimedia.org/r/900427

Change 900144 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/OAuth@wmf/1.40.0-wmf.27] Revert "build: Remove pinning of indirect lcobucci/jwt dependency"

https://gerrit.wikimedia.org/r/900144

Change 900409 merged by jenkins-bot:

[mediawiki/vendor@master] Revert "Upgrading lcobucci/jwt (4.1.5 => 4.3.0)"

https://gerrit.wikimedia.org/r/900409

Change 900140 merged by jenkins-bot:

[mediawiki/extensions/OAuth@master] Revert "build: Remove pinning of indirect lcobucci/jwt dependency"

https://gerrit.wikimedia.org/r/900140

Change 900143 merged by jenkins-bot:

[mediawiki/extensions/OAuth@REL1_40] Revert "build: Remove pinning of indirect lcobucci/jwt dependency"

https://gerrit.wikimedia.org/r/900143

Change 900427 merged by jenkins-bot:

[mediawiki/vendor@wmf/1.40.0-wmf.27] Revert "Upgrading lcobucci/jwt (4.1.5 => 4.3.0)"

https://gerrit.wikimedia.org/r/900427

Mentioned in SAL (#wikimedia-operations) [2023-03-16T20:12:43Z] <brennen@deploy2002> Started scap: Backport for [[gerrit:900427|Revert "Upgrading lcobucci/jwt (4.1.5 => 4.3.0)" (T321160)]]

Mentioned in SAL (#wikimedia-operations) [2023-03-16T20:14:38Z] <brennen@deploy2002> brennen and jforrester: Backport for [[gerrit:900427|Revert "Upgrading lcobucci/jwt (4.1.5 => 4.3.0)" (T321160)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-03-16T20:21:49Z] <brennen@deploy2002> Finished scap: Backport for [[gerrit:900427|Revert "Upgrading lcobucci/jwt (4.1.5 => 4.3.0)" (T321160)]] (duration: 09m 06s)

brennen lowered the priority of this task from Unbreak Now! to Needs Triage.Mar 16 2023, 8:54 PM

Seems fixed in production; removing as train blocker, leaving open for any needed followup.

There are some editors who still cannot log in to their Wikipedia Library account (T332349). We are using mwoauth for authentication.

Thank you @Jdforrester-WMF for fixing.

logstash says there were 167 errors, which is surprisingly low.

But in general, OAuth is one of the tools (along with some other "global" extensions like CentralAuth) where group1 (meta) is used even for things that are, from a user perspective, Wikipedia-related (e.g. the Wikipedia Library), so we should probably react more aggressively to group1 errors coming from such extensions. @brennen not sure if there's a good place to document that.

The revert has been deployed to production and backported to 1.40 so I think this task is resolved - T261462: Migrate OAuth extension back from wikimedia/oauth2-server fork to upstream can be used for following up.

But in general, OAuth is one of the tools (along with some other "global" extensions like CentralAuth) where group1 (meta) is used even for things that are, from a user perspective, Wikipedia-related (e.g. the Wikipedia Library), so we should probably react more aggressively to group1 errors coming from such extensions. @brennen not sure if there's a good place to document that.

Good thought.

I believe wikitech:Deployments/Holding_the_train#Issues_that_hold_the_train is considered canonical for this, though it could probably use some updating.

Change 900144 abandoned by Jforrester:

[mediawiki/extensions/OAuth@wmf/1.40.0-wmf.27] Revert "build: Remove pinning of indirect lcobucci/jwt dependency"

Reason:

We didn't need to backport so this wasn't needed in the end.

https://gerrit.wikimedia.org/r/900144