500 status responses and fatal errors for OAuth authenticated API requests on beta servers
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Chippyy
	Mar 22 2016, 2:23 PM

Description

Hello,

I'm developing the Wikimaps Warper (http://warper.wmflabs.org/) locally which uses the API with authenticated calls via OAuth and locally the code communicates with commons.wikimedia.beta.wmflabs.org The application is getting a 500 status from GET and POST requests to the API called via OAuth.

I'm also seeing the same error response using the Ruby OAuth library and crafting the request, authentication etc manually in the command line.

I'm also seeing the same error using https://tools.wmflabs.org/oauth-hello-world/beta.php which talks to deployment.wikimedia.beta.wmflabs.org
The "Post to your talk page" link should write "Hello World" to the users talk page. It has worked in the past: http://deployment.wikimedia.beta.wmflabs.org/wiki/User:Chippyy

Authenticating the user succeeds via OAuth, and getting user details via the JWT token also succeeds with no errors.

One example of the request that I'm trying is a GET request http://commons.wikimedia.beta.wmflabs.org/w/ap
i.php?action=query&meta=userinfo&uiprop=rights|editcount&format=json"

and the hello-world request is a POST request seen in the code: https://tools.wmflabs.org/oauth-hello-world/beta.php?action=download

The error in the response body is:

PHP fatal error: <br/>
        Argument 3 to hash_hmac() must be of type ?string, bool given

I imagine this may be related to https://phabricator.wikimedia.org/T126316 "Catchable fatal error: Argument 3 to hash_hmac() must be of type ?string, bool given in /srv/mediawiki/php-master/includes/utils/MWCryptHash.php on line 108" related to the beta cluster and on which task I added a comment yesterday but I thought I would create a new one for this issue as it seems more specific.

many thanks in advance,

Tim

(p.s. I hope that I've added the correct tags to the project, I couldn't find one that just dealt with things on beta servers)

Related Objects

Mentioned In: T130896: Getting logged out frequently in Beta Cluster
T129763: beta wikipedia test builds fail
T126316: Catchable fatal error: Argument 3 to hash_hmac() must be of type ?string, bool given in /srv/mediawiki/php-master/includes/utils/MWCryptHash.php on line 108
Mentioned Here: T129763: beta wikipedia test builds fail

Event Timeline

Chippyy created this task.Mar 22 2016, 2:23 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 22 2016, 2:23 PM

Tgr mentioned this in T126316: Catchable fatal error: Argument 3 to hash_hmac() must be of type ?string, bool given in /srv/mediawiki/php-master/includes/utils/MWCryptHash.php on line 108.Mar 23 2016, 10:49 AM

Chippyy added a project: wikimaps-warper.Mar 23 2016, 11:35 AM

When retracing the steps locally and forcing hash_hmac to throw an exception, this is the stack trace I get:

Unparsable API response:
Exception encountered, of type "Exception"
[1cfb3e16] /w/api.php   Exception from line 108 of /vagrant/mediawiki/includes/utils/MWCryptHash.php: hash_hmac
Backtrace:
#0 /vagrant/mediawiki/includes/session/SessionProvider.php(479): MWCryptHash::hmac(string, string, boolean)
#1 /vagrant/mediawiki/extensions/OAuth/api/MWOAuthSessionProvider.php(126): MediaWiki\Session\SessionProvider->hashToSessionId(string)
#2 /vagrant/mediawiki/includes/session/SessionManager.php(640): MediaWiki\Extensions\OAuth\MWOAuthSessionProvider->provideSessionInfo(WebRequest)
#3 /vagrant/mediawiki/includes/session/SessionManager.php(181): MediaWiki\Session\SessionManager->getSessionInfoForRequest(WebRequest)
#4 /vagrant/mediawiki/includes/WebRequest.php(664): MediaWiki\Session\SessionManager->getSessionForRequest(WebRequest)
#5 /vagrant/mediawiki/includes/session/SessionManager.php(120): WebRequest->getSession()
#6 /vagrant/mediawiki/includes/Setup.php(743): MediaWiki\Session\SessionManager::getGlobalSession()
#7 /vagrant/mediawiki/includes/WebStart.php(133): require_once(string)
#8 /vagrant/mediawiki/api.php(38): require(string)
#9 /var/www/w/api.php(5): require(string)
#10 {main}

SessionProvider->hashToSessionId() calls MWCryptHash::hmac with $key ?: $this->config->get( 'SecretKey' ); apparently $wgSecretKey is not set on the beta cluster.

I have set $wgSecretKey on beta but it didn't help. https://gerrit.wikimedia.org/r/279413 did not help either, it still throws a fatal error, although now one with no message. (Also OAuth fails all the time with random nondeterministic exceptions, with nothing logged in OAuth.log. I suspect that's due to memcached problems, not sure why the errors are not logged though.)

OAuth on beta is fully broken now... A typical OAuth log excerpt is

2016-03-25 08:25:58 deployment-mediawiki02 deploymentwiki 1.27.0-alpha OAuth DEBUG: MediaWiki\Extensions\OAuth\MWOAuthRequest::fromRequest: parameters:
Array
(
    [title] => Special:OAuth/initiate
    [format] => json
    [oauth_callback] => oob
    [oauth_consumer_key] => 1d17b68a7c5e105d57f187d6e705e8cf
    [oauth_version] => 1.0
    [oauth_nonce] => 6fe2c73234372b1530c1a792d244cbca
    [oauth_timestamp] => 1458894357
    [oauth_signature_method] => HMAC-SHA1
    [oauth_signature] => kqVtnffHhXlhi6gjMI94L6bGf/4=
)
  
2016-03-25 08:25:58 deployment-mediawiki02 deploymentwiki 1.27.0-alpha OAuth DEBUG: MediaWiki\Extensions\OAuth\SpecialMWOAuth::execute: Consumer '1d17b68a7c5e105d57f187d6e705e8cf' getting temporary credentials  
2016-03-25 08:25:58 deployment-mediawiki02 deploymentwiki 1.27.0-alpha OAuth DEBUG: MediaWiki\Extensions\OAuth\OAuthServer::get_consumer: getting consumer for '1d17b68a7c5e105d57f187d6e705e8cf'  
2016-03-25 08:25:58 deployment-mediawiki02 deploymentwiki 1.27.0-alpha OAuth DEBUG: MediaWiki\Extensions\OAuth\OAuthSignatureMethod::check_signature: Expecting: 'kqVtnffHhXlhi6gjMI94L6bGf/4='  
2016-03-25 08:25:58 deployment-mediawiki02 deploymentwiki 1.27.0-alpha OAuth DEBUG: MediaWiki\Extensions\OAuth\OAuthUtil::build_http_query called with params:
Array
(
    [title] => Special:OAuth/initiate
    [format] => json
    [oauth_callback] => oob
    [oauth_consumer_key] => 1d17b68a7c5e105d57f187d6e705e8cf
    [oauth_version] => 1.0
    [oauth_nonce] => 6fe2c73234372b1530c1a792d244cbca
    [oauth_timestamp] => 1458894357
    [oauth_signature_method] => HMAC-SHA1
)
  
2016-03-25 08:25:58 deployment-mediawiki02 deploymentwiki 1.27.0-alpha OAuth DEBUG: MediaWiki\Extensions\OAuth\OAuthSignatureMethod_HMAC_SHA1::build_signature: Base string: 'GET&http%3A%2F%2Fdeployment.wikimedia.beta.wmflabs.org%2Fw%2Findex.php&format%3Djson%26oauth_callback%3Doob%26oauth_consumer_key%3D1d17b68a7c5e105d57f187d6e705e8cf%26oauth_nonce%3D6fe2c73234372b1530c1a792d244cbca%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1458894357%26oauth_version%3D1.0%26title%3DSpecial%253AOAuth%252Finitiate'  
2016-03-25 08:25:58 deployment-mediawiki02 deploymentwiki 1.27.0-alpha OAuth DEBUG: MediaWiki\Extensions\OAuth\OAuthSignatureMethod_HMAC_SHA1::build_signature: HMAC Key: '9ce92f<redacted>&'  
2016-03-25 08:25:58 deployment-mediawiki02 deploymentwiki 1.27.0-alpha OAuth DEBUG: MediaWiki\Extensions\OAuth\OAuthSignatureMethod::check_signature: Built: 'h7CqJZmffDBktGO2hQYx/m5oG6M='  
2016-03-25 08:25:58 deployment-mediawiki02 deploymentwiki 1.27.0-alpha OAuth INFO: MediaWiki\Extensions\OAuth\OAuthServer::check_signature: Signature check (MediaWiki\Extensions\OAuth\OAuthSignatureMethod_HMAC_SHA1) failed  
2016-03-25 08:25:58 deployment-mediawiki02 deploymentwiki 1.27.0-alpha OAuth WARNING: MediaWiki\Extensions\OAuth\SpecialMWOAuth::execute: Exception Invalid signature  
[Exception MediaWiki\Extensions\OAuth\OAuthException] (/srv/mediawiki/php-master/extensions/OAuth/lib/OAuth.php:720) Invalid signature
  #0 /srv/mediawiki/php-master/extensions/OAuth/backend/MWOAuthServer.php(49): MediaWiki\Extensions\OAuth\OAuthServer->check_signature(MediaWiki\Extensions\OAuth\MWOAuthRequest, MediaWiki\Extensions\OAuth\MWOAuthConsumer, NULL)
  #1 /srv/mediawiki/php-master/extensions/OAuth/frontend/specialpages/SpecialMWOAuth.php(61): MediaWiki\Extensions\OAuth\MWOAuthServer->fetch_request_token(MediaWiki\Extensions\OAuth\MWOAuthRequest)
  #2 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(407): MediaWiki\Extensions\OAuth\SpecialMWOAuth->execute(string)
  #3 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(565): SpecialPage->run(string)
  #4 /srv/mediawiki/php-master/includes/MediaWiki.php(282): SpecialPageFactory::executePath(Title, RequestContext)
  #5 /srv/mediawiki/php-master/includes/MediaWiki.php(738): MediaWiki->performRequest()
  #6 /srv/mediawiki/php-master/includes/MediaWiki.php(519): MediaWiki->main()
  #7 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
  #8 /srv/mediawiki/w/index.php(3): include(string)
  #9 {main}

Not sure where that HMAC key comes from, it should be the secret key of the consumer but it's not. I f I replace it with the correct key, I get the expected signature.

The HMAC key is from $consumer->secret which is HMAC'd from $wgOAuthSecretKey (which falls back to $wgSecretKey) and $consumer->secretKey if the global is set, and is simply $consumer->secretKey otherwise. So either that HMAC is applied inconsistently (which should not be the case since the same setup works in production) or somehow the global is not always picked up in labs.

Turns out I didn't quite understand how the OAuth extension works. The secret key stored in the DB is not actually the same as the consumer secret of the OAuth protocol; it's HMAC'd with the MediaWiki secret key first, as an extra layer of defense against SQL injection. So setting the secret key for the beta cluster changed the consumer secret of all OAuth consumers :(

I'd rather not unset it because the beta cluster should be configured similarly to production where possible, but that means beta OAuth apps will have to update their secret key. Sorry about that :(

Pinging @Anomie, @Chippyy, @Krenair, @jayvdb (guessing User:Pywikibot-oauth is him).

My application in beta was just for testing and doesn't matter really.

Is this the cause of T129763: beta wikipedia test builds fail ?

Pywikibot-oauth is anyone in the labs group pywikibot. I've left a note to that effect on https://meta.wikimedia.org/wiki/User:Pywikibot-oauth

And if its credentials are modified, we need to set the new credentials in the Travis CI private environment variables for the repo wikimedia/pywikibot-core, but other than that nothing breaks (and those tests have been failing for quite a while, so no harm done by breaking them doubly).

jayvdb added projects: Pywikibot-OAuth, Pywikibot-tests.Mar 26 2016, 3:52 PM

Restricted Application added a subscriber: pywikibot-bugs-list. · View Herald TranscriptMar 26 2016, 3:52 PM

In T130611#2153076, @jayvdb wrote:

Is this the cause of T129763: beta wikipedia test builds fail ?

Yes. The 'lots of 503 errors' thing is the original bug this task was intended to fix (WARNING: Http response status 503 WARNING: Non-JSON response received from server wpbeta:en; the server may be down - a fatal error in the OAuth JSON response, example); the signature errors (WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Invalid signature - example) are caused by the secret key change.

jayvdb mentioned this in T129763: beta wikipedia test builds fail.Mar 27 2016, 2:11 AM

Tgr mentioned this in T130896: Getting logged out frequently in Beta Cluster.Mar 28 2016, 4:01 PM

@Tgr Thanks, have updated the secret key and I was successfully able to call the API via Oauth! Seems all fixed now!

For others in the future: I updated the key by clicking on the manage link on my relevant registration Special:OAuthConsumerRegistration/list and checking "Reset the secret key to a new value" checkbox in the "Update OAuth consumer application" form, and giving a reason in the text box.

Many thanks!

I've also updated the credentials of Pywikibot-oauth successfully. Is there anything else to be done for this task?

I don't think so.

Tgr closed this task as Resolved.May 18 2016, 5:55 PM

Tgr claimed this task.

Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:39 PM

Restricted Application added a project: Pywikibot. · View Herald TranscriptOct 16 2020, 5:39 PM

500 status responses and fatal errors for OAuth authenticated API requests on beta servers Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

500 status responses and fatal errors for OAuth authenticated API requests on beta servers
Closed, ResolvedPublic
Actions