Attempt to login into Phabricator fails with a 503 Service Unavailable error
Closed, ResolvedPublic

Description

How to reproduce:

  1. Log out of WM Phab and your SUL WM account.
  2. Navigate to https://phabricator.wikimedia.org
  3. Press the button "Login or Register (MediaWiki)"

Result:
After about 15s, an error page with the following contents appears:

WIKIMEDIA FOUNDATION
Error
Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.
If you report this error to the Wikimedia System Administrators, please include the details below.
Request: POST http://phabricator.wikimedia.org/auth/login/mediawiki:mediawiki/, from 10.64.0.172 via cp1044 cp1044 ([10.64.0.172]:80), Varnish XID 1595403262
Forwarded for: <redacted>, 10.64.0.172
Error: 503, Service Unavailable at Thu, 11 Dec 2014 01:11:25 GMT

Expected: Request to log me in into MediaWiki.org

Rillke created this task.Dec 11 2014, 1:13 AM
Rillke updated the task description. (Show Details)
Rillke raised the priority of this task from to Needs Triage.
Rillke changed Security from none to None.
Rillke added a subscriber: Rillke.
chasemp added a subscriber: chasemp.
Qgil triaged this task as Unbreak Now! priority.Dec 11 2014, 8:07 AM
Qgil added a subscriber: Qgil.

Hm, same here:

Wikimedia Foundation
Error

Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.
If you report this error to the Wikimedia System Administrators, please include the details below.
Request: POST http://phabricator.wikimedia.org/auth/login/mediawiki:mediawiki/, from 10.64.0.171 via cp1043 cp1043 ([10.64.0.171]:80), Varnish XID 1226470680
Forwarded for: phabupdate, 2003:5b:ee1e:af64:c8f1:9342:dc2d:faec, 10.64.0.171
Error: 503, Service Unavailable at Thu, 11 Dec 2014 08:05:55 GMT

If this is happening for everybody, then this means that users without LDAP ending their session will not be able to log in again. Sound like trouble is coming unless this is fixed soon. Setting to Unbreak Now!

the error is with mediawiki, not phab:

On https://www.mediawiki.org/w/index.php?title=Special:OAuth/ I see this error:

A database query error has occurred. This may indicate a bug in the software.
Function: MediaWiki\Extensions\OAuth\MWOAuthConsumer::newFromKey
Error: 1054 Unknown column 'oarc_callback_is_prefix' in 'field list'

Nemo_bis added a subscriber: TTO.Dec 11 2014, 8:34 AM

@Qgil: did you mean to add a subscriber or add a project tag?

I didn't even know you could subscribe a project...

Qgil added a comment.Dec 11 2014, 9:36 AM

Add a subscriber, just to CC whoever is following that project (not sure whether it works). :) I don't think there is a problem in the extension itself, and I still think that it is suspicious that this happens right after our Phabricator upgrade.

TTO added a comment.Dec 11 2014, 9:45 AM

Probably the database wasn't updated after https://gerrit.wikimedia.org/r/#/c/153983/ was merged.

Nemo_bis added a subscriber: Nemo_bis.
In T78223#841048, @TTO wrote:

Probably the database wasn't updated after https://gerrit.wikimedia.org/r/#/c/153983/ was merged.

As a reminder, the process is https://wikitech.wikimedia.org/wiki/Schema_changes

Rillke updated the task description. (Show Details)Dec 11 2014, 1:21 PM
Dzahn added a subscriber: Dzahn.Dec 11 2014, 1:29 PM

informed ops and our dba about the query error, thanks Nemo for pointing out the process for schema changes already

Dzahn added a comment.Dec 11 2014, 1:37 PM

afaict either all the .sql files from https://gerrit.wikimedia.org/r/#/c/153983/9 need to be run by DBA (springle) or the change should be temp. reverted until we have the schema change via proper process

Aklapper added a subscriber: greg.Dec 11 2014, 1:45 PM

Potentially the same issues: T78209, T78247, T78223

Dzahn added a comment.Dec 11 2014, 1:48 PM

DBA now looking at tables...

or https://gerrit.wikimedia.org/r/#/c/179130/

we are getting more reports on IRC

Dzahn added a comment.Dec 11 2014, 1:51 PM

Sean applied a hotfix. the error appears to be gone for me. Could you guys please try again and confirm?

informed ops and our dba about the query error,

@Dzahn Reported this this morning in #wikimedia-operations but no one cared. So whom should I ping next time?

@Dzahn: I can confirm that it's fixed.

mmodell closed this task as Resolved.Dec 11 2014, 1:54 PM
mmodell claimed this task.

The DDL in backend/schema/mysql/callback_is_prefix.sql has been applied to mediawiki.org schema, but nothing more.

Wondering why https://gerrit.wikimedia.org/r/#/c/153983/ only includes oarc_callback_is_prefix field in the patch, and not the base OAuth schema? Would new installs break too?

Mitar added a subscriber: Mitar.Dec 12 2014, 11:24 PM
Mitar removed a subscriber: Mitar.
Mitar added a subscriber: Mitar.
Mitar added a comment.Dec 15 2014, 1:21 PM

Was this now deployed or reverted back? Was schema applied?

Was schema applied?

Yes.