Page MenuHomePhabricator

Attempt to login into Phabricator fails with a 503 Service Unavailable error
Closed, ResolvedPublic

Description

How to reproduce:

  1. Log out of WM Phab and your SUL WM account.
  2. Navigate to https://phabricator.wikimedia.org
  3. Press the button "Login or Register (MediaWiki)"

Result:
After about 15s, an error page with the following contents appears:

WIKIMEDIA FOUNDATION
Error
Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.
If you report this error to the Wikimedia System Administrators, please include the details below.
Request: POST http://phabricator.wikimedia.org/auth/login/mediawiki:mediawiki/, from 10.64.0.172 via cp1044 cp1044 ([10.64.0.172]:80), Varnish XID 1595403262
Forwarded for: <redacted>, 10.64.0.172
Error: 503, Service Unavailable at Thu, 11 Dec 2014 01:11:25 GMT

Expected: Request to log me in into MediaWiki.org

Event Timeline

Rillke raised the priority of this task from to Needs Triage.
Rillke updated the task description. (Show Details)
Rillke added projects: Phabricator, acl*sre-team.
Rillke changed Security from none to None.
Rillke subscribed.
Qgil triaged this task as Unbreak Now! priority.Dec 11 2014, 8:07 AM
Qgil subscribed.

Hm, same here:

Wikimedia Foundation
Error

Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.
If you report this error to the Wikimedia System Administrators, please include the details below.
Request: POST http://phabricator.wikimedia.org/auth/login/mediawiki:mediawiki/, from 10.64.0.171 via cp1043 cp1043 ([10.64.0.171]:80), Varnish XID 1226470680
Forwarded for: phabupdate, 2003:5b:ee1e:af64:c8f1:9342:dc2d:faec, 10.64.0.171
Error: 503, Service Unavailable at Thu, 11 Dec 2014 08:05:55 GMT

If this is happening for everybody, then this means that users without LDAP ending their session will not be able to log in again. Sound like trouble is coming unless this is fixed soon. Setting to Unbreak Now!

the error is with mediawiki, not phab:

On https://www.mediawiki.org/w/index.php?title=Special:OAuth/ I see this error:

A database query error has occurred. This may indicate a bug in the software.
Function: MediaWiki\Extensions\OAuth\MWOAuthConsumer::newFromKey
Error: 1054 Unknown column 'oarc_callback_is_prefix' in 'field list'

@Qgil: did you mean to add a subscriber or add a project tag?

I didn't even know you could subscribe a project...

Add a subscriber, just to CC whoever is following that project (not sure whether it works). :) I don't think there is a problem in the extension itself, and I still think that it is suspicious that this happens right after our Phabricator upgrade.

Probably the database wasn't updated after https://gerrit.wikimedia.org/r/#/c/153983/ was merged.

informed ops and our dba about the query error, thanks Nemo for pointing out the process for schema changes already

afaict either all the .sql files from https://gerrit.wikimedia.org/r/#/c/153983/9 need to be run by DBA (springle) or the change should be temp. reverted until we have the schema change via proper process

DBA now looking at tables...

or https://gerrit.wikimedia.org/r/#/c/179130/

we are getting more reports on IRC

Sean applied a hotfix. the error appears to be gone for me. Could you guys please try again and confirm?

informed ops and our dba about the query error,

@Dzahn Reported this this morning in #wikimedia-operations but no one cared. So whom should I ping next time?

mmodell claimed this task.

The DDL in backend/schema/mysql/callback_is_prefix.sql has been applied to mediawiki.org schema, but nothing more.

Wondering why https://gerrit.wikimedia.org/r/#/c/153983/ only includes oarc_callback_is_prefix field in the patch, and not the base OAuth schema? Would new installs break too?

Was this now deployed or reverted back? Was schema applied?