Page MenuHomePhabricator

Certain users are unable to log into their account (HTTP 503 upon login attempt)
Closed, ResolvedPublic

Description

Author: sushant_savla

Description:
Screen short showing error


Users unable to login in most cases, getting error 503 from varnish in return of the login form. If an incorrect password is entered, the form gracefully reloads. The issue seems quite widespread but is hard to reproduce: at least 5 users across 4 wikis.

For each affected user, the issue is often user account- and/or wiki-specific: sometimes a user is unable to access an increasingly large number of wikis; sometimes bypassing caches helps. No other pattern was detected.

Asked for some multi-week grep -c for action=submitlogin&type=login 503 errors in logs, requires shell user.
Some example timestamps: 15 Nov 2014 11:53:10 GMT, 08 Dec 2014 12:32:30 GMT, 08 Dec 2014 12:59:58 GMT, 10 Dec 2014 15:39:46 GMT, 11 Dec 2014 13:18:26 GMT.


When i try login in throogh Mozilla Fire fox i get error message sinc last 4 -5 days.

screen shot of error message enclosed/

Error reads

Wikimedia Foundation

Error

Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.
If you report this error to the Wikimedia System Administrators, please include the details below.
Request: POST http://gu.wikipedia.org/w/index.php?title=%E0%AA%B5%E0%AA%BF%E0%AA%B6%E0%AB%87%E0%AA%B7:%E0%AA%B8%E0%AA%AD%E0%AB%8D%E0%AA%AF%E0%AA%AA%E0%AB%8D%E0%AA%B0%E0%AA%B5%E0%AB%87%E0%AA%B6&action=submitlogin&type=login&returnto=%E0%AA%AE%E0%AB%81%E0%AA%96%E0%AA%AA%E0%AB%83%E0%AA%B7%E0%AB%8D%E0%AA%A0, from 10.64.0.103 via cp1055 cp1055 ([10.64.32.107]:3128), Varnish XID 3251470154
Forwarded for: 120.60.225.75, 208.80.154.77, 10.64.0.103
Error: 503, Service Unavailable at Sat, 15 Nov 2014 11:53:10 GMT

Same problem is not there in IE or Opera.


Attached:

Details

Reference
bz73462

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Glaisher triaged this task as High priority.EditedNov 24 2014, 4:59 PM
Glaisher added a subscriber: Glaisher.

Another user has also reported such an error here while trying to log in at io.wiktionary.org

Is this still an issue?

Stryn added a subscriber: Stryn.EditedNov 24 2014, 8:42 PM

I don't know if that's the same problem, but on 19th of November one user on the Finnish Wikipedia reported that he couldn't log in by using Firefox. When he tried to login he was redirected to Wikimedia Foundation page.

He managed to log in by using another browser and he was able to log in to another Wikipedia by using Firefox. I told him to clean up his cache in Firefox. He told that he cleaned his cache and rebooted his computer but it didn't help. Then he updated some FF plugins, used CCleaner to clean up much more of things and then deleted all FF history and voilà, he was again able to log in on Firefox.

Glaisher set Security to None.
Zache added a subscriber: Zache.EditedDec 8 2014, 3:57 PM

This comment is follow up comment to Stryn comment. Paju who was original reporter did give more detailed error report (copy pasted in below) and Velma confirmed that there has been problems whole week. Information from Paju and Velma translated to english ( 1, 2):

  • Paju: Problem seems to be account related. Problem persist when he is trying to log in with different computer, ip or browser with user "Paju" to fiwiki. At same time login with other accounts it working. Paju doesn't use unified login so he wasn't able to test if login via other wikipedias is working. Changing password doesn't fix the problem. Also it is also tested that using Firefox "private window tab" doesn't help. If password is changed via reset password link then login will work once, but then fail again.
  • Velma: Problems to login with user name "Velma" has been ongoing whole week and she confirmed that problem persist when different computers and browsers are used. Problem also exists when svwiki, commons or wikidata is used. Alternative account seems to be working.

Example error messsage 1 provided by Paju ; browser 1

Wikimedia Foundation
Error
Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.
If you report this error to the Wikimedia System Administrators, please include the details below.
Request: POST http://fi.wikipedia.org/w/index.php?title=Toiminnot:Kirjaudu_sis%C3%A4%C3%A4n&action=submitlogin&type=login&returnto=Nukketeatteri+Sampo&returntoquery=oldid%3D2037078, from 10.20.0.138 via cp1053 cp1053 ([10.64.32.105]:3128), Varnish XID 2354709726
Forwarded for: IP-REMOVED-BY-PAJU, 10.20.0.140, 10.20.0.140, 10.20.0.138
Error: 503, Service Unavailable at Mon, 08 Dec 2014 12:32:30 GMT

Example error messsage 2 provided by Paju, different browser, 20 minutes after the password change.

Wikimedia Foundation
Error
Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.
If you report this error to the Wikimedia System Administrators, please include the details below.
Request: POST http://fi.wikipedia.org/w/index.php?title=Toiminnot:Kirjaudu_sis%C3%A4%C3%A4n&action=submitlogin&type=login&returnto=Wikipedia:Etusivu, :from 91.198.174.72 via cp1068 cp1068 ([10.64.0.105]:3128), Varnish XID 523992095
Forwarded for: IP-REMOVED-BY-PAJU, 10.20.0.140, 10.20.0.140, 91.198.174.72
Error: 503, Service Unavailable at Mon, 08 Dec 2014 12:59:58 GMT
TTO renamed this task from 503 error in Firefox trying to log into gu.wp to Certain users are unable to log into their account (HTTP 503 upon login attempt).Dec 10 2014, 6:20 AM
TTO updated the task description. (Show Details)
TTO added a subscriber: TTO.
Nemo_bis updated the task description. (Show Details)Dec 10 2014, 6:41 AM
Nemo_bis added a subscriber: Nemo_bis.
Nemo_bis updated the task description. (Show Details)Dec 10 2014, 8:24 AM
Nemo_bis added a project: Shell.
Zache updated the task description. (Show Details)Dec 10 2014, 8:40 AM
Zache added a comment.Dec 10 2014, 8:43 AM

Edited the description because problem seems to be user account specific. Person can fail with one user_name with error 503 but successfully login with another with same computer.

MBH added a subscriber: MBH.Dec 10 2014, 6:53 PM
hoo added a subscriber: hoo.Dec 10 2014, 7:15 PM

Has also been reported at dewiki: https://de.wikipedia.org/w/index.php?title=Wikipedia:Fragen_zur_Wikipedia&oldid=136658267#Login_Problem

I looked into the logs quickly, but wasn't able to find anything relevant :/

Paju added a subscriber: Paju.Dec 11 2014, 2:22 PM

It should be noted that the system seems to recognize when user is using correct password (case 1) and when not (case 2). In case 1, user Paju (fi.wikipedia) is redirected to Wikimedia Foundation Error page, as shown above (or below, for 3rd time). If user Paju inserts erroneous password (case 2), system acts normally and notifies him about erroneous password and requests typing password again. So the problem seems to be somewhere after the initial user recognition.

Example of the error message 3, browser as in Example 1, IP has been renewed.

Wikimedia Foundation
Error

Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.
If you report this error to the Wikimedia System Administrators, please include the details below.

Request: POST http://fi.wikipedia.org/w/index.php?title=Toiminnot:Kirjaudu_sis%C3%A4%C3%A4n&action=submitlogin&type=login&returnto=Minsk, from 91.198.174.62 via cp1067 cp1067 ([10.64.0.104]:3128), Varnish XID 1668654212
Forwarded for: IP-REMOVED-BY-PAJU, 91.198.174.71, 91.198.174.71, 91.198.174.62
Error: 503, Service Unavailable at Thu, 11 Dec 2014 13:18:26 GMT

Here the IP number 91.198.174.xx belongs to Wikimedia Foundation Inc. (Netherlands) and the IP number 10.64.0.yy to IANA (United States - Internet Assigned Numbers Authority).

Zache added a comment.EditedDec 11 2014, 4:30 PM

Hmm, can somebody push this to the higher priority of Foundation tech guys? Most likely this is a bug in server side so user's cannot do anything without access to server logs. However currently this is also repeatable so somebody should least take debug information from server side to the ticket.

Stryn added a comment.Dec 11 2014, 4:56 PM

Should be noted that all of those users who have had problems with login seems to be without a SUL account.
User:Artomo who is one of the users who had problems made a SUL account (using Special:MergeAccount) on 27 November 2014.
User:Velma who is one of the users who had problems made a SUL account (using Special:MergeAccount) on 10 December 2014. And AFAIK Velma is now able to use Wikipedia normally.

awight removed a subscriber: awight.Dec 11 2014, 5:02 PM
Nemo_bis updated the task description. (Show Details)Dec 11 2014, 5:02 PM
Nemo_bis added a subscriber: awight.
Nemo_bis removed a subscriber: awight.

@Joe and I looked into this just now and didn't see anything helpful in the logs (we can see the errors though). Would any of you who have the error be able to join #wikimedia-tech so we can watch your login process as it happens?

So in this case it's a non-global account: https://meta.wikimedia.org/wiki/Special:CentralAuth/Paju

Do we know when this user was last able to login? I'm guessing this is specific to wmf11, since that just hit the wikipedias yesterday?

Nemo_bis added a comment.EditedDec 11 2014, 5:20 PM

If Stryn's intuition is right:

  • grepping centralauth.log for "CentralAuth.*Migrat" might bring up something interesting (although it would be very weird for this to be browser-specific);
  • someone should merge Paju's account, if that's not enough we'd know to look somewhere else.

Do we know when this user was last able to login? I'm guessing this is specific to wmf11, since that just hit the wikipedias yesterday?

No, this has been going on for weeks.

Paju added a comment.Dec 11 2014, 5:30 PM

No merging of my Paju account, please. There are at least two different persons using username Paju in wiki projects. I object merging.

Nemo_bis updated the task description. (Show Details)Dec 11 2014, 5:32 PM
Paju added a comment.Dec 11 2014, 5:43 PM

User Paju is able to login to fi.wikipedia any time, but only through password change (new temporal password requested & this is inserted & password is changed when login). Relogin with new password fails. However some front line authentication system (by Wikimedia Foundation?) seems to recognize when correct password is inserted (redirect to Wikimedia Foundation Error -message) and when incorrect password is given (normal "Login error. Incorrect password entered. Please try again." failure message due to incorrect passwd occurs in Finnish). User Paju does not have problem in login with sv.wikipedia, en.wikipedia, commons, etc, i.e. problem occurs only with fi.wikipedia.

User Paju does not have problem in login with sv.wikipedia, en.wikipedia, commons, etc, i.e. problem occurs only with fi.wikipedia.

This makes one think the error might be happening between line 940 and line 955 of CentralAuthUser.php...

User Paju does not have problem in login with sv.wikipedia, en.wikipedia, commons, etc, i.e. problem occurs only with fi.wikipedia.

This makes one think the error might be happening between line 940 and line 955 of CentralAuthUser.php...

@Legoktm found log entries from line 954 in all the cases specified, so we hit that, and return false to line 80 of CentralAuthPlugin.php.

And since bad passwords are correctly handled, we likely get back to User.php line 3835. And the hash is current, so we don't need to update the password, so User::checkPassword should return true.

Since temporary passwords seem to work, but not the set password, then the failure is probably in SpecialUserlogin between lines 790-807.

It seems that the login only fails on that wiki, which is the best candidat for the "home wiki" when a global account would be created.

Possible workaround: Use another wiki to visit [[Special:MergeAccount]] and try to create a global account.

MBH added a comment.Dec 13 2014, 3:02 AM

It's not always possible - ru:Ibidem can't create SUL due to username collision of different users.

It seems that the login only fails on that wiki, which is the best candidat for the "home wiki" when a global account would be created.

If the concept of "home wiki" is unknown (due to T16234):
Is seems that the login fails on that wiki, which will be prefered for the global account, if there would be a user collision which can not be resolved by merging with the same email or by password.

User Pieroweert added tekst to T8450 after it was closed as duplicate. I'm copying his text here:
Have the same problem with my original account (Gebruiker:Piero) on nl.wikipedia. No problem with computer or IP-adress. Made a new account (Gebruiker:Pieroweert) which works perfectly. But I have no access to my original lists. Can not log in on Gebruiker:Piero, can log in on Gebruiker:Pieroweert

The same problem with my original account (Scriber) on ru.wikipedia. No problem with computer or IP-adress. I have no access to my original account. But I can log in on "Scriber en". What am I to do?

I have no access to my original account. But I can log in on "Scriber en". What am I to do?

Try to make a new account as "Scriber" and then merge them.

I can't make a new account as "Scriber" and then merge it with "Scriber en" because wikipedia replies "Account creation error. Username entered already in use. Please choose a different name" (account "Scriber" already exists in en.wikipedia).

I also can't make a new account as "Scriber" at meta.wikimedia which is replies: "The name "Scriber" is too similar to tethe following username: "Scrib3r" . Please choose another name". - So, is "too similar to the username: Scrib3r" - the real reason for refuse of making this new account?

I have no access to my original account. But I can log in on "Scriber en". What am I to do?

Try to make a new account as "Scriber" and then merge them.

For Scriber we need someone who would be able to create account to him, ignoring existing usernames, and send password via email.
I could have done it technically in ru.wiki, but the nickname is already occupied here...

I've already done that yesterday.

Rubin16 added a comment.EditedDec 15 2014, 4:45 PM

Seems that we succeed in Scriber situation: I created him an account "Scriber" in one of WMF wikis, where he never logged before and where I can push "ignore similar existing names" option and sent autogenerated password to him.

Scriber set a new password, was able to login and created SUL from there - now he can login in ru.wiki, too.

I'll try to do it for Ibidem too

Same for Mirzali on Zazaki Wikipedia, see last task merged into this one.

I tried to reproduce this by creating a new user, deleting the global account, adding another local account with a different password. Then logged into the winning account. Everything works fine :(.

@Joe said ops could try to debug this if they can coordinate with someone who's account is affected.

User:Pirags (also hasn't global account) also has the problem
https://lv.wikipedia.org/wiki/Vikip%C4%93dija:Kopienas_port%C4%81ls#Piesl.C4.93g.C5.A1an.C4.81s_probl.C4.93ma_Vikip.C4.93dijai

Told him, that he should try Special:MergeAccounts. Will report later, if some help is needed.

Guys, can't this be fixed?

Paju added a comment.Dec 16 2014, 10:10 PM
  1. I wonder if something has been done to fix the problem "...the failure is probably in SpecialUserlogin between lines 790-807." reported by User:Legoktm earlier?
  1. Also, is there some "login unification" (?) policy preventing users to create new account say in French wikipedia, if the same account is used in some other wikipedia? I.e. I have just tried to create new account User:Paju in fr.wikipedia and uk.wikipedia, but that is not allowed. My intention was to try to create and delete global account for user Paju, based from one of these new accounts, and see if this helps fi.wikipedia user:Paju to relogin. During last 12...13 days relogin has not been possible, causing edition activity to drop 90% or more. Making the participation more difficult (this software bug does that) evicts volunteer users away from wikipedia.

The error message was similar as reported by User:Scriber_en earlier, in French: ''Erreur lors de la création du compte. Le nom « Paju » est trop similaire à l’utilisateur suivant : Paju. Veuillez choisir un autre nom.'' (Error creating account. The name "Paju" is too similar to the following user: Paju. Please choose another name.)

So far, there is no User:Paju in French nor in Ukranian wikipedia. In other projects, there are already more than one users (persons) with Paju account. I would not mind if there were more users with this username in the future. As an example, somebody from the city of Paju in South Korea, could create username Paju to be used in ko.wikipedia. The whole "login unification" approach, forcing users to "put all the eggs in one basket" appears to me as an oligarchic/ maintenance driven approach. I found this approach also risky (dangerous) for writers from/or writing about totalitarian countries. Already now network activities are heavily censored there. Writing to non-local wiki allows expressing issues which might be risky to write in "local" wiki project. If "login unification" is a voluntary approach just for those users who wish to protect their username forever in the whole universe, it might be ok in some limited cases. But trying to force all users to give up or take over their username identity globally, especially if they have no intention to do so or if there are several users with the same username, is absolutely not an acceptable approach.

Change 180346 had a related patch set uploaded (by Legoktm):
Temporarily disable $wgCentralAuthAutoMigrate

https://gerrit.wikimedia.org/r/180346

Patch-For-Review

Change 180346 merged by jenkins-bot:
Temporarily disable $wgCentralAuthAutoMigrate

https://gerrit.wikimedia.org/r/180346

Legoktm added a subscriber: ori.Dec 16 2014, 11:43 PM

@ori noticed that there were OOM error mesages corresponding with the same timestamp as the centralauth debug log entries. After disabling $wgCentralAuthAutoMigrate, MG1 reported on IRC that they were able to login successfully.

Still unsure why the automigration process was causing OOMs...

Keegan added a subscriber: Keegan.Dec 16 2014, 11:51 PM
Savh added a subscriber: Savh.Dec 16 2014, 11:53 PM

If another user who was experiencing the 503s can confirm they can login now, I'll go ahead and close this. And filed T78727: Investigate and fix OOMs caused during account globalization for followup.

TTO closed this task as Resolved.Dec 17 2014, 12:03 AM
TTO claimed this task.

From enwiki VPT:

OK, I was just able to log on. If someone implemented one of the older bug patches mentioned most recently in the ticket, it worked. Daniel Case (talk) 23:42, 16 December 2014 (UTC)

TTO removed TTO as the assignee of this task.Dec 17 2014, 12:04 AM
Paju added a comment.Dec 17 2014, 11:28 AM

Also I confirm that for first time since 4th Dec. 2014, I have been able to login and relogin to fi.wikipedia with existing password normally. I validated this during last 30 min with IE, Firefox and Opera browsers with laptop and/or tablet (i.e. in two different operating systems). So, the problem seems to be solved. Thanks a lot!

Elitre added a subscriber: Elitre.Dec 17 2014, 3:37 PM
bd808 moved this task from Done to Archive on the MediaWiki-Core-Team board.Dec 22 2014, 10:39 PM

There are recent similar reports from today in {T87300} and T87305: Users are unable to change account password (HTTP 503). Sure this is really fixed? Could someone please take a look at those?