Adding CSteipp to CC list.

(In reply to comment #5)

I'd like to post that this bug is possibly causing other issues such as global
accounts not possible to be deleted as in
https://meta.wikimedia.org/w/index.php?title=Steward_requests/SUL_requests&oldid=4102655#Chhagan_medatiya.

Regards.

This seems to be investigated at bug 35792

(In reply to comment #6)

Adding CSteipp to CC list.

(In reply to comment #5)

I'd like to post that this bug is possibly causing other issues such as global
accounts not possible to be deleted as in
https://meta.wikimedia.org/w/index.php?title=Steward_requests/SUL_requests&oldid=4102655#Chhagan_medatiya.

Regards.

This seems to be investigated at bug 35792

I don't think it's the same as 35792, there you'd have empty global accounts that disappear after a bit.

Bumping up priority, we're getting more instances and users experiencing this issue. It really needs to be looked at.

Snowolf, can you link to where you're seeing issues? I've been looking at this on and off, but have not been able to catch anything in the logs. Recent examples might give us some more details.

Also, does anyone know if the code for sulinfo is available anywhere? The difference that Félix saw between the lists may give a clue about what's happening.

Seems to be: https://svn.toolserver.org/svnroot/quentinv57/public_html/tools/sulinfo.php

(In reply to comment #9)

Snowolf, can you link to where you're seeing issues? I've been looking at this
on and off, but have not been able to catch anything in the logs. Recent
examples might give us some more details.

Also, does anyone know if the code for sulinfo is available anywhere? The
difference that Félix saw between the lists may give a clue about what's
happening.

I think Snowolf meant this recent example: https://meta.wikimedia.org/w/index.php?title=Steward_requests/SUL_requests&oldid=4318719#PiRSquared17

(In reply to comment #11)

(In reply to comment #9)

Snowolf, can you link to where you're seeing issues? I've been looking at this
on and off, but have not been able to catch anything in the logs. Recent
examples might give us some more details.

Also, does anyone know if the code for sulinfo is available anywhere? The
difference that Félix saw between the lists may give a clue about what's
happening.

I think Snowolf meant this recent example:
https://meta.wikimedia.org/w/index.php?title=Steward_requests/SUL_requests&oldid=4318719#PiRSquared17

Not sure about more recent instances, as I am clueless as to at which point the account becomes unattached w/ different password; there's more instances that recently have come to light, such as https://gu.wikisource.org/w/index.php?title=%E0%AA%B5%E0%AA%BF%E0%AA%B6%E0%AB%87%E0%AA%B7%3A%E0%AA%B2%E0%AB%89%E0%AA%97&type=&user=PiRSquared17+2&page=&year=&month=-1 which Trijnstel pointed to (the account has been renamed to that username now); however, Restu20 has an account created 4 days ago that should be part of this same bug: https://toolserver.org/~quentinv57/tools/sulinfo.php?username=Restu20&showinactivity=1&showblocks=1&showlocked=1

I have added Quentinv57, the developer of Sulinfo, to the CC for this bug given the issues you have raised.

Chris, in the meanwhile, would it be possible for you (ops) to manually merge again some accounts? See bug 39997 for example (closed but not acted upon it). Thanks.

Bumping this up. Loss of account access seems like a fairly serious issue to me, particularly for steward accounts.

It happened to global user Suprememangaka, who has lost access to his wikidata account.

Felix, thanks for the report. Sprememangaka did not show up in the dberrors, exception, or fatal logs for Oct 30 - Nov 1. I also noticed that wikidata is on the sulinfo listing, but not on the CentralAuth listing.

(In reply to comment #16)

[...] I also noticed that wikidata is on
the sulinfo listing, but not on the CentralAuth listing.

True: CentralAuth only lists unified accounts, while sulinfo also shows the unattached accounts in a separate table.

Thanks.

(In reply to comment #16)

Felix, thanks for the report. Sprememangaka did not show up in the dberrors,
exception, or fatal logs for Oct 30 - Nov 1. I also noticed that wikidata is on
the sulinfo listing, but not on the CentralAuth listing.

I believe it is relying on the toolserver's db listing all wikis, which has not been updated yet.

(In reply to comment #18)

I believe it is relying on the toolserver's db listing all wikis, which has not
been updated yet.

Not sure if it's toolserver db only. I'm still unable to log in on some wikis (cfr bug 39997). Thanks.

(In reply to comment #19)

(In reply to comment #18)

I believe it is relying on the toolserver's db listing all wikis, which has not
been updated yet.

Not sure if it's toolserver db only. I'm still unable to log in on some wikis
(cfr bug 39997). Thanks.

No no I mean the fact that wikidata doesn't show up yet is because of that.

Reading this I'm kind of lost so pardon my potentially weird questions:

Is the assumption that this happens in sulinfo because there is no update yet to the Toolserver's database, or does this only refer to investigating the reasons for the reported problem?

Quoting Nemo_bis: "See also bug 39060, bug 39996, bug 35792 and bug 39997."

(In reply to comment #21)

Reading this I'm kind of lost so pardon my potentially weird questions:

Is the assumption that this happens in sulinfo because there is no update yet
to the Toolserver's database, or does this only refer to investigating the
reasons for the reported problem?

Quoting Nemo_bis: "See also bug 39060, bug 39996, bug 35792 and bug 39997."

Sulinfo has nothing to do with the problem.

There's an unrelated gerrit change 36683 prompted by bug 42614 which might affect this, removing some warnings (there's plenty, apparently) and "Create a CentralAuthUser object for a user who is known to be unattached".
Do we now have a worse or better way to track actual SUL failures?

Chris: Do you know if there is any more info needed to hunt down the reason for the problems here?

The bug is not reproducible, so whenever we put in a fix for CentralAuth / autocreate, I try to keep track to see if this is still showing up.

As Nemo pointed out, Tim made a change last week that may have helped (although the fix was for a bug that resulted in a database error, which has not been the case for any of the other users reported on this ticket). So I think at this point I'm waiting to see if it's still happening on 1.21wmf6 instances.

Adding any reported cases of users affected by this would also help.

Thanks! So for the time being we are waiting. Decreasing severity by one level.

(In reply to comment #25)

The bug is not reproducible,

[...]

So I think at this
point I'm waiting to see if it's still happening on 1.21wmf6 instances.

Could anybody of the affected users please state here if this problem still exists?

(In reply to comment #27)

Could anybody of the affected users please state here if this problem still
exists?

Unattached accounts never fix themselves alone, they stay unattached forever.
As for new instances of the problem, someone should run a query on the DB to find new unattached accounts which share username with a global account; except or rogue bureaucrats, that will be the list of instances of this bug.

(In reply to comment #27)

Could anybody of the affected users please state here if this problem still
exists?

Yes Andre. The issue still happens, and I've stated in bug 39997 (which is *not* solved) 10 local accounts of mine are were detached from the global in spite of being clearly mine because the logs shows they performed steward actions.

As Nemo said above, somebody with shell/root access should manually merge those again.

Thanks.

Actually I said only that someone with shell access (or maybe just TS) can and should run a query to *list* them. There can be false positives, so I'm not sure it can be done with a maintenance script.

For those who do not have more than one "dead" local account, they can have it renamed locally so they can let the SUL create it again for them.

(In reply to comment #15)

It happened to global user Suprememangaka, who has lost access to his
wikidata
account.

Another wikivoyage dead local account (registered on October 29th. Suprememangaka had registered on the 31st): https://toolserver.org/~quentinv57/sulinfo/Mentifisto

Added some comments to bug 39997 that may be relevant.

This happened to me too.

https://toolserver.org/~quentinv57/sulinfo/Liangent
https://oc.wikipedia.org/wiki/Especial:Jornal/Liangent?uselang=en

I'm not eager to get my account back though...

Possibly another example, with a steward account:

https://meta.wikimedia.org/w/index.php?title=Steward_requests/SUL_requests&oldid=5321968#QuiteUnusual_Problem

neil wrote:

I'm the one with the "quite unusual problem", in comment 34. If it helps, there was one difference in what I was doing on the wikis where my account became detached and the ones that didn't. For the first renames I did, I went to the project, edited my preferences to change the language to en, then went to RENAMEUSER and completed the renames. For the ones that broke, I used a direct link over https rather than navigate round the Wiki. For example, I used https://example.wikipedia.org/w/index.php?title=Special:RenameUser&oldusername=AnOldName&newusername=ANewName&reason=why+not

I've no idea if this is the cause, but maybe the local account creation process isn't being invoked or completed properly if you enter the project via this route rather than the normal route a user would take.

neil wrote:

Per the above, I tried it again as a test. Using this URL https://yo.wikipedia.org/w/index.php?title=Special:RenameUser&oldusername=Rafy&newusername=Kathovo&reason=Reason

and the result was that the local account has been created, but detached from the SUL account. Hopefully this will give you enough to track it down.

Confirmed! Thanks, QU! I tried using my test account [[Special:CentralAuth/Pirsquared17]] and clicking on the yo.wikipedia link in comment 36. When I looked at CentralAuth, I noticed the yowiki account wasn't attached. After logging out, I cannot log back in. Here's the log: https://yo.wikipedia.org/wiki/Pàtàkì:Log/Pirsquared17

(In reply to comment #35)

I'm the one with the "quite unusual problem", in comment 34. If it helps,
there
was one difference in what I was doing on the wikis where my account became
detached and the ones that didn't. For the first renames I did, I went to the
project, edited my preferences to change the language to en, then went to
RENAMEUSER and completed the renames. For the ones that broke, I used a
direct
link over https rather than navigate round the Wiki. For example, I used
https://example.wikipedia.org/w/index.php?title=Special:
RenameUser&oldusername=AnOldName&newusername=ANewName&reason=why+not

I've no idea if this is the cause, but maybe the local account creation
process
isn't being invoked or completed properly if you enter the project via this
route rather than the normal route a user would take.

If it helps, I was also renaming accounts when I faced the same problem. Bug 39997 has more info, specially the sixth comment.

I just noticed that my account (User:Matma Rex) is unattached at sco.wikipedia (which I didn't even edit). There were no renames involved.

Related URL: https://gerrit.wikimedia.org/r/57671 (Gerrit Change Ie87319134d92fe6ecb9cf46cf3aba408724bd361)

I sanitized the automatic user creation in https://gerrit.wikimedia.org/r/57671 and I hope that this silently kills the problem.

neil wrote:

Tested using the same scenario as I noted above. Worked successfully (i.e., the account was created and attached correctly) - thanks.

Seems fixed.

(In reply to comment #43)

Seems fixed.

Seems to early to say, we don't have a reliable way to reproduce the problem do we.

(In reply to comment #44)

(In reply to comment #43)

Seems fixed.

Seems to early to say, we don't have a reliable way to reproduce the problem
do
we.

Indeed, there was no fix (merged) yet and we don't have long term data either.

Indeed, it's too early to tell: User:Mentifisto is having the same issue again (he'd got his account renamed on wikidata, and his new account still didn't merge).

• http://www.wikidata.org/w/index.php?title=Special%3ALog&type=renameuser&user=&page=User%3AMentifisto&year=&month=-1&tagfilter=
• http://www.wikidata.org/wiki/Special:Log/Mentifisto

Another?

https://meta.wikimedia.org/wiki/Steward_requests/SUL_requests#wikinews_logins_not_working

(In reply to comment #44)

(In reply to comment #43)

Seems fixed.

Seems to early to say, we don't have a reliable way to reproduce the problem
do
we.

Actually we do. See comment 36 and comment 37. I was able to recreate this problem with User:Pirsquared17@zuwiki by going to https://zu.wikipedia.org/wiki/Special:RenameUser before the account existed. Still broken.

(In reply to comment #48)

Actually we do. See comment 36 and comment 37.

Those always looked like another issue to me.

The Rename link is able to reproduce this in production, although I'm not able to reproduce the issue in a development system, so I'm not entirely sure which combination of production settings are causing the issue.

Hoo's patch (gerrit 57671) has not been merged yet. I think we can merge it today, so it will go out with 1.22wmf2. So if this rename issue is still around after April 24, we'll know that didn't fix it.

Has anyone tried to replicate the problem on beta (http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page and associated sites)? I can try that later today, but that may give us a good place to study the problem.

(In reply to comment #50)

Has anyone tried to replicate the problem on beta
(http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page and associated sites)? I
can try that later today, but that may give us a good place to study the
problem.

Maybe, or maybe not. The problem seems to be consequence of some sort of race condition on the centralauth database, and this is going to be completely different on Labs.

I was not able to reproduce on en.wikibooks.beta.wmflabs.org.

Also, I tried this again in production (using the same, unprivileged account that yielded the unattached account on yowiki). This time I logged in over http, and visited:
https://ru.wikipedia.org/w/index.php?title=Special:RenameUser&oldusername=Rafy&newusername=Kathovo&reason=Reason
http://zh.wikipedia.org/w/index.php?title=Special:RenameUser&oldusername=Rafy&newusername=Kathovo&reason=Reason

Both links caused *attached* accounts to be created on ruwiki and zhwiki.

However, there was a comment above about guwikisource, so I also tried

https://gu.wikisource.org/w/index.php?title=Special:RenameUser&oldusername=Rafy&newusername=Kathovo&reason=Reason

which produced another *unattached* account there.

So maybe the issue is specific to how those wikis are setup in production?

(In reply to comment #52)

So maybe the issue is specific to how those wikis are setup in production?

In this case my patch would probably (hopefully) still hit. That might even be a good point to start debugging (though I neither know how you debug in production nor do I have the required access)

I'm just logging in with a personal, unprivileged account, hit those urls, and then check Special:CentralAuth to see if those wikis are showing up as attached for the account (and also double checking in the db... but that has matched the web interface every time). So anyone can test, if they have an account that isn't already attached on those wikis.

(In reply to comment #49)

(In reply to comment #48)

Actually we do. See comment 36 and comment 37.

Those always looked like another issue to me.

Someone should see if it happens with other special pages too, or if it's just a RenameUser thing.

Tegel.svwp wrote:

(In reply to comment #27)

(In reply to comment #25)

The bug is not reproducible,

[...]

So I think at this
point I'm waiting to see if it's still happening on 1.21wmf6 instances.

Could anybody of the affected users please state here if this problem still
exists?

It's the same for me with my account at he.wikivoyage.
https://meta.wikimedia.org/wiki/Special:CentralAuth/Tegel

It's not possible to merge it or login locally.

Updates? Is this still occurring?

I think it could have occured here: https://meta.wikimedia.org/wiki/Special:Permalink/7517074#Ajraddatz

(In reply to MF-Warburg from comment #58)

I think it could have occured here:
https://meta.wikimedia.org/wiki/Special:Permalink/7517074#Ajraddatz

I just checked that... in a nutshell:
The account was auto created in 2011. At some point in time it probably got deattached from the global account, that's all I can say as the account rename/ recreation took place before I got notified.

As this incident happened some time back, I wouldn't consider that much of a recent issue.

Closing as WORKSFORME as there don't seem to be any recent issue (and the bug can't be confirmed anymore).
If this occurs again, please reopen.

Define recent?

quentinv57 wrote:

Does this bug include the recent created global accounts which do not exist locally on login.wikimedia.org ?

(In reply to Nemo from comment #61)

Define recent?

As there have been many changes to CentralAuth, I would say 4 months maybe... 6 months for extra sanity if you have anything at hand.

(In reply to Quentinv57 from comment #62)

Does this bug include the recent created global accounts which do not exist
locally on login.wikimedia.org ?

No, that's another issue (not sure there's a bug for that... probably that occurs if people don't follow the 302s). This one is about accounts being created with an invalid (global) state (but they at least get created).

I've checked the usernames mentioned above, they don't have new cases but they didn't see their accounts fixed either. Do you want yet another bug, in Wikimedia product, for that?
I don't think I'm going to write a query to see if and where we have unattached autocreated accounts, it would be useless anyway because I can't know if their email and password works. Will we never know?

(In reply to Nemo from comment #65)

I've checked the usernames mentioned above, they don't have new cases but
they didn't see their accounts fixed either. Do you want yet another bug, in
Wikimedia product, for that?
I don't think I'm going to write a query to see if and where we have
unattached autocreated accounts, it would be useless anyway because I can't
know if their email and password works. Will we never know?

Another bug about this would be nice... if you (or someone else) provides a query I can maybe run it in production. Depending on how big the issues are we might want to write a maintenance script for that (which also is a separate issue).

I can help with whatever is needed to clean this up.

I don't think this is solved at all, there is no shortage of recent examples. For instance, on Meta (silly query):

mysql> SELECT log_title, log_user, log_timestamp FROM logging JOIN user ON logging.log_action = 'autocreate' AND log_timestamp > 20130831000000 AND logging.log_user = user.user_id WHERE user.user_name NOT IN ( SELECT lu_name FROM centralauth_p.localuser lu WHERE lu.lu_wiki = 'metawiki' ) LIMIT 10;
+------------------------+----------+----------------+

+------------------------+----------+----------------+

+------------------------+----------+----------------+
10 rows in set (3 min 47.56 sec)

(In reply to Nemo from comment #67)

I don't think this is solved at all, there is no shortage of recent
examples. For instance, on Meta (silly query):
[...]

Ok, that indeed looks scary... not sure how soon I'll find time for this, but I guess I need to further look into that (and maybe try to chase it down in production...).

(In reply to Marius Hoch from comment #68)

Ok, that indeed looks scary... not sure how soon I'll find time for this,
but I guess I need to further look into that (and maybe try to chase it down
in production...).

hoo: Any look with that? :-/

Any new local accounts will be unattached if the global account has any currently unattached accounts. Maybe that's what's going on?

(In reply to Aaron Schulz from comment #70)

Any new local accounts will be unattached if the global account has any
currently unattached accounts.

I doubt this is generally correct: I got dozens new correctly attached accounts after the unattached ones were created. [[Special:CentralAuth/Nemo_bis]]

Maybe that's what's going on?

That could make things worse but it's definitely not the (original) source of the problem. Worth checking if the most recent autocreated unattached accounts fall under this case though.

(In reply to Aaron Schulz from comment #70)

Any new local accounts will be unattached if the global account has any
currently unattached accounts. Maybe that's what's going on?

That's not true... if you're logged in globally CentralAuth will attempt to add a user and attach it unconditionally. It's supposed to either create an account and attach it or to not create an account.
CentralAuth at no point should create invalid users which can only be fixed by renaming or sysadmin intervention (and I have no idea why that happens, although I've already stepped through the code multiple times).

Look at the addUser() method, which has the line:

if ( !$central->exists() && !$central->listUnattached() ) {
...
}

(In reply to Aaron Schulz from comment #73)

Look at the addUser() method, which has the line:

if ( !$central->exists() && !$central->listUnattached() ) {
...
}

And how does that work when the various DBs are not in sync, which is the whole point of this bug? :-)

Recent counter-example (same query as above): https://tools.wmflabs.org/pathoschild-contrib/stalktoy/index.php?target=Captain114b
Unattached account autocreated on 2014-05-14 05:26, gets a correctly attached account 6 minutes later (and 9 more after 2 min).

(In reply to Aaron Schulz from comment #73)

Look at the addUser() method, which has the line:

if ( !$central->exists() && !$central->listUnattached() ) {
...
}

Relevant part here is: !$central->exists()
Please read what this bug is about before commenting.

(In reply to Marius Hoch from comment #75)

(In reply to Aaron Schulz from comment #73)

Look at the addUser() method, which has the line:

if ( !$central->exists() && !$central->listUnattached() ) {
...
}

Relevant part here is: !$central->exists()
Please read what this bug is about before commenting.

Thanks.

Change 146051 had a related patch set uploaded by Legoktm:
Add verbose debug logging for bug 39996

https://gerrit.wikimedia.org/r/146051

In CentralAuthHooks::attemptAddUser (used for autocreations), we call $user->addToDatabase(). That works fine, since these users are created with blank passwords/emails, as should happen.

The attachment OTOH isn't happening. 7 users had this happen to them on commons in the past 2 days, so I looked at them. None of them had localnames rows. I just ran migratePass0 (coincidentally) 2 days ago, so we know those tables were in sync.

So, my hypothesis is that for whatever reason, CentralAuthPlugin::initUser is not being executed properly, specifically the part where it attaches and creates the localnames row.

In any case, my plan for now is to just add extremely verbose debug logging, and see what happens.

Change 146051 merged by jenkins-bot:
Add verbose debug logging for bug 39996

https://gerrit.wikimedia.org/r/146051

Change 146150 had a related patch set uploaded by Legoktm:
Add verbose debug logging for bug 39996

https://gerrit.wikimedia.org/r/146150

Change 146151 had a related patch set uploaded by Legoktm:
Add verbose debug logging for bug 39996

https://gerrit.wikimedia.org/r/146151

Change 146150 abandoned by Legoktm:
Add verbose debug logging for bug 39996

https://gerrit.wikimedia.org/r/146150

Change 146150 restored by Legoktm:
Add verbose debug logging for bug 39996

https://gerrit.wikimedia.org/r/146150

Change 146150 merged by jenkins-bot:
Add verbose debug logging for bug 39996

https://gerrit.wikimedia.org/r/146150

Change 146151 merged by jenkins-bot:
Add verbose debug logging for bug 39996

https://gerrit.wikimedia.org/r/146151

As an update, some more broken accounts have been created, all still on commonswiki. I looked at the log entries for two of the accounts briefly, but didn't find anything obvious. Will be doing some more investigating tomorrow.

As another update: I've just deployed a fix for a race condition which *might* be the cause of this bug. Please stay tuned.

(In reply to Marius Hoch from comment #87)

As another update: I've just deployed a fix for a race condition which
*might* be the cause of this bug. Please stay tuned.

That didn't fix it AFAIS, sorry.

New debug logging is pointing to some weird TimedMediaHandler requests, which is filed as bug 69453.

I also found some accounts that the script said was broken as of August 1, but are now fixed???

I327ad84de8b6d16f8d17376add97689bb276d11a was deployed today, which should fix one of the causes of the issue. Also we deployed a logging change to see if there were any open database transactions.

As of a few hours ago there are 726 broken accounts, hopefully it'll stay that way. I'll check again in a day or two to see if it goes up.

(In reply to Kunal Mehta (Legoktm) from comment #90)

I327ad84de8b6d16f8d17376add97689bb276d11a was deployed today, which should
fix one of the causes of the issue. Also we deployed a logging change to see
if there were any open database transactions.

As of a few hours ago there are 726 broken accounts, hopefully it'll stay
that way. I'll check again in a day or two to see if it goes up.

Helped a little bit, only 764 broken accounts now. There are some action=token requests coming in from mobile that seem to be causing it now.

Deployed I715c3e2b77ba28efc36a375ee214021f1334a1d1 today, we'll see how that goes.

Lots of bugs on Bugzilla are about broken things...

There have been no new broken accounts since Sept 5th (possibly earlier). Yay!

Closing this as fixed, but I'll keep an eye on it to make sure it doesn't come back.

• Bug 29234 has been marked as a duplicate of this bug. ***

• Bug 35792 has been marked as a duplicate of this bug. ***

• Bug 39060 has been marked as a duplicate of this bug. ***