Page MenuHomePhabricator

Gerrit has created duplicate accounts for some users
Closed, ResolvedPublic

Description

Since upgraded to 2.15 we have noticed several accounts have been duplicated thus when they log in they see no reviews.

For example kaldari account has been duplicated and jbranaa.

Possibly-Related upstream tasks:

Event Timeline

Paladox created this task.Jun 13 2018, 10:01 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 13 2018, 10:01 AM
Paladox triaged this task as High priority.Jun 13 2018, 10:01 AM
Paladox added subscribers: Jrbranaa, kaldari.
Dzahn added a subscriber: Dzahn.Jun 13 2018, 10:11 AM

T138672 might be an example of this bug

[17:51:42] <paladox> you git clone All-Users as your ssh user (which has admin and you need to grant the access db to the admins) then you edit .git/config and replace refs/heads with refs/ then git pull
[17:52:00] <paladox> and then git checkout origin/meta/external-ids

use "grep -rnw './' -e '<user>'" to make it easier for you to find users as they are stored in folders.

@kaldari account has two external ids if i look at http://gerrit.wikimedia.org/r/accounts/?q=name:Kaldari+email:rkaldari@wikimedia.org&n=2

[
  {
    "_account_id": 78
  },
  {
    "_account_id": 6099
  }
]

where as @Jrbranaa has one.

FYI, I basically can't do any development work in the meantime since I can't use gerrit (can't git pull, git push, etc.). Would appreciate if this gets fixed soon. Thanks!

Paladox raised the priority of this task from High to Unbreak Now!.Jun 13 2018, 6:23 PM
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptJun 13 2018, 6:23 PM

ok I'm confused. external_id 6099 doesn't appear to be in the gerrit database, at least not in the account_external_ids table

@mmodell see https://phabricator.wikimedia.org/T197083#4280194 (we migrated to notedb so it's now in All-Users instead of the db)

remote: Resolving deltas: 100% (2/2)
remote: Counting objects: 38496, done
remote: Branch refs/meta/external-ids:
remote: You are not allowed to perform this operation.
remote: To push into this reference you need 'Push' rights.
remote: User: twentyafterfour
remote: Please read the documentation and contact an administrator
remote: if you feel the configuration is incorrect
remote: Processing changes: refs: 1, done    
To ssh://twentyafterfour@gerrit.wikimedia.org:29418/All-Users
 ! [remote rejected]           HEAD -> refs/meta/external-ids (prohibited by Gerrit: ref update access denied)

ok I added push rights for the external-ids ref and now I got a ton of errors due to non-unique emails

cleaning this up will be difficult

They reverted it for performance reasons

I wonder if this might be related to the old T152640 bug (which also affected me).

Yes that seems related as 2.15 uses the index again I think

We already set that :)

Just ssh is case sensitive and logins are case insensitive.

@kaldari can you try logging into gerrit again? I think I might have it sorted.

@Jrbranaa Can you try as well? Your duplicate account has been disabled and stripped of its email address.

@Jrbranaa Can you try as well? Your duplicate account has been disabled and stripped of its email address.

@mmodell Looks like it's working. Thanks.

Paladox lowered the priority of this task from Unbreak Now! to High.Jun 14 2018, 4:54 PM

@mmodell: Didn't fix it for me. I can log into Gerrit on the web, but my account has nothing in it. Still can't use gerrit via git, so no change for me.

@kaldari: ok I'll poke around some more and see what else I can find.

@mmodell updated the upstream report with the current best plan he has and answers to all their specific questions: https://bugs.chromium.org/p/gerrit/issues/detail?id=9256#c13

@kaldari has a workaround currently (using a new user account, unfortunately — but works for now).

We'll wait to see what response upstream has to our current plan of action.

mmodell updated the task description. (Show Details)Jun 18 2018, 6:09 PM

Change 441032 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] Gerrit: Set log level to debug for AccountManager

https://gerrit.wikimedia.org/r/441032

Change 441032 abandoned by Paladox:
Gerrit: Set log level to debug for AccountManager

Reason:
Actually on second thought you can use https://gerrit-review.googlesource.com/Documentation/cmd-logging-set-level.html to do this

https://gerrit.wikimedia.org/r/441032

Paladox removed a subscriber: gerritbot.

Hey y'all, it's been 2 weeks and I still can't use my gerrit account. I've set up a temporary account, but I thought "temporary" meant a day, not 2 weeks. I still need to get my real gerrit account working again as that's the one that people assign as a reviewer, etc. This is having a detrimental effect on my work.

Hey y'all, it's been 2 weeks and I still can't use my gerrit account. I've set up a temporary account, but I thought "temporary" meant a day, not 2 weeks. I still need to get my real gerrit account working again as that's the one that people assign as a reviewer, etc. This is having a detrimental effect on my work.

We're still working towards finding the root problem and verifying a solution with upstream. You can follow along on or chime in on the upstream task: https://bugs.chromium.org/p/gerrit/issues/detail?id=9256#c19 The plan was to remove the duplicate account from the NoteDB inside gerrit; however, NoteDB is built as some kind of DAG so there may be pitfalls to simply removing your accountId file that we can't foresee. I'd like to verify the solution with upstream before we schedule the necessary gerrit downtime to implement it and run a full reindex (our current best plan).

I'm sorry this is a pain. You, as far as I can tell from digging in this new NoteDB thing, may be the only person affected by this. We're working to set this straight but Release-Engineering-Team is missing our resident gerrit internals expert so we're learning a lot over a short period, thanks for bearing with us.

CommunityTechBot renamed this task from s4aaaaaaaa to Gerrit has created duplicate accounts for some users.Jul 2 2018, 8:09 AM
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: Aklapper.

@thcipriani: Some more information...

Logging into https://gerrit.wikimedia.org/ with either kaldari or Kaldari fails ("Authentication failed.").

sshing with kaldari fails, but Kaldari works:

WMF645:core kaldari$ ssh -p 29418 kaldari@gerrit.wikimedia.org
kaldari@gerrit.wikimedia.org: Permission denied (publickey).
WMF645:core kaldari$ ssh -p 29418 Kaldari@gerrit.wikimedia.org

  ****    Welcome to Gerrit Code Review    ****

  Hi kaldari, you have successfully connected over SSH.

  Unfortunately, interactive shells are disabled.
  To clone a hosted Git repository, use:

  git clone ssh://Kaldari@gerrit.wikimedia.org:29418/REPOSITORY_NAME.git

Connection to gerrit.wikimedia.org closed.

@thcipriani, @mmodell: Just to let you know, I would be totally fine with having all my gerrit accounts deleted (and their history, etc.), if it meant I could successfully use gerrit again under the username kaldari (as that's the username I've been using for 8+ years and that everyone knows). I've been locked out of my account for 2 months now, and I know y'all are doing your best to address the problem, but maybe it's time to consider more drastic ideas. Any thoughts on that?

@kaldari: noted. I'll discuss it with @thcipriani and we will decide on the best course of action to get your account back to normal asap. I'm sorry this has been taking forever. It's really frustrating, and even more so for you than for us I'm sure.

Quick update. I pushed up a new patchset for review for the plugin we're working on to resolve the problem in the notedb database: https://gerrit-review.googlesource.com/c/plugins/wmf-fixshadowuser/+/190030

There is no documentation for how to delete an account using notedb since it's new and upstream doesn't want users to be deleted (https://groups.google.com/d/msg/repo-discuss/JlV0i9w5lCY/F25h2t_HQzQJ). I'll ask how to do this and what the ramifications are if you're fine with losing all history.

I'm totally fine with losing all history at this point :)

Quick update. I pushed up a new patchset for review for the plugin we're working on to resolve the problem in the notedb database: https://gerrit-review.googlesource.com/c/plugins/wmf-fixshadowuser/+/190030
There is no documentation for how to delete an account using notedb since it's new and upstream doesn't want users to be deleted (https://groups.google.com/d/msg/repo-discuss/JlV0i9w5lCY/F25h2t_HQzQJ). I'll ask how to do this and what the ramifications are if you're fine with losing all history.

There's a plugin that delete accounts (though would possibly need to be changed for ldap?) the plugin won't work with ldap.

see https://gerrit-review.googlesource.com/admin/repos/plugins/account https://gerrit-review.googlesource.com/q/project:plugins/account

This should hopefully be resolved, at least for @kaldari, on Thursday.

Again, I apologize for the long delay.

@kaldari: Can you confirm whether logging into your old gerrit account is now working and whether you can see your history?

@mmodell: OMG, it works! I can log into my old account and all the history is still there too! Thank you! Thank you!

BTW, I noticed that some of the old bogus accounts like "Kaldari" (uppercase) and "kaldari2" are now gone (or merged?). Would it be possible to delete or merge my temporary account I made to work around the bug (which also uses rkaldari@wikimedia.org)? No rush, just wanted to ask in case it's easy now. Thanks!

hashar removed a subscriber: hashar.Aug 20 2018, 9:31 AM

This should hopefully be resolved, at least for @kaldari, on Thursday.

Do we know of more affected users?

mmodell closed this task as Resolved.Aug 29 2018, 4:05 PM
mmodell claimed this task.

I manually fixed all of the others that I'm aware of.

BTW, I noticed that some of the old bogus accounts like "Kaldari" (uppercase) and "kaldari2" are now gone (or merged?). Would it be possible to delete or merge my temporary account I made to work around the bug (which also uses rkaldari@wikimedia.org)? No rush, just wanted to ask in case it's easy now. Thanks!

AFAIK we aren't able to merge accounts.