Since upgraded to 2.15 we have noticed several accounts have been duplicated thus when they log in they see no reviews.
For example kaldari account has been duplicated and jbranaa.
Possibly-Related upstream tasks:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Gerrit: Set log level to debug for AccountManager | operations/puppet | production | +4 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • mmodell | T197083 Gerrit has created duplicate accounts for some users | |||
| Invalid | None | T197257 gerrit: run "LocalUsernamesToLowerCase" utility on gerrit host to fix mixed case usernames in gerrit's accounts db | |||
| Declined | None | T199749 Gerrit plugin to edit accountid |
[17:51:42] <paladox> you git clone All-Users as your ssh user (which has admin and you need to grant the access db to the admins) then you edit .git/config and replace refs/heads with refs/ then git pull
[17:52:00] <paladox> and then git checkout origin/meta/external-ids
use "grep -rnw './' -e '<user>'" to make it easier for you to find users as they are stored in folders.
@kaldari account has two external ids if i look at http://gerrit.wikimedia.org/r/accounts/?q=name:Kaldari+email:rkaldari@wikimedia.org&n=2
[
{
"_account_id": 78
},
{
"_account_id": 6099
}
]where as @Jrbranaa has one.
FYI, I basically can't do any development work in the meantime since I can't use gerrit (can't git pull, git push, etc.). Would appreciate if this gets fixed soon. Thanks!
ok I'm confused. external_id 6099 doesn't appear to be in the gerrit database, at least not in the account_external_ids table
@mmodell see https://phabricator.wikimedia.org/T197083#4280194 (we migrated to notedb so it's now in All-Users instead of the db)
remote: Resolving deltas: 100% (2/2) remote: Counting objects: 38496, done remote: Branch refs/meta/external-ids: remote: You are not allowed to perform this operation. remote: To push into this reference you need 'Push' rights. remote: User: twentyafterfour remote: Please read the documentation and contact an administrator remote: if you feel the configuration is incorrect remote: Processing changes: refs: 1, done To ssh://twentyafterfour@gerrit.wikimedia.org:29418/All-Users ! [remote rejected] HEAD -> refs/meta/external-ids (prohibited by Gerrit: ref update access denied)
ok I added push rights for the external-ids ref and now I got a ton of errors due to non-unique emails
this change prevents duplicate emails: https://gerrit-review.googlesource.com/c/gerrit/+/170091
@Jrbranaa Can you try as well? Your duplicate account has been disabled and stripped of its email address.
@mmodell: Didn't fix it for me. I can log into Gerrit on the web, but my account has nothing in it. Still can't use gerrit via git, so no change for me.
@mmodell updated the upstream report with the current best plan he has and answers to all their specific questions: https://bugs.chromium.org/p/gerrit/issues/detail?id=9256#c13
@kaldari has a workaround currently (using a new user account, unfortunately — but works for now).
We'll wait to see what response upstream has to our current plan of action.
Change 441032 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] Gerrit: Set log level to debug for AccountManager
Change 441032 abandoned by Paladox:
Gerrit: Set log level to debug for AccountManager
Reason:
Actually on second thought you can use https://gerrit-review.googlesource.com/Documentation/cmd-logging-set-level.html to do this
Hey y'all, it's been 2 weeks and I still can't use my gerrit account. I've set up a temporary account, but I thought "temporary" meant a day, not 2 weeks. I still need to get my real gerrit account working again as that's the one that people assign as a reviewer, etc. This is having a detrimental effect on my work.
We're still working towards finding the root problem and verifying a solution with upstream. You can follow along on or chime in on the upstream task: https://bugs.chromium.org/p/gerrit/issues/detail?id=9256#c19 The plan was to remove the duplicate account from the NoteDB inside gerrit; however, NoteDB is built as some kind of DAG so there may be pitfalls to simply removing your accountId file that we can't foresee. I'd like to verify the solution with upstream before we schedule the necessary gerrit downtime to implement it and run a full reindex (our current best plan).
I'm sorry this is a pain. You, as far as I can tell from digging in this new NoteDB thing, may be the only person affected by this. We're working to set this straight but Release-Engineering-Team is missing our resident gerrit internals expert so we're learning a lot over a short period, thanks for bearing with us.
@thcipriani: Some more information...
Logging into https://gerrit.wikimedia.org/ with either kaldari or Kaldari fails ("Authentication failed.").
sshing with kaldari fails, but Kaldari works:
WMF645:core kaldari$ ssh -p 29418 kaldari@gerrit.wikimedia.org kaldari@gerrit.wikimedia.org: Permission denied (publickey). WMF645:core kaldari$ ssh -p 29418 Kaldari@gerrit.wikimedia.org **** Welcome to Gerrit Code Review **** Hi kaldari, you have successfully connected over SSH. Unfortunately, interactive shells are disabled. To clone a hosted Git repository, use: git clone ssh://Kaldari@gerrit.wikimedia.org:29418/REPOSITORY_NAME.git Connection to gerrit.wikimedia.org closed.
@thcipriani, @mmodell: Just to let you know, I would be totally fine with having all my gerrit accounts deleted (and their history, etc.), if it meant I could successfully use gerrit again under the username kaldari (as that's the username I've been using for 8+ years and that everyone knows). I've been locked out of my account for 2 months now, and I know y'all are doing your best to address the problem, but maybe it's time to consider more drastic ideas. Any thoughts on that?
@kaldari: noted. I'll discuss it with @thcipriani and we will decide on the best course of action to get your account back to normal asap. I'm sorry this has been taking forever. It's really frustrating, and even more so for you than for us I'm sure.
Quick update. I pushed up a new patchset for review for the plugin we're working on to resolve the problem in the notedb database: https://gerrit-review.googlesource.com/c/plugins/wmf-fixshadowuser/+/190030
There is no documentation for how to delete an account using notedb since it's new and upstream doesn't want users to be deleted (https://groups.google.com/d/msg/repo-discuss/JlV0i9w5lCY/F25h2t_HQzQJ). I'll ask how to do this and what the ramifications are if you're fine with losing all history.
There's a plugin that delete accounts (though would possibly need to be changed for ldap?) the plugin won't work with ldap.
see https://gerrit-review.googlesource.com/admin/repos/plugins/account https://gerrit-review.googlesource.com/q/project:plugins/account
This should hopefully be resolved, at least for @kaldari, on Thursday.
Again, I apologize for the long delay.
@kaldari: Can you confirm whether logging into your old gerrit account is now working and whether you can see your history?
@mmodell: OMG, it works! I can log into my old account and all the history is still there too! Thank you! Thank you!
BTW, I noticed that some of the old bogus accounts like "Kaldari" (uppercase) and "kaldari2" are now gone (or merged?). Would it be possible to delete or merge my temporary account I made to work around the bug (which also uses rkaldari@wikimedia.org)? No rush, just wanted to ask in case it's easy now. Thanks!