Using wrong pw or 2fa code gives "Invalid credentials" so this appears to be something different.

I tried disabling and re-enabling 2fa. Still the same error.

Seems like a user-specific problem. Login works for me.

Yeah, well, still not working for me. Maybe somebody could take a look.

@dschwen still happening? @Andrew ping if so

Yes, it is still happening.

@dschwen is 2fa working for you via wikitech currently? Can you disable and renable 2fa and see if it works in both venues?

Most often this is a result of clock drift on the device providing the 2fa code. Rebooting your phone might help.

I'll try that, but I seriously doubt this is the issue here. I use GA for a whole bunch of services and horizon is the only one that gives me grief. (and compared to https://time.is/ my phone is within one second)

Also, @Legoktm mentioned above:

Using wrong pw or 2fa code gives "Invalid credentials" so this appears to be something different.

Yeah, no change

Ok, up till now I had no pressure to get on horizon, but I need to rebuild an instance now, and being unable to log in is becoming a major showstopper for me now. I'd really appreciate some help on this.

This is because of a case mismatch between ldap and mediawiki. The mediawiki user_name table had the username 'Dschwen' but ldap had the cn as 'dschwen'.

Keystone was first looking the user up in ldap (case insensitive, as you might hope) but then our custom 2fa keystone plugin used the cn from the ldap record to do a query in the oathauth db. The ensuing case mismatch threw a 400.

I've changed dschwen's ldap cn to be 'Dschwen' and all is well. It appears that all semi-modern ldap records already have a cn that matches the mediawiki username, so probably this is a legacy of some tiny bug in OSM from many years ago.

NEVERTHELESS -- this is stupid and shouldn't happen. Keystone should either use bd808's 2fa auth plugin, or do something moderately smarter when searching for the 2fa token.

Change 345231 had a related patch set uploaded (by Andrew Bogott):
[operations/puppet@production] Keystone 2fa: Use the wikitech API rather than checking the db directly.

https://gerrit.wikimedia.org/r/345231

Change 345231 merged by Andrew Bogott:
[operations/puppet@production] Keystone 2fa: Use the wikitech API rather than checking the db directly.

https://gerrit.wikimedia.org/r/345231