Page MenuHomePhabricator

Move from local wiki user IDs to global IDs
Closed, ResolvedPublic

Description

Until now, when Wikipedians register on the library card platform, their database username is generated as [language code][numbers]. This means that if they sign in from another language on the first login page, another user account is created. We should provide users with the same account regardless of the language Wikipedia they select and/or oauth from.

Event Timeline

Okay, I have a naive (read: lazy) start to this up on staging. I dropped all options but meta from the login menu. The menu will go away, so that we'll just have a login button that takes you straight to meta.

I visited https://meta.wikimedia.org/wiki/Special:MergeAccount and got a SUL created for myself.
I then popped into the staging database, and dropped the language prefix on uids:

UPDATE auth_user SET username = SUBSTR(username, 3);

I had to drop several accounts that had registered under multiple home wikis (including @Samwalton9), leaving just one account per person. Note that this was also naive, as I didn't check to see if everything actually had a two-character prefix. I'll check to make sure I'm actually setting that correctly for the real deal.

OAuth works out of the box in this configuration, as we're doing less than we were doing before. The pieces that will almost certainly need some attention are authorization bits, such as edit count. We won't be collecting home wiki out of the box with this setup either.

@Samwalton9, we could test the edit count by deleting your remaining account, and having you sign back up. I'm guessing that the current behavior would be to capture your edits on meta, rather than in aggregate across all wikis.

Awesome, seems to be working!

There were a few accounts that didn't start with a language code (andromeda, vagrant, twl-tester; https://twlight-staging.wmflabs.org/admin/auth/user/?p=1), but I'm not sure that they're being used right now anyway.

Did you do a new check for duplicate users on live? I know I was the only one with a duplicate username last time you checked, but it might be worth a quick check before we do this live in case someone has recently applied from multiple logins.

I don't seem to be able to find myself in the user list in /admins, other than my previous en login. Would that be because it's realised I'm the same user as the one who previously logged in from en?

My account seems to just be grabbing my meta edits as my edit count, yeah. Going forward we'd want the edit count to be across all languages.

Yeah, don't worry about those oddball accounts. Those are all dev/test accounts.

We have a few duplicate users on live. I checked two different ways. Once for auth users:

select SUBSTR(username, 3) AS auth_user from auth_user GROUP BY auth_user HAVING COUNT(*) >1;

and once for WPL editors

select wp_username from users_editor GROUP BY wp_username HAVING COUNT(*) >1;

It's still a very small number of users. I just stuck those queries in here for convenience as we work on this issue.

Since you had duplicate accounts, I just arbitrarily threw one out. In this case, I kept the en account.

BTW, I peeked through the API docs last night and put together the query we'll use to fetch global edits and home wiki for a given user. The result is pretty close to the data structure we were already getting from OAuth, so it shouldn't be too difficult to integrate into the authorization code.
Here's an example query for @Samwalton9
https://www.mediawiki.org/w/api.php?action=query&meta=globaluserinfo&guiuser=Samwalton9&guiprop=editcount&format=json&formatversion=2

We should make sure that all accounts are SUL accounts. Jason had issues logging in to Meta until he went to Special:MergeAccount. If that's still necessary we may need to present users with a link at login in the case of issues.

Okay, I pushed an update to staging. @Samwalton9, can you login there again and see if your edit count gets correctly updated?

Mmk, I went ahead and updated the sign in option in the navigation to skip the intermediate login page.
I'm going to delete your account from staging, so that you can do a fresh signup.
We'll need to make sure that we're capturing edit counts correctly when we mint new accounts, not just on subsequent logins.

Related question:
It turns out that it's very easy to get the home wiki from the global user info query, though it's in a slightly different format than what we were using before. Do we want to
a) capture that moving forward
and do we need to
b) care about previously recorded values?

fulfilling a should be less than 10 lines of code altogether.
fulfilling b will require munging our existing data and checking it for correctness.

And, @Samwalton9 no longer has an account on staging. I'd like to encourage him to sign up again!

When I click Log in on the main page (where the home wiki select was previously), I now get an Internal server error (on the URL https://twlight-staging.wmflabs.org/oauth/callback/?oauth_verifier=6e882e31edf7964f6adef5b3ed532707&oauth_token=6c1f4c7b8315381da849149559b82217).

it's very easy to get the home wiki from the global user info query

What does the query define as 'home wiki'? The place the account was created? The place with most edits? Something else? Maybe we should do neither and simply let the user define their home wiki, depending on that answer.

Well, that blows up a little earlier in the process than I had hoped it would. I'll delete my own account and work through the account creation code path.

There's a good discussion of what the home wiki represents in SUL here:
https://meta.wikimedia.org/wiki/Wikimedia_Forum/Archives/2011-08#Changing_the_SUL_home_wiki

If we're only using it for statistical purposes, we may be able to get useful information about our user base by making separate api queries for metrics.

I think the only other reasons we'd want home wiki is for automatically detecting language and for defining a 'not blocked on home wiki' library bundle rule (which may or may not be what that rule looks like)

@Samwalton9 That's good to know. It was the language code that was making the login process blow up on initial sign in. For now, I just updated to work with meta, which is a garbage language code. I'd recommend that if we want to track that for a user, we should just use the language code requested by their browser. Eg, if their browser is set to ask for french when they sign in, the language code would get set to french for that user. Is this something we're using for statistical purposes? We don't need to record the value to give users the correct translation if we have it.

'not blocked on home wiki' should be a thing I can fish out of the api.
looks like I also need to update the code that grabs email addresses.

You should be able to sign in now.

I'd recommend that if we want to track that for a user, we should just use the language code requested by their browser.

Oh I didn't know that was possible - definitely sounds like the better way of doing it!

'not blocked on home wiki' should be a thing I can fish out of the api.

Cool, that's something we can decide if we want later (T153431)

Signed in successfully! Got a message about no email, but looks like you're on that already :)

So, when we fetch the global user info, we get back 'home wiki' which is the account they were logged into when they merged accounts. That's in the form of a wiki db code.
we can map that to an actionable url, by asking for a site matrix from the api
https://meta.wikimedia.org/w/api.php?action=sitematrix&smtype=language&smsiteprop=url|dbname&format=json&formatversion=2
We can use the mapped url to then query the home wiki to see if that user is in the list of blocked people
https://en.wikipedia.org/w/api.php?action=query&list=blocks&bkusers=Samwalton9&format=json&formatversion=2

If we don't like using the the globaluser info provided home wiki, we'll need to use some other means to guess it, or to prompt the user (which is where we were with the login menu).

Still working out the lack of email addresses coming through, as it turns out I haven't done anything particularly silly there, which was my starting assumption.

So, update on user emails. It looks like meta simply isn't returning that information as part of its OAuth response. There was some pre-existing code that called the userinfo api (distinct from the global_userinfo api) in an attempt to check for disabled mail for the account, and I thought that would be a good place to grab some additional user info.

Things were looking hinky, so I added in some additional checks. It turns out the way that api was being used in this app was totally nonfunctional. If we want to use data from the userinfo api, we need to have the authenticated user's browser make the request and give us the data, rather than have our server make the request on the backend. So I'd call that a bug. Moving those requests client side means javascript, which I know is a thing we want to generally avoid.

A non code intensive way for us to solve the no email problem would be for us to ask the entities responsible for the oauth config on meta to start including email addresses in their responses.

I could also try moving to another backend. It looks like hitting some specific paths on www.wikipedia.org directs me to a local wiki based on my geolocation. I'm testing feeding that into our oauth config to see if we can let wikipedia.org make a best guess on our behalf about where to start.

Hmm. Would it be helpful if I got you in touch with someone who's been through the OAuth-and-I-need-your-email process before for another tool?

Phabricator itself, for example, logs in through OAuth on MediaWiki.org, and seems to somehow grab email in the process.

Hmm, there may be something else going on here. I changed back to en.wikipedia.org on staging and I'm still not getting an email address. I then went to production, where we haven't changed any of this code, and put in a garbage address for myself. I signed out and signed back in and the garbage address was still there. It should have been overwritten with my correct email address.

@Samwalton9 can we check with users to see if their email is getting autofilled on first sign in? If not, this may be a change in how wikipedia is responding to oauth requests generally.

@Samwalton9 The other thing for us to check would be non-unified accounts, to see if their is a difference there.

We can verify this with some test accounts.

Update:

I've been doing some out of band communication along with research on wikimedia's oauth implementation. I think that we may simply need to request an updated oauth key with slightly different grants, but I'm waiting to hear back on what the grants are for our current keys.

So, we should be fixed on production and I'm waiting for a new oauth key to fix staging. Staging, test, and wikipedialibrary oauth keys all had different grants requested for them. test was the only one that actually had adequate grants requested for fetching email addresses and it was the url actually get hit for logins on the live site until I closed T161620, at which time we would have stopped getting email addresses automatically.

Having a single server acting as both test and prod was part of the problem, having inconsistent grants for the keys across environments was another. This was made more difficult by a display bug in wikipedia that made it impossible to view the grants on some of our keys.

Got the email fix in place on staging as well. Will spend a little time tearing down the changes I made to troubleshoot this completely preventable bug.

I commented out all of the references to the nonfunctional authorization check for disabled mail.
I synced back a copy of the live site's database from today. I looked through the account duplicates on staging, and found that in each case:

  • the person had no more than two accounts
  • Only one of those accounts actually owned things like applications or administrative data
  • I was able to delete the account that didn't own anything without problems.
Log entry: Deleted "eo5911131."
Log entry: Deleted "fi12287001."
Log entry: Deleted "en29493."

I reconfigured staging so it is accessible to @Samwalton9 again.

I haven't removed the references to home wiki/language code at this point, since they aren't directly hurting functionality.

I did go ahead and force everyone's home wiki to 'meta' however.

UPDATE users_editor SET home_wiki = 'meta';

Any reference in the interface to the user account will now use meta.wikimedia.org.

Seems to be working well! Logged in to my own account no problem, and logged in on a fresh account no problem too - email intact :)

@Samwalton9 a potential small wrinkle:
When you login to the staging site in a browser configured for a non-english language, is the language on the meta login page correct? If not, I can fix that up with a small change to code. I think this may not currently display the user's preferred language on the login page, but I may not be adequately convincing Wikipedia that I prefer Finnish

For the record: Test this just now, and the platform works correctly, but meta displays in English.

Update pushed to staging, the login page should now be in the language requested by the browser.

there are some css selector changes in this update, so things will look funny unless you do a page reload.

The login/signup button just takes me to the meta wiki homepage now :(

I see what's happened. I'm correctly requesting the localized login page, but I'm not correctly telling wikipedia how to complete the oauth request after that. working.

Okay, try again. Login should work, and login page should be in your language.

Hm. Not being asked to confirm OAUTH now; I click Log in and I get logged in straight away. Tested on multiple browsers + accounts (new and existing). Is that intended behaviour?

The oauth process doesn't allow for any language selection to be baked into the request, so we're preempting the oauth request with a localized login request. By the time the user starts the oauth process, they have already signed in. As long as you're landing on our site as the correct user, we should be good.

Is there any requirement for users to approve the platform to log them in? All other OAuth applications I'm aware of have the confirmation step, so it would be worth finding out if it's necessary (presumably somewhere on this page: https://www.mediawiki.org/wiki/OAuth/For_Developers).

@Samwalton9 this is actually the exact same behavior we already have if a user logs into our site when they are signed into wikipedia beforehand. It's a limitation of the mediawiki oauth implementation. You can test by logging into en.wikipedia..org, going to the live site, and selecting english as your wiki. Upon clicking login, you'll be directed immediately back to our site with a session. No confirmation page.

@Samwalton9 I understand your concern on this one though. We're moving this from an edge case to 100% of logins. I can file a ticket against the mediawiki oauth extension, and maybe even spend some time coming up with a pull request for fixing it, but it would take time for them to review and implement changes, if they decided to make any.

Hm, I still get the "Hi, in order to complete your request we need your permission" dialog box on the live site, even when logged in to Wikipedia already. Maybe this isn't an issue, I'm just not sure why that step disappeared.

Now that's interesting. We're getting different behavior there. What browser are you using?

Chrome. But I just tried it on my laptop and got logged in instantly, and now the same on my PC. I'm certain this wasn't the case before today or earlier... Very strange. Maybe just forget I said anything!

So, we're good to push this change, yes?

Not sure what I thought I was waiting on here. Prepping changes on the production site.

Changes pushed, editor IDs migrated. Oauth tested and working. No more giant login menu!

Fun fact:

We just shed about 600 net lines while making this change. The platform lost 858 lines of code and gained 260 to take their place.