SUL3 is on the testwikis now, and could use some extra manual testing.
Some things that could be tested:
Login
- basic password login - PASSED ✅
- "keep me logged in checkbox" (should result in centralauth_Token cookie with 1-year expiry on the wiki where you are logging in) - PASSED ✅
NOTE: [Test Above] The centralauth_Token cookie is set on the top-level/parent domain .wikipedia.org and has a 1 year expiry (when I log in via test wikipedia) and also the 1-year expiry applies to the local domain where I'm logged into (test.wikipedia.org).
- forced password change on login when having a weak password - PASSED ✅
NOTE: [Test Above] A forced change password view was presented to me on successful login with with password. View says: Your password is not valid: Passwords must be at least 10 characters. The password entered is in a list of very commonly used passwords. Please choose a more unique password. Please choose a new password now, or click "Skip" to change it later.
- login with a temporary password - PASSED ✅
- login with a temporary password, on a different wiki than where the email was sent from - PASSED ✅
NOTE: [Test Above] The temporary password I got on testwiki also worked during login on testwikidata.
- captcha appears after a few failed login attempts for the same user, and prevents further login attempts unless correctly filled out - PASSED ✅
NOTE: The above test passes. A CAPTCHA security check is triggered after 3 failed login attempts for the same user.
- login gets throttled after even more failed attempts - PASSED ✅
NOTE: [Test Above] After like 7 - 10 times of failed attempt, I got the message You have made too many recent login attempts. Please wait 5 minutes before trying again..
-
blocked user cannot login, gets reasonable error message- per T384232#10578447 -
same for locally spam-blacklisted user (test both the JS dropdown on the username field, and the form submit)- not used for login. - [autocreation] same for locally title-blacklisted user (test both the JS dropdown on the username field, and the form submit) - PASSED ✅
- [autocreation] same for username blocked by AbuseFilter (test both the JS dropdown on the username field, and the form submit) - PASSED ✅
NOTE: [Test Above] Create the rule (ref. https://test.wikipedia.org/wiki/Special:AbuseFilter/277) to restrict accounts with cSUL3 in their username on test wiki (after creating the account on mediawiki-wiki). On trying to login on testwiki, the rule caught the autocreation and blocked it with message: Auto-creation of a local account failed: This action has been automatically identified as harmful, and you have been prevented from executing it. In addition, to protect Wikipedia, your user account and all associated IP addresses have been blocked from editing. If this has occurred in error, please contact an administrator. A brief description of the abuse rule which your action matched is: Test AF rule - block username with cSUL3 from autocreation. See AF log: https://test.wikipedia.org/w/index.php?title=Special:AbuseLog&wpSearchFilter=277
- same when trying to log in on a closed wiki (see T387736) - PENDING 🚧
NOTE: [Test Above] No closed test wikis as for now. Will wait until SUL3 rolls out to stewardwiki.
- test one of the mitigations in PrivateSettings - PASSED ✅
- TOTP second factor - PASSED ✅
- WebAuthn second factor (nice to have - T376021; probably no way to set up a working key on auth.wikimedia.org at this point)
- starting signup, switching to login via user menu - PASSED ✅
- security reauthentication (e.g. when doing a password change) works - PASSED ✅
NOTE: [Test above] I did this several times with Special:ChangeCredentials to be sure. The way it works is a little different. So after I login and visit the change credentials special page, I leave it open for more than 5mins (.ref) then try to change the password, I get redirected to the same special page (with a authUniqueId query param) and the password inputted doesn't work. I get re-presented the change credentials to make a password change again (which I think that's the security reauth code path being triggered). But I was expecting that it will ask me to login again but it doesn't. The sec reauth code path does make an attempt to redirect to the UserLogin special page but it looks like because I still have an active session, that just works (like autologin?). But in the end, the password change doesn't succeed and we should maybe let the user know? rather than just present the page again?
- OAuth flow: use OAuth-based identity while not being logged in on Wikimedia (the OAuth Authorization would have to happen on testwiki, not sure if there's an existing tool like that, or we need to create a new tool for testing)
- login via a permission error redirect (e.g. visit Special:Preferences while logged out) - PASSED ✅
NOTE: [Test Above] Visited Special:Preferences and got redirected to the shared domain login page. After successful login, get redirected back to the special preferences page.
- NewUserMessage autocreate welcome message - PASSED ✅
NOTE: The links are correct too: https://test2.wikipedia.org/wiki/User_talk:Tgr-test-c1121055
- checkuser data is logged after successful login, including client hints - PASSED ✅
- checkuser data is logged after failed login login, including client hints - PASSED ✅
- LoginNotify email is sent after login from new device - PASSED ✅
- LoginNotify email is sent after failed login - PASSED ✅
-
login via fallback URL (once T377140 is done)- (declined per T377140#10575339)
Signup
- basic user account creation - PASSED ✅
- email notification gets sent, links use canonical domain - PASSED ✅
NOTE: [Test Above] Yes, the link works correctly. It's in the canonical form e.g. https://test.wikipedia.org/wiki/Special:ConfirmEmail/<code-REDACTED>
- captcha works - PASSED ✅
- signup gets throttled after a few successful signups from the same IP; can be unthrottled with resetAuthenticationThrottle - PASSED ✅
NOTE: [Test above] Per this config, the limit is set to 6 account creations in 24 hours (from the same IP) by default. I created TestAccountC1 to TestAccountC6 and when I trying to create TestAccountC7, I got Visitors to this wiki using your IP address have created 6 accounts in the last day, which is the maximum allowed in this time period. As a result, visitors using this IP address cannot create any more accounts at the moment. If you are at an event where contributing to Wikimedia projects is the focus, please see Requesting temporary lift of IP cap to help resolve this issue.
- blocked user cannot sign up, gets reasonable error message - PASSED ✅
- same for spam-blacklisted user (test both the JS dropdown on the username field, and the form submit) - PASSED ✅
NOTE: [TestAbove] The SpamBlacklist extension provides a global email blacklist (.ref) that was used to test this. Together with @Krinkle (I tip my hat to you), we found out that this uses PrivateSettings.php under the hood when blocking creating an account with an email in the blacklist. The error message was obscure, it faked the use of acct_creation_throttle_hit (see en.json in core) instead of actually reporting about the use of a spam blacklisted email.
- test one of the mitigations in PrivateSettings - PASSED ✅
- same for title-blacklisted user (test both the JS dropdown on the username field, and the form submit) - PASSED ✅
NOTE: [Test above] Tested with the global title blacklist (.ref) and got The username "TestAccountWMF" has been banned from creation. It matches the following disallowed titles list entry: .*(WMF|WⅯF).* <newaccountonly|antispoof> (for example). TODO: add a local test wiki title blacklist on new account only and try it out.
- same for username caught by AntiSpoof (test both the JS dropdown on the username field, and the form submit) - PASSED ✅
NOTE: [Test above] Tried creating an account "T3stAccount" and got The username "T3stAccount" is too similar to the following username: Test Account. Please choose another username.
- same for username blocked by AbuseFilter (test both the JS dropdown on the username field, and the form submit)
NOTE: I've not yet seen something that is blocked by an abuse filter (for account creation) that is not already covered by title blacklist. For example trying to create user: Dgrawp has an abuse filter: https://test.wikipedia.org/wiki/Special:AbuseFilter/24 but is already covered by title blacklist. I get the error: The username "Dgrawp" has been banned from creation. It matches the following disallowed titles list entry: .*[GⒼĜĞĠĢƓǤǦǴḠ].*[RŔⓇŖŘȐȒṘṚṜṞ®].*[AǼAÀⒶÁÂÃÄÅĀĂĄǍǞǠǺȀȂȦḀẠẢẤẦẨẪẬẮẰẲẴẶÆ4@].*[WŴẀẂẄẆẈ₩].*[PƤṔṖǷ₧ÞþΡρРр].* <newaccountonly>
NOTE: Extra checks locally, I get This action has been automatically identified as harmful, and therefore disallowed. If you believe your action was constructive, please inform an administrator of what you were trying to do. A brief description of the abuse rule which your action matched is: Avoid account creation with certain names, so this works.
- signup is disallowed on a closed wiki - PASSED ✅
- something something IPReputation? not sure if this is testable in production
- starting login, switching to signup via form button - PASSED ✅
NOTE: [Test Above] On testwiki, the link that this button sits on has a campaign=loginCTA parameter in the URL and that works correctly too.
- starting login, switching to signup via user menu - PASSED ✅
- GrowthExperiments signup flow (signup should end with Special:WelcomeSurvey) - PASSED ✅
- GrowthExperiments signup-during-edit flow (signup from VE should end with Special:WelcomeSurvey after page save) - PASSED ✅
- customized landing pages (especially that it doesn't result in auth.wikimedia.org URLs for some inolved page / message due to parser cache pollution) - PASSED ✅
NOTE: Per GE campaign pattern, I tested with https://test.wikipedia.org/w/index.php?title=Special:CreateAccount&returnto=Main+Page&campaign=typage-test-test-1, you can see that on the right side of the signup form, it looks different from without that campaign param supplied. Also, the links on that landing page works properly.
- NewUserMessage welcome message (especially that it doesn't end up with auth.wikimedia.org URLs due to parser cache pollution) - PASSED ✅
- campaign URL parameter results in user preference correctly set - PASSED ✅
- incubatortestwiki-project/incubatortestwiki-code user preferences correctly set when using a signup link with testwikiproject / testwikicode query parameters on Incubator - FAILED ❌
NOTE: [Test above] Wikimedia Incubator extension is not enabled on test wikis it seems.
- temp user creation via edit works in basic editor - PASSED ✅
- temp user creation via edit works in some JS editor, e.g. DiscussionTools - PASSED ✅
NOTE: [Test Above] The discussion topic created (with a corresponding temp account) that is signed by the temp account. See: https://test.wikipedia.org/w/index.php?title=Talk%3ATest_page#Test_test
- temp user signing up for named account - PASSED ✅
API
- login via action=clientlogin (on a local domain) - PASSED ✅
- bot login via action=login - PASSED ✅
- action=logout - PASSED ✅
(future: make sure credentials change APIs are unaffected after T362715)
Central session
- after login or signup, user should be logged in on other registrable domains - PASSED ✅
NOTE: [Test Above] Logged into test.wikipedia.org and was also logged into test2.wikipedia.org. Also visited test.wikidata.org, it detected that I was centrally logged in and I reloaded to gain a session there.
- "keep me logged in" state is transferred correctly - PASSED ✅
- after deleting cookies on a given domain, centrally logged-in user should autologin - PASSED ✅
- after deleting cookies on a given domain and setting CentralAuthAnon=1 cookie, centrally logged-in user should autologin when clicking login link - PASSED ✅
NOTE: [Test Above] Cleared all cookies on the given domain and set only the CentralAuthAnon cookie to value 1 and it works.
- after temp user creation, temp user should be logged in on other registrable domains - FLAKY 🚧
NOTE: [Test Above] This doesn't work as expected. If I make an edit as anon user on test.wikipedia.org, I was expecting that going to test2.wikipedia.org, I would be logged-in there but I don't have that experience. Visiting test2wiki shows I logged out (no temp account). I filed T383136: [SUL3] Make edge-login to work for temporary accounts (on wikifarm with same parent domain) related to it. To add, this is flaky, doesn't work all the time (works sometimes <~2025-24242 on https://test.wikipedia.org/wiki/Test_page> and not for others <~2025-23466 on https://test.wikipedia.org/wiki/Sandbox>), this is the first time I'm experiencing this in production and have experienced this locally too.
- "keep me logged in" state is transferred correctly - FLAKY 🚧
NOTE: [Test above] Works but flaky as mentioned in the bullet above this one. So when it works, state is transferred but when it doesn't work, it's not transferred.
- after deleting cookies on a given domain, centrally logged-in temp user should autologin - PASSED ✅
- after deleting cookies on a given domain and setting CentralAuthAnon=1 cookie, centrally logged-in temp user should autologin when clicking login link - PASSED ✅
- logout clears the centralauth_* cookies on registrable domains other than the current one - PASSED ✅
Instrumentation
NOTE: followed the docs at https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To#EventStreams and it seems to work nicely.
- accountcreation/account_conversion gets logged during login page view + after successful login, and has correct SUL3 flag - PASSED ✅
- serversideaccountcreation gets logged after signup and has correct SUL3 flag - PASSED ✅
- accountcreation/account_conversion gets logged during signup page view + after successful signup, and has correct SUL3 flag - PASSED ✅
- accountcreation/block gets logged after signup attempt from a blocked IP, and has correct SUL3 flag - PASSED ✅
- sul3_authentication_start_total / sul3_authentication_end_total gets incremented during login / signup - PASSED ✅
NOTE: [Test above] I could confirm this with: https://grafana-rw.wikimedia.org/d/0xzeAwSSk/derick-s-playground?orgId=1&viewPanel=16&from=1740405063098&to=1740405627260 (you can see how that window rises). I did a lot of logins in SUL3 mode.
Rollout
-
using the fallback login link results in SUL2 login workflow. Central autologin remains functional. (once T377140 is done)- (declined per T377140#10575339)
NOTE: [Test Above] Using the usesul3=0 link triggers SUL2 flow correctly.
- same for signup. User still gets opted into SUL3.
- when signing up on a SUL3-for-signup-enabled wiki, the next login on the same registrable domain will result in a SUL3 flow. (once T377144 is done)
- when an existing user got opted into SUL3, their next login (on any registrable domain where they have logged in recently) will result in a SUL3 flow. (once T384215 is done)
Other
- taking a long time to fill out the login form (more than the 5 minute login session expiry)
NOTE: this behaves poorly in the scenario where SUL3 is enabled for signup but not login: user goes to shared domain, creates account after 5+ minutes, ends up on the local domain login page with a session expiry error. The signup was successful but 1) no way to know, 2) it's not clear to the user why they are seeing a login form when they were trying to sign up.
(future: credentials change workflows after T362715)