@greg I completely agree that this ended up in the planning email but both @Anomie and I are a bit at a loss as to what @dduvall was suggesting for the new tests to do. I guess the first thing we need to do is work that out? I remember him saying something about a series of tests that would run with massive parallelism?

Yep, that's the first step: clarifying purpose and plan.

It was sort of a stab in the dark, but my thought was to try performing a high number of concurrent logins on testwiki or test2wiki—or somewhere else we can have a setup that's close enough to production and has both SessionManager and CentralAuth in the mix, like mw1017. I would need a pool of user accounts (maybe ~100 to start) and somewhere I could set up and run Gatling.

It's admittedly a very brute force approach so, since y'all know the nature of this code much more intimately, let me know if this is categorically a waste of time and I won't bother with it.

In general and in light of this particular case, I do think it's a good idea to broaden our end-to-end test cases for authentication and exercise them regularly against production wikis—I'd also like to make this a formal part of the train process to verify group0 and group1. Right now, we have 5 end-to-end test scenarios and they only run against BC; one scenario had also been falsely failing for over a month before I fixed it late last week so I kind of doubt we would have seen a useful failure had it occurred.

Not sure what benefit to expect from parallel logins, but having test for the various CA scenarios would be very useful. Some scenarios with significantly different code paths that come to mind:

• login on loginwiki
• login on another wiki
• autologin on a $wgCentralAuthAutoLoginWikis wiki after logging in to a wiki on the same second-level domain
• same with a browser that does not support third-party cookies
• autologin on a $wgCentralAuthAutoLoginWikis wiki after logging in to a wiki on a different second-level domain
• autologin on a non-$wgCentralAuthAutoLoginWikis wiki
• autologin on a non-$wgCentralAuthAutoLoginWikis wiki with Javascript disabled

+ most of the above with "remember me" cookie set, with or without an active central session
+ most of the above with account creation instead of login
+ most of the above with autocreation instead of autologin
+ most of the above with API requests instead of a browser

• OAuth request
• bot password request

Edit: also, there are lots of different expiration scenarios (token cookie exists but session cookie expired, redis session expired, redis central session expired, loginwiki session expired...)

Some more information about CentralAuth code paths:

• Anomie's summary of how it works
• sketch of the control flow

Thanks, @Tgr! It's tremendously helpful to have such well-defined scenarios to work from.

In our meeting last week, we came to the conclusion that this testing would not necessarily block wmf.13 since it's very unlikely to reproduce the reported behavior(s)—our current WebDriver-based test suite for CA does not reproduce them—but we will commit to improving coverage for the scenarios you've laid out. Having broader end-to-end tests for session management and integrating review of such test results as part of a healthier MW train should help to surface future issues before deployment to group1 and group2.

@greg, I'm not sure about priority at this point but perhaps this is something @zeljkofilipin or I can coordinate on with @Tgr, @Anomie et al early the next quarter? (As part of our "better MW train" commitment/narrative.)

I think we need these tests before rolling out AuthManager which is hopefully going to deploy this quarter.