SessionManager unpersists the session if it does not match the stored session data (ie. it deletes the session/token cookies if they do not match the redis data). As far as I can see that should only happen for users who did not click "remember me" (otherwise they can be authenticated by user_token even if the session storage is inaccessible).

A user noticing that purge works strangely but not noticing that they have been logged out means that either

• they just did not notice that the user bar is gone. Unlikely but possible I guess.
• MediaWiki sees them as logged-out but generates a logged-in skin for them. I can't see that happening.
• a CSRF token error? that could be caused by session loss / session read failure, but I don't think we require a token from logged-in users.
• OTOH, IIRC, we don't allow purge on GET anymore and do some JS trick to turn it into a POST. Ori probably knows more about that. Maybe that process involves a token or some other session dependency. Or maybe it just failed in some other way due to a network error and fell back to the non-JS behavior of showing the form?

No clue about the CAS error. Those do happen naturally (but rarely) when two processes try to update the user record the same time, or when a process that tries to update the user record is working from outdated user data. I guess if the user object cannot be cached due to network issues, that could make them more frequent?

Change 305595 had a related patch set uploaded (by Aaron Schulz):
Defer preferences update and avoid CAS errors in onUserLoggedIn()

https://gerrit.wikimedia.org/r/305595

As for "example 1", that is normal behavior now (the form must be POSTed whether logged in or not).

In T143367#2566297, @Tgr wrote:

No clue about the CAS error. Those do happen naturally (but rarely) when two processes try to update the user record the same time, or when a process that tries to update the user record is working from outdated user data. I guess if the user object cannot be cached due to network issues, that could make them more frequent?

"No clue" and "happens naturally" don't sound very reassuring :) Any ideas on how to proceed here? We should get to the bottom of this.

For what it's worth:

faidon@fluorine:/a/mw-log$ grep -c 'CAS update failed' exception.log 
560
faidon@fluorine:/a/mw-log$ grep 'CAS update failed' exception.log | grep -c '2016-08-18 12:[01]'
146

12:05 - 12:13 is when the first network partition that I mentioned above happened, so it definitely looks like there is a correlation here.

Could be cache purges not making it to some mc servers and stale cached version of user setting loading and saved (which will error out). In any case, the patch above should avoid the CAS error problem.

Change 305595 merged by jenkins-bot:
Defer preferences update and avoid CAS errors in onUserLoggedIn()

https://gerrit.wikimedia.org/r/305595

(we ask anonymous users for confirmation on action=purge, but not logged-in users)

That's no longer true as of 1.28.0-wmf.15, see rMWc2ce6a1b6085: Require POST for action=purge in PurgeAction.

Tagging us to follow-up and confirm this is gone.

@faidon Are there any known timestamps in the last 30 days during which we know a similar situation existed in the data centre, to help correlate with error logs?

It's been a couple of years since I filed this and I don't remember much since, so unfortunately I don't have any more insight at this point. These kind of widespread network events are very rare and there are no such outages recently I'm afraid. We could figure out ways to simulate them from e.g. mwdebug though, although I doubt that anyone has the time to investigate this in such depth, so I don't particularly disagree with resolving this task instead.

That said, I did a quick count (and hope I didn't mess it up), and it seems like the exception logs are filled with CAS update failed on user_touched error messages, with about ~1000/day being emitted, which currently accounts to ~30-40% of all exception log lines. I'm not sure if these are real errors or just noise, but regardless, it would be nice to get to a healthy baseline so that we can use those error messages as a meaningful signal of an operational issue. Perhaps this belongs to a separate task, though; thoughts, @Krinkle, @Imarlier et al?

The CAS error is basically edit conflict for user changes (probably the login token as the other fields don't change much); some of that happening is normal. 1000/day is more than I'd have expected but still not that much for a hundred thousand users.

There is no reason to log it as an exception though, that was probably only done as a pre-structured-logging way of getting a stack trace. -> https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/446593

In T143367#4434374, @Tgr wrote:

-> https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/446593

I note that patch won't fix the "CAS update failed on user_touched" message faidon mentioned, only the rather few "CAS update failed on gu_cas_token" messages.