As evidenced during the investigation of T211721, we don't just write session data to the sessions redis cluster, but we also write all data from anything in MediaWiki that uses MediaWikiServices::getInstance()->getMainObjectStash() to the local redis cluster.
This breaks in a multi-dc setup for a number of reasons, first and foremost that we replicate redis data from the master DC to the others, but not the other way around as redis doesn't support multi-master replication.
Status quo
The MediaWiki "Main stash" is backed in WMF production by a Redis cluster labelled "redis_sessions", and is co-located on the Memcached hardware. It is perceived as having the following feaures:
- Fairly high persistence. (It is not a cache, and the data is not recoverable in case of loss. But it is expected to lose data in main stash if under pressure under hopefully rare circumstances.) Examples of user impact:
- session - User gets logged out and loses any session-backend data (e.g. book composition).
- echo - Notifications might be wrongly marked as read or unread.
- watchlist - Reviewed edits might show up agaihn as unreviewed.
- resourceloader - Version hash churn would cause CDN and browser cache misses for a while.
- Replication. (Data is eventually consistent and readable from both DCs)
- Fast (low latency).
Note that:
- "DC-local writable" is not on this list (mainstash only requires master writes), but given WMF is not yet mulit-dc we have more or less assumed that sessions are always locally writable and we need to keep performing sessions writes locally in a multi-DC world.
- "Replication" is on the list and implemented in one direction for Redis at WMF. This would suffice if we only needed master writes, but for sessions we need local writes as well.
Future of sessions
Move Session and Echo data out of the MainStash into their own store that supports local writes and bi-di replication. This is tracked under T206016 and T222851.
Future of mainstash
To fix the behaviour of the software in a multi-dc scenario, I see the following possibilities, depending on what type of storage guarantees we want to have:
- Option 1: If we don't need data to be consistent cross-dc: After we migrate the sessions data to their own datastore, we turn off replication and leave the current Redis clusters operating separately in each DC.
- Option 2: If we need cross-dc consistency, but we don't need the data to have a guaranteed persistence: We can migrate the stash to mcrouter.
- Option 3: If we need all of the above, plus persistence: We might need to migrate that to the same (or a similar) service to the one we will use for sessions.
I would very much prefer to be able to avoid the last option, if at all possible.