Simplify MediaWiki session store at WMF
Open, LowPublic
Actions

Assigned To

None

Authored By

	pmiazga
	Apr 11 2024, 2:38 PM

Description

What

Simplify the stack around SessionBackend in production and encourage increasing reuse of software where possible.

Proposal 1: Move from Cassandra to Memcached (2024)

Deprecate Kask/Cassandra use for keeping MediaWiki session data.

The current proposal is to use Memcached instead.

We expect SRE will require a minimum of 3 hosts per DC for any tier 1 service. Our team don't see a requirement for a separate hardware per-se, as it could be co-located existing servers (eg main memcache even?), but this is for SRE to decide.

The memcache instances have to be redundant for data within DCs, so in case of failover/switch we do not lose users session data. Hence mcrouter would best-effort write the same data to all three hosts.

We should not reuse the main memcache cluster because it is important that we don't cause session data to get evicted by unrelated LRU pressure. MediaWiki intentionally fills up memcache, and we don't want the session data of less-active users removed periodically.

Related Objects
Search...

Status	Assigned	Task
Open	None	T362335 Simplify MediaWiki session store at WMF
Declined	None	T362331 SessionBackend: remove dependency on Kask/Cassandra
Declined	None	T362336 Update SessionBackend wmf-config to write to both stores
Open	None	T362337 Decom Kask service and RESTBagOStuff

Event Timeline

@Krinkle @Tgr did I miss anything ?

What would that mean on the MediaWiki level? Building a new cache service? None of the options under https://www.mediawiki.org/wiki/Object_cache#Services have cross-DC replication, right? (Other than the main stash, but that's SQL-based and too slow.)

There is no requirement for a separate hardware, we can add additional memcache service on existing servers.
The requirement for separate memcache servers is that we don't want the session data to get evicted.

These sound somewhat contradictory.

In T362335#9708452, @Tgr wrote:

What would that mean on the MediaWiki level? Building a new cache service? None of the options under https://www.mediawiki.org/wiki/Object_cache#Services have cross-DC replication, right? (Other than the main stash, but that's SQL-based and too slow.)

There is no requirement for a separate hardware, we can add additional memcache service on existing servers.
The requirement for separate memcache servers is that we don't want the session data to get evicted.

These sound somewhat contradictory.

Sorry, brain shortcut - what I meant was we don't need a separate metal, we can run additional memcache instances on machines that already host memcache - in other words - re-use the same metal, but add extra memcache instance on a different port to specifically handle session data.

pmiazga moved this task from Inbox, needs triage to Needs refinement on the MediaWiki-Platform-Team board.Apr 12 2024, 9:49 AM

Krinkle renamed this task from Decide where to store Session Data to Decide on MediaWiki session store at WMF (2024).Apr 12 2024, 2:53 PM

Krinkle triaged this task as Medium priority.

Krinkle updated the task description. (Show Details)

Krinkle renamed this task from Decide on MediaWiki session store at WMF (2024) to Simplify MediaWiki session store at WMF.Mar 10 2025, 9:59 PM

Krinkle removed a parent task: T362331: SessionBackend: remove dependency on Kask/Cassandra.

Krinkle added a subtask: T362331: SessionBackend: remove dependency on Kask/Cassandra.

Krinkle added subtasks: T362336: Update SessionBackend wmf-config to write to both stores, T362337: Decom Kask service and RESTBagOStuff.

Krinkle updated the task description. (Show Details)

In T362331#10023547, @Joe wrote:

If we start [using Memched for sessions], we either:

a) Make the lives of SREs hell because they can't restart memcached without a lot of pre-configuring and syncing, and this would still not guarantee data consistency
b) Cause unexpected behaviour, race conditions and all kinds of UX problems

[…]

For what its worth, I did consider operational flexibility, but perhaps insufficiently. From the proposal:

From the task description § Proposal 1:

We expect […] 3 hosts per DC […]. The memcache instances have to be redundant for data within DCs, so in case of failover/switch we do not lose users session data. Hence mcrouter would best-effort write the same data to all three hosts.

We should not reuse the main memcache cluster because […] unrelated LRU pressure.

These instances are separate from main memcached cluster. The longest duration of any session store key is 24 hours. This is because it's not really used as a session store. It is a mostly vacant dataset, where the keys serve to reserve that a session ID exists and is valid, with the data array being effectively empty, apart from a lazy-initialized CSRF token seed (which we could derive in other ways), and beyond that a space for short-lived temporary data to carry between two consecutive HTTP requests. The "real" session store is the user_token column in the MySQL database, i.e. this powers "Remember me" and any multi-day session.

With a time window that short, we can tolerate restarts/loss of 2/3 hosts in a given 24 hour window. The third would be fine to lose/restart 24h after that, as by then the other two will have built up sessions still valid. Those using "Remember me" would be unaffected even after a 3/3 loss.

In any event, I'm happy to withdraw this proposal given the current situation is acceptable/preferred by SRE. The intent is to reduce the need for additional expertise and operations of the rarely-used Cassandra data store (after MySQL/Memcached), for a use case that hold very little data, and imho doesn't have enough unique needs to justify it. For FY23-25, we did not prioritize this proposal. I'll decline the subtasks specific to the above proposal, and leave this open for future exploration/simplification, possibly in FY25-26.

While not in-scope of this task, I would suggest pursuing the following if we're sticking with Kask/Cassandra for sessions:

Kask service ownership. Kask is currently unassigned on mw:Maintainers. Last commits in Gerrit were by Eric (now SRE/DBA) and Giuseppe and Alex (SRE/ServiceOps). The repo appears to have moved from there to an unspecified location in GitLab (not indexed by Codesearch, not linked form mw:Maintainers, SessionStorage docs on Wikitech, or Kask docs on mediawiki.org).
Cassandra storage amplification. The current 1:100 ratio seems unsustainable (>100GB disk space required per <1G of data, per T313496#8096630)

As a future proposal, if we implement Cassandra support directly, it would:

address the issue that RESTBagOStuff is an incomplete/non-compliant BagOStuff implementation (by removing it),
address the footgun that TTL is ignored and needs to be duplicated outside MediaWiki in the Kask helmchart,
address the Kask serivce ownership question (by removing it),
make it easy to adopt the CassandraBagOStuff for MainStash or ParserCache in the future. Note that there is afaik no current demand from DBAs for this, for ParserCache after T373037: Make ParserCache more like a ring, nor MainStash after T383327: Re-architecture mainstash (x2) to allow easier maintenance .