Page MenuHomePhabricator

Provision anonymous session storage
Open, MediumPublic

Description

The work necessary to separate storage of anonymous sessions is complete (T402853), and a new instance of Kask¹ will be used (see: T402850).

Strawperson name: sessionstore-anon(?)


1: Using the same (sessionstore) Cassandra cluster.

Event Timeline

Eevans updated the task description. (Show Details)
Eevans triaged this task as Medium priority.Oct 31 2025, 6:18 PM
Eevans updated the task description. (Show Details)

@Tgr what portion of the overall workload is anon? Is there a dashboard for this?

The sessionstore is used for MediaWiki core sessions and CentralAuth sessions (which are never anonymous). Those can be differentiated by key group on the BagOStuff Prometheus dashboard:

coreCentralAuth
GET/s (hits)1.5K200
GET/s (misses)4060
GET MB/s2.50.06
POST/s50050
POST MB/s10.01
DELETE/s25-

(This is based on last week's data.)

For core sessions, the SessionStore Prometheus metrics differentiate between anonymous and authenticated operations:

anonymousauthenticated
GET/s1501.35K
POST/s300200
DELETE/s208

Sanity checks:

  • The session writes Logstash dashboard has 370M writes for the last week (it's sampled 1:1000 so you have to add three magnitudes to all its numbers), out of which 235M are anonymous. Matches the 60% ratio from Prometheus; 500/s * 3600 * 24 * 7 = 300M / week, so it pencils out (there's a ~20% discrepancy but I'm just eyeballing the charts so that much is normal).
  • The sessionstore service Prometheus metrics (which do not differentiate between core and CentralAuth sessions) have ~1.75K GET/s, 600 POST/s and 25 DELETE/s, which, again, close enough for eyeballing.

Bot traffic, which is the unpredictable part of our session load, is anonymous. SessionStore metrics don't go back for very long, but we can check the session writes dashboard for the last 3 months (the retention limit for Logstash) for anon writes: the largest daily sum in that range is about 4x the recent ones. The BagOStuff dashboard shows 1.5K POST/s and 200 DELETE/s as the peak value for the last year, so a 3x and 8x increase respectively. (The POST increases look like bot spikes. The DELETE increase was one time and I think it might have been the mass credentials stuffing attack, since DELETE would only happen during a successful login?) The Graphite data for BagOStuff, available between 2021-02 and 2025-04, also shows a max 4x difference between normal and peak POST load, with little variation in GETs (granted Graphite was disabled just around the time when session loads started to go up). Given that half of writes are anonymous, and the authenticated writes probably don't fluctuate much, that means an 5-7x increase of anon writes.

GETs are pretty stable over time. You can see the credentials stuffing attack (I think?) and the time during which NetworkSession was enabled, but those are <2x increases, and other than those, the variation is <30%. We have no way to estimate historical anonymous reads (we didn't differentate between anon and authenticated reads until maybe a month ago, and anon reads are such a small part of total reads that the number of total reads being stable doesn't really mean much) but it's reasonable to assume that they scale linearly with writes.

So, if you include the CentralAuth writes, roughly 10% of all reads and 50% of all writes are anonymous, and the separate service for anonymous core sessions will get reads and writes in the range of hundreds per second, and deletes in the range of tens per second, with maybe a 10x temporary increase expected during bot spikes.

Screenshot Capture - 2025-11-02 - 18-12-43.png (530×2 px, 139 KB)
Screenshot Capture - 2025-11-02 - 18-12-59.png (766×3 px, 299 KB)
Screenshot Capture - 2025-11-02 - 18-24-35.png (532×2 px, 133 KB)
Screenshot Capture - 2025-11-02 - 18-24-47.png (756×3 px, 195 KB)
core session reads (last year)core session writes (last year)core session reads (old Graphite data)core session writes (old Graphite data)
Screenshot Capture - 2025-11-02 - 18-29-51.png (810×2 px, 338 KB)
Screenshot Capture - 2025-11-02 - 18-30-05.png (810×2 px, 263 KB)
anonymous session operations (last week)authenticated session operations (last week)