@Smalyshev & I just did a bit of brainstorming. We mostly went through the requirements for session storage, and then looked at how we could isolate access to sessions by controlling access with a firewalled service. Notes:

Requirements

• TTLs
• A good amount of overwriting (but, with optimizations, still a read-biased workload)
• Key-value; value size relatively (?) small

Questions:

• Request volume: each authenticated page request - volume?
  • Order of magnitude: thousands req/s
  • Write volume, with optimizations relatively small fraction of that (< 1/10?)
• Blob value size: mean / max
• Default session TTL?
• Overall size: Check Redis memory size

Action items:

• Gabriel: Set up a basic REST service prototype (key-value)
• Stas: write BagOfStuff HTTP implementation
  • Check out whether VirtualRESTService-type class makes sense
  • Example: https://github.com/wikimedia/mediawiki/blob/c42f06642091e1cb1142066b0a52a69a0c7e783d/includes/libs/virtualrest/RestbaseVirtualRESTService.php
• Investigate open questions

In T137272#2366546, @GWicke wrote:

• Overall size: Check Redis memory size

Total: 1405 Mb in use, 9000 Mb max (500 Mb x 18 servers).

In T137272#2366546, @GWicke wrote:

• Request volume: each authenticated page request - volume?

Crude estimates for determining orders of magnitude -- I ran:

redis-cli -a "$(sudo grep -Po '(?<=requirepass ).*' /etc/redis/tcp_6379.conf)" monitor | \
  grep -P 'GET.*MWSession' | \
  pv --line-mode --average-rate >/dev/null

The rate was about 490/s on one server, so roughly 9,000/s total.

Using the same technique to estimate the rate of SETs: 4/s per server, 72/s total.

Thank you, @ori! This information is very helpful.

I checked the session size on my local vagrant install, it's 780 bytes, so not too big. Of course, productions sessions may be bigger but doesn't look like it's huge.

Change 293554 had a related patch set uploaded (by Smalyshev):
Create BagOStuff implementation to talk to RestBase

https://gerrit.wikimedia.org/r/293554

A basic session storage service prototype using hyperswitch is available at https://github.com/gwicke/authoid. As a baseline, read throughput on a laptop is about 3k req/s.

Change 293554 merged by jenkins-bot:
Create BagOStuff implementation to talk to RestBase

https://gerrit.wikimedia.org/r/293554

BTW, right now we have deleteObjectsExpiringBefore implemented as stub. Should we implement some operation for it in REST API and RESTBagOfStuff?

@GWicke, @aaron - any ideas how that should work?

@Smalyshev, if we use the Cassandra backend we'll get fixed TTL expiry for free. Lets hold off on the interface until we know whether we need to implement explicit expiry, or not.