Page MenuHomePhabricator

Create BagOStuff subclass for HTTP
Closed, ResolvedPublic

Description

This can be used for cassandra session storage (with a different cluster though). Cross-DC traffic will need to use TLS/SSL.

See also

Event Timeline

aaron created this task.Jun 8 2016, 6:52 AM
GWicke renamed this task from Create restbase BagOStuff subclass to Create restbase BagOStuff subclass (session storage).Jun 8 2016, 4:08 PM
GWicke updated the task description. (Show Details)
Smalyshev claimed this task.Jun 8 2016, 4:11 PM
GWicke added a subscriber: GWicke.EditedJun 8 2016, 11:32 PM

@Smalyshev & I just did a bit of brainstorming. We mostly went through the requirements for session storage, and then looked at how we could isolate access to sessions by controlling access with a firewalled service. Notes:

Requirements

  • TTLs
  • A good amount of overwriting (but, with optimizations, still a read-biased workload)
  • Key-value; value size relatively (?) small

Questions:

  • Request volume: each authenticated page request - volume?
    • Order of magnitude: thousands req/s
    • Write volume, with optimizations relatively small fraction of that (< 1/10?)
  • Blob value size: mean / max
  • Default session TTL?
  • Overall size: Check Redis memory size

Action items:

ori added a subscriber: ori.Jun 8 2016, 11:40 PM
  • Overall size: Check Redis memory size

Total: 1405 Mb in use, 9000 Mb max (500 Mb x 18 servers).

ori added a comment.Jun 8 2016, 11:51 PM
  • Request volume: each authenticated page request - volume?

Crude estimates for determining orders of magnitude -- I ran:

redis-cli -a "$(sudo grep -Po '(?<=requirepass ).*' /etc/redis/tcp_6379.conf)" monitor | \
  grep -P 'GET.*MWSession' | \
  pv --line-mode --average-rate >/dev/null

The rate was about 490/s on one server, so roughly 9,000/s total.

Using the same technique to estimate the rate of SETs: 4/s per server, 72/s total.

Thank you, @ori! This information is very helpful.

I checked the session size on my local vagrant install, it's 780 bytes, so not too big. Of course, productions sessions may be bigger but doesn't look like it's huge.

aaron moved this task from Inbox to Radar on the Performance-Team board.Jun 9 2016, 8:55 PM

Change 293554 had a related patch set uploaded (by Smalyshev):
Create BagOStuff implementation to talk to RestBase

https://gerrit.wikimedia.org/r/293554

A basic session storage service prototype using hyperswitch is available at https://github.com/gwicke/authoid. As a baseline, read throughput on a laptop is about 3k req/s.

Change 293554 merged by jenkins-bot:
Create BagOStuff implementation to talk to RestBase

https://gerrit.wikimedia.org/r/293554

Smalyshev closed this task as Resolved.Jul 6 2016, 6:36 PM
Krinkle renamed this task from Create restbase BagOStuff subclass (session storage) to Create BagOStuff subclass for HTTP.Jul 20 2016, 4:40 PM

BTW, right now we have deleteObjectsExpiringBefore implemented as stub. Should we implement some operation for it in REST API and RESTBagOfStuff?

@GWicke, @aaron - any ideas how that should work?

@Smalyshev, if we use the Cassandra backend we'll get fixed TTL expiry for free. Lets hold off on the interface until we know whether we need to implement explicit expiry, or not.