Page MenuHomePhabricator

Consider REST with SSL (HyperSwitch/Cassandra) for session storage
Closed, ResolvedPublic

Description

User sessions and chronology protector positions are stored in Redis now.

For ChronologyProtector (and possibly the sessions in edge cases), we want writes to happen in all DCs at once. BagOStuff::set() already supports this concept, but we don't respect it in our current Redis setup (which relies on Redis replication). We also can't handle the case when a single Redis server goes down in a DC (since get/set would re-hash differently in each DC).

T111575 has a patch to mitigate the sync write and sever failure cases. One limitation to that approach is that it only tries writes once instead of being eventually consistent. For sessions and ChronologyProtector, this is OK (since they regenerate), but not ideal. It also requires encryption between redis <=> apaches for cross DC writes. Redis does not support this, so hacks would be needed.

Cassandra could handle all of these problems. For example, the php-driver supports SSL per http://datastax.github.io/php-driver/features/ssl_encryption/ . Another library is https://github.com/duoshuo/php-cassandra . https://gerrit.wikimedia.org/r/#/c/238370/ could be changed to use one of these.

Probably a tiny cluster with SSDs could be used, since the use case is different (high read, latency sensitive, low writes) than any of the others, and should be isolated a bit due to the nature of sessions.


Since the creation of this task the plan has evolved to not use Cassandra from MediaWiki PHP directly, but instead communicate with Cassandra via a HyperSwitch instance (HyperSwitch is the library behind Wikimedia RESTBase).

Event Timeline

aaron created this task.May 9 2016, 8:39 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptMay 9 2016, 8:39 PM

I think we could accommodate that as part of the existing RESTBase install, as I imagine we are talking about a rather limited amount of data. We might also get away with MW not needing a php Cassandra driver if we set up appropriate routes in RESTBase. Because we are talking about sensitive data which definitely shouldn't be exposed to external clients, we can set up a fake domain in RESTBase, which would allow MW to reference it via VRS while at the same time not exposing it to the public.

Joe added a subscriber: Joe.
Joe added a comment.May 12 2016, 5:53 AM

I would like to keep session management in a pretty confined environment - I would frankly avoid using the cassandra cluster we use for restbase for it.

I would like to keep session management in a pretty confined environment - I would frankly avoid using the cassandra cluster we use for restbase for it.

Makes sense. I would prefer to do so too, but proposed it as a quick'n'dirty work-around. I do agree, though, that keeping such information in Cassandra greatly simplifies session storage when cross- and multi-DC concepts come into play.

Krinkle updated the task description. (Show Details)May 12 2016, 5:03 PM
aaron renamed this task from Consider cassandra for session storage (and SSL) to Consider cassandra for session storage (with SSL).May 14 2016, 12:57 PM

Regarding the drivers - php-driver is a PHP extension, so we'd need to port it to HHVM if we want to use it in production. I'll check how php-cassandra works, it may be easier.

aaron added a comment.Jun 8 2016, 4:50 PM

If we go with restbase, then the subclass will just use MultiHttpClient. It could either talk to a local http restbase endpoint (which itself uses TLS) or the client can also use https for good measure (if that is supported). The main thing I care about is that no unencrypted traffic goes across DCs.

aaron renamed this task from Consider cassandra for session storage (with SSL) to Consider restbase/cassandra for session storage (with SSL).Jun 9 2016, 10:36 PM
Krinkle renamed this task from Consider restbase/cassandra for session storage (with SSL) to Consider REST with SSL (HyperSwitch/Cassandra) for session storage.Jul 20 2016, 4:25 PM
Krinkle updated the task description. (Show Details)
elukey added a subscriber: elukey.Feb 6 2017, 9:47 AM
aaron lowered the priority of this task from Normal to Low.May 26 2017, 11:40 PM
Joe added a comment.May 29 2017, 6:56 AM

I've seen there hasn't been much going on on this task, but I want to have the opportunity to say that I don't think it's a good idea to put a software created for other purposes (HS) to serve our sessions.

Specifically, we want to create a dedicated service that is simple and specialized enough to be able to add features like (first things off the top of my head) brute-force-attack detection or token generation.

I've seen there hasn't been much going on on this task, but I want to have the opportunity to say that I don't think it's a good idea to put a software created for other purposes (HS) to serve our sessions.
Specifically, we want to create a dedicated service that is simple and specialized enough to be able to add features like (first things off the top of my head) brute-force-attack detection or token generation.

HyperSwitch is a generic REST API router library with added candy like pluggable (route) filters and a request template compiler. It itself does not expose any functionality, so it seems like a perfect fit for what we want here, given that the service's API will be relatively simple (get/set), leaving us room to focus on the actual issues that you mentioned.

Joe added a comment.May 29 2017, 2:55 PM

I've seen there hasn't been much going on on this task, but I want to have the opportunity to say that I don't think it's a good idea to put a software created for other purposes (HS) to serve our sessions.
Specifically, we want to create a dedicated service that is simple and specialized enough to be able to add features like (first things off the top of my head) brute-force-attack detection or token generation.

HyperSwitch is a generic REST API router library with added candy like pluggable (route) filters and a request template compiler. It itself does not expose any functionality, so it seems like a perfect fit for what we want here, given that the service's API will be relatively simple (get/set), leaving us room to focus on the actual issues that you mentioned.

Oh, sorry, I must have misread: what the task means is that we'll use the HS library while writing a new service? That makes much more sense, thanks :)

GWicke moved this task from Backlog to blocked on the Services board.Jul 12 2017, 5:20 PM
GWicke edited projects, added Services (blocked); removed Services.
aaron removed aaron as the assignee of this task.Oct 5 2017, 1:12 AM

@tstarling It's our (Perf team's) impression that we wouldn't be the ones taking this on. Does that seem right to you?

Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.

@tstarling It's our (Perf team's) impression that we wouldn't be the ones taking this on. Does that seem right to you?

We will tackle this in the second half of FY18/19 as part of the platform evolution CDP.

mobrovac closed this task as Resolved.Feb 27 2019, 11:35 PM
mobrovac claimed this task.

The session management service is being currently developed. The work is tracked in T206016: Create a service for session storage and related tickets. Resolving this one, as this is (almost) the route we went with :P