Change Details

User sessions and chronology protector positions are stored in redis now. For the later (and possibly the former in some edge cases), we want writes to happen in all DCs at once. BagOStuff::set() already supports this concept, but we don't respect it in our current redis setup (which relies on redis replication). We also can't handle the case when a single redis server goes down in a DC (since get/set would re-hash differently in each DC). https://phabricator.wikimedia.org/T111575 has a patch to mitigate the sync write and sever failure cases. One limitation to that approach is that it only tries writes once instead of being eventually consistent. For sessions and ChronologyProtector, this is OK (since they regenerate), but not ideal. It also requires encryption between redis <=> apaches for cross DC writes. Redis does not support this, so hacks would be needed. Cassandra could handle all of these problems. For example, the php-driver supports SSL per http://datastax.github.io/php-driver/features/ssl_encryption/ . Another library is https://github.com/duoshuo/php-cassandra . https://gerrit.wikimedia.org/r/#/c/238370/ could be changed to use one of these. Probably a tiny cluster with SSDs could be used, since the use case is different (high read, latency sensitive, low writes) than any of the others, and should be isolated a bit due to the nature of sessions.

User sessions and chronology protector positions are stored in redis now. For the latter (and possibly the former in some edge cases), we want writes to happen in all DCs at once. BagOStuff::set() already supports this concept, but we don't respect it in our current redis setup (which relies on redis replication). We also can't handle the case when a single redis server goes down in a DC (since get/set would re-hash differently in each DC). https://phabricator.wikimedia.org/T111575 has a patch to mitigate the sync write and sever failure cases. One limitation to that approach is that it only tries writes once instead of being eventually consistent. For sessions and ChronologyProtector, this is OK (since they regenerate), but not ideal. It also requires encryption between redis <=> apaches for cross DC writes. Redis does not support this, so hacks would be needed. Cassandra could handle all of these problems. For example, the php-driver supports SSL per http://datastax.github.io/php-driver/features/ssl_encryption/ . Another library is https://github.com/duoshuo/php-cassandra . https://gerrit.wikimedia.org/r/#/c/238370/ could be changed to use one of these. Probably a tiny cluster with SSDs could be used, since the use case is different (high read, latency sensitive, low writes) than any of the others, and should be isolated a bit due to the nature of sessions.

User sessions and chronology protector positions are stored in redis now. For the latter (and possibly the former in some edge cases), we want writes to happen in all DCs at once. BagOStuff::set() already supports this concept, but we don't respect it in our current redis setup (which relies on redis replication). We also can't handle the case when a single redis server goes down in a DC (since get/set would re-hash differently in each DC). https://phabricator.wikimedia.org/T111575 has a patch to mitigate the sync write and sever failure cases. One limitation to that approach is that it only tries writes once instead of being eventually consistent. For sessions and ChronologyProtector, this is OK (since they regenerate), but not ideal. It also requires encryption between redis <=> apaches for cross DC writes. Redis does not support this, so hacks would be needed. Cassandra could handle all of these problems. For example, the php-driver supports SSL per http://datastax.github.io/php-driver/features/ssl_encryption/ . Another library is https://github.com/duoshuo/php-cassandra . https://gerrit.wikimedia.org/r/#/c/238370/ could be changed to use one of these. Probably a tiny cluster with SSDs could be used, since the use case is different (high read, latency sensitive, low writes) than any of the others, and should be isolated a bit due to the nature of sessions.