The nodejs clients DCAwareRoundRobinPolicy allows client fail-over to nodes in another DC in the event no local nodes are available. Enabling this is desirable, but implicitly requires that we encrypt the client connection.
Configuring the nodejs client is fairly straightforward, but like with T108953, we will need keys and certificates generated for each node/instance, and a copy of the node/instance certificates from T108953 available in PEM format to each node/instance.
var options = { policies: { loadBalancing: new cassandra.loadBalancing.DCAwareRoundRobinPolicy() }, sslOptions: { key: '...', // string or Buffer containing the certificate in PEM format cert: '...', // string or Buffer containing the certificate in PEM format ca: '...' // an array of strings of Buffers containing trusted certs in PEM format }, }; var client = new Client(options);
http://docs.datastax.com/en/developer/nodejs-driver/3.2/api/type.ClientOptions
Update
RESTBase support for client encryption is there; Configuring it is as simple as:
table: backend: cassandra ... # See: https://nodejs.org/api/tls.html#tls_tls_createsecurecontext_options tls: key: <path> cert: <path> ca: <path> ...
Cassandra does in fact support optional client encryption, which will vastly simplify rollout.
There are a few approaches to this we could take, depending on the problem we're trying to solve:
- Encryption-only
- Optional
- Required
- Encryption + certificate-based client auth
- Optional
- Required
If all we are concerned with is the avoidance of drivers communicating in the clear (which risks exposure of the password), then (1a) would be very easy to setup, and we do not need to worry about using (real) keys/certs on the client. A trivial puppet changeset to enable client encryption in Cassandra and a rolling restart later, and we're done.
(2a) would provide additional security, but at the expense of additional complexity. cassandra-ca-manager already creates all of the key material we will need, but would need to be taught to export the key in PEM encoded format as well. For existing keys, we'd need to do this manually, commit the result to the private repo, and deploy them to the nodes.
Either of (1b), (2a), or (2b) will also require that we also come up with an answer for locally sourced cqlsh connections (straightforward I assume, but worth mentioning).
Proposed rollout strategy (eqiad)
1. Update RESTBase config for defaultConsistency of localOne
1. Disable puppet on restbase100[1-9].eqiad nodes
1. Merge cassandra.yaml config change that enables client encryption
1. Reenable puppet and restart Cassandra on 1001, 1003, and 1005, applying new encryption settings from #3.
1. Rollout RESTBase configuration change to enable client encryption
1. Restart Cassandra on 1002, 1004, 1006, and 100[7-9], (applying new encryption settings from #2).
1. Restore RESTBase config for defaultConsistency to localQuorum
The idea here being that when client encryption is enabled on 1001, 1003, and 1005, existing restbase instances that would connect to these, will fail-over to the remaining nodes of the cluster when unable to connect in the clear (1, 3, and 5 represent nodes in each of racks A, B, and C). Next, RESTBase is reconfigured to use encryption and restarted, with connections failing over as needed to 1001, 1003, and 1005 when encrypted connections to 1002, 1004, 1006, and 100[7-9] fail. Finally, the client encryption settings are applied to remaining Cassandra nodes (1002, 1004, 1006, and 100[7-9]). Setting the consistency level to localOne ensures that queries succeed during the window when only 1 node in each rack is accessible.
Alternative rollout strategy (eqiad)
1. Configure codfw nodes for client encryption
1. Update RESTBase config to set localDc to codfw
1. Rollout RESTBase configuration change to enable client encryption, (and restart RESTBase)
1. Merge cassandra.yaml config change that enables client encryption (and restart eqiad Cassandra nodes)
1. Restore RESTBase config for localDc to eqiad
See also: T108953