Page MenuHomePhabricator

use non-default credentials when authenticating to Cassandra
Closed, ResolvedPublic

Description

When authentication is enabled on a new Cassandra cluster, a default (super)user and corresponding password is automatically created. RESTBase currently makes use of this default. RESTBase requires a broad set of permissions to operate, but it does not require superuser (it does not need to add or remove other users, nor grant/revoke permissions). Additionally, authentication of the default user is performed at QUORUM, where all other users (even manually created superusers) are authenticated at ONE, which could cause serious problems when we are deployed across multiple datacenters.

I've added some documentation to https://wikitech.wikimedia.org/wiki/Cassandra#Authentication for what I think should be done to the existing (and any future clusters).

TL;DR

We need to replace the default cassandra superuser with one of our own, and create an application user for RESTBase.

Event Timeline

Eevans created this task.Mar 13 2015, 1:08 AM
Eevans claimed this task.
Eevans raised the priority of this task from to Medium.
Eevans updated the task description. (Show Details)
Eevans added a subscriber: Eevans.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2015, 1:08 AM
GWicke moved this task from Backlog to In progress on the RESTBase board.Mar 17 2015, 8:16 PM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptJun 29 2015, 5:26 PM

plan:

  • create restbase user on labs/staging/production clusters
  • grant permissions accordingly
  • switch user/password in puppet for restbase
  • roll-restart restbase

To summarize a discussion with @fgiunchedi:

We will have puppet:

  • Write a cqlshrc file to /etc/cassandra that includes the superuser username and password (allowing puppet to invoke cqlsh as user 'cassandra' with superuser privileges)
  • Write a cqlsh script to /etc/cassandra, that includes the CREATE USER <user> IF NOT EXISTS... statement, along with the relevant GRANT statements.
  • Invoke cqlsh --cqlshrc=/etc/cassandra/cqlshrc -f /etc/cassandra/<script>.cql <hostname>
    • Optional: Invoke the above only if the user does not already exist

This will not provide for a way to change the password, but since that is racey anway, the idea is to never change a password, to only ever add new users (configuring RESTBase to use them in a second pass).

Change 239000 had a related patch set uploaded (by Eevans):
create application users (WIP)

https://gerrit.wikimedia.org/r/239000

Eevans updated the task description. (Show Details)Sep 17 2015, 7:19 PM
Eevans set Security to None.

Change 239000 merged by Filippo Giunchedi:
create application users

https://gerrit.wikimedia.org/r/239000

Change 239341 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: provision restbase user/password

https://gerrit.wikimedia.org/r/239341

Change 239341 merged by Filippo Giunchedi:
cassandra: provision restbase user/password

https://gerrit.wikimedia.org/r/239341

Change 239398 had a related patch set uploaded (by Filippo Giunchedi):
restbase: add cassandra password for test cluster

https://gerrit.wikimedia.org/r/239398

Change 239398 merged by Filippo Giunchedi:
restbase: add cassandra password for test cluster

https://gerrit.wikimedia.org/r/239398

Change 239793 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: provision restbase user

https://gerrit.wikimedia.org/r/239793

Change 239793 merged by Filippo Giunchedi:
cassandra: provision restbase user

https://gerrit.wikimedia.org/r/239793

Change 239829 had a related patch set uploaded (by Filippo Giunchedi):
restbase: set cassandra credentials

https://gerrit.wikimedia.org/r/239829

Change 239829 merged by Filippo Giunchedi:
restbase: set cassandra credentials

https://gerrit.wikimedia.org/r/239829

The dedicated RESTBase user is now deployed, so a quick test just to verify that the previous issues we saw with QUORUM auth reads are solved:

From xenon, authentication as the default user succeeds; Total replication is 6 (all nodes).

eevans@xenon:~$ cqlsh -u cassandra `hostname -i`
Password: 
Connected to services-test at 10.64.0.200:9042.
[cqlsh 5.0.1 | Cassandra 2.1.8 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra@cqlsh> 
cassandra@cqlsh> 
cassandra@cqlsh> 
cassandra@cqlsh> describe keyspace system_auth

CREATE KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', 'codfw': '3', 'eqiad': '3'}  AND durable_writes = true;
...

We then stop Cassandra on restbase-test200[1-3] and attempt to log in as user Cassandra, (QUORUM is 4 nodes, and there are only 3 available at this point):

eevans@xenon:~$ cqlsh -u cassandra `hostname -i`
Password: 
Connection error: ('Unable to connect to any servers', {'10.64.0.200': AuthenticationFailed(u'Failed to authenticate to 10.64.0.200: code=0100 [Bad credentials] message="org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM"',)})

Which fails (as expected), but as the restbase user...

eevans@xenon:~$ cqlsh -u restbase `hostname -i`
Password: 
Connected to services-test at 10.64.0.200:9042.
[cqlsh 5.0.1 | Cassandra 2.1.8 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
restbase@cqlsh> 
eevans@xenon:~$

TL;DR

Looks good.

The relevant prod patch has been merged as well. Should we close this then?

mobrovac removed a subscriber: gerritbot.
fgiunchedi closed this task as Resolved.Sep 24 2015, 8:55 AM

yep, this has been deployed everywhere now