plan:

• create restbase user on labs/staging/production clusters
• grant permissions accordingly
• switch user/password in puppet for restbase
• roll-restart restbase

To summarize a discussion with @fgiunchedi:

We will have puppet:

• Write a cqlshrc file to /etc/cassandra that includes the superuser username and password (allowing puppet to invoke cqlsh as user 'cassandra' with superuser privileges)
• Write a cqlsh script to /etc/cassandra, that includes the CREATE USER <user> IF NOT EXISTS... statement, along with the relevant GRANT statements.
• Invoke cqlsh --cqlshrc=/etc/cassandra/cqlshrc -f /etc/cassandra/<script>.cql <hostname>
  • Optional: Invoke the above only if the user does not already exist

This will not provide for a way to change the password, but since that is racey anway, the idea is to never change a password, to only ever add new users (configuring RESTBase to use them in a second pass).

Change 239000 had a related patch set uploaded (by Eevans):
create application users (WIP)

https://gerrit.wikimedia.org/r/239000

Change 239000 merged by Filippo Giunchedi:
create application users

https://gerrit.wikimedia.org/r/239000

Change 239341 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: provision restbase user/password

https://gerrit.wikimedia.org/r/239341

Change 239341 merged by Filippo Giunchedi:
cassandra: provision restbase user/password

https://gerrit.wikimedia.org/r/239341

Change 239398 had a related patch set uploaded (by Filippo Giunchedi):
restbase: add cassandra password for test cluster

https://gerrit.wikimedia.org/r/239398

Change 239398 merged by Filippo Giunchedi:
restbase: add cassandra password for test cluster

https://gerrit.wikimedia.org/r/239398

Change 239793 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: provision restbase user

https://gerrit.wikimedia.org/r/239793

Change 239793 merged by Filippo Giunchedi:
cassandra: provision restbase user

https://gerrit.wikimedia.org/r/239793

Change 239829 had a related patch set uploaded (by Filippo Giunchedi):
restbase: set cassandra credentials

https://gerrit.wikimedia.org/r/239829

Change 239829 merged by Filippo Giunchedi:
restbase: set cassandra credentials

https://gerrit.wikimedia.org/r/239829

The dedicated RESTBase user is now deployed, so a quick test just to verify that the previous issues we saw with QUORUM auth reads are solved:

From xenon, authentication as the default user succeeds; Total replication is 6 (all nodes).

eevans@xenon:~$ cqlsh -u cassandra `hostname -i`
Password: 
Connected to services-test at 10.64.0.200:9042.
[cqlsh 5.0.1 | Cassandra 2.1.8 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cassandra@cqlsh> 
cassandra@cqlsh> 
cassandra@cqlsh> 
cassandra@cqlsh> describe keyspace system_auth

CREATE KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', 'codfw': '3', 'eqiad': '3'}  AND durable_writes = true;
...

We then stop Cassandra on restbase-test200[1-3] and attempt to log in as user Cassandra, (QUORUM is 4 nodes, and there are only 3 available at this point):

eevans@xenon:~$ cqlsh -u cassandra `hostname -i`
Password: 
Connection error: ('Unable to connect to any servers', {'10.64.0.200': AuthenticationFailed(u'Failed to authenticate to 10.64.0.200: code=0100 [Bad credentials] message="org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM"',)})

Which fails (as expected), but as the restbase user...

eevans@xenon:~$ cqlsh -u restbase `hostname -i`
Password: 
Connected to services-test at 10.64.0.200:9042.
[cqlsh 5.0.1 | Cassandra 2.1.8 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
restbase@cqlsh> 
eevans@xenon:~$

TL;DR

Looks good.

The relevant prod patch has been merged as well. Should we close this then?

Created T113622: replace default Cassandra superuser, to track the remaining issue.