Page MenuHomePhabricator

enable authenticated access to Cassandra JMX
Open, NormalPublic

Description

Ideally all Cassandra JMX client connections would require authentication, and unfortunately, Cassandra does not enable this in its default configuration.

Event Timeline

Eevans created this task.Mar 12 2015, 1:32 AM
Eevans claimed this task.
Eevans raised the priority of this task from to Normal.
Eevans updated the task description. (Show Details)
Eevans added a subscriber: Eevans.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 12 2015, 1:32 AM

Change 196133 had a related patch set uploaded (by Eevans):
enable authenticated access to Cassandra JMX

https://gerrit.wikimedia.org/r/196133

GWicke added a subscriber: GWicke.EditedMar 15 2015, 5:34 PM

I'm a bit worried that with a password the usage of nodetool will become quite complicated, even if only accessing localhost. Setting a password per user in a .nodetoolrc or even passing it in as a commandline parameter both don't sound very secure or practical.

I think managing this at the network level might result in better security & usability. I see two main options:

  1. Lock down access to the JMX port to cluster nodes only. Main advantage is that this would still allow us to run nodetool against other cluster nodes, for example to coordinate repairs.
  1. Lock down access to the JMX port to localhost and possibly a specific group only. This would be both very secure and practical. The main downside is that we'd have to orchestrate repairs with SSH or salt, possibly from a central admin node.

See also T92680: iptables firewall to limit access to Cassandra services for a bigger-picture discussion.

I think managing this at the network level might result in better security & usability. I see two main options:
[ ... ]
See also T92680: securing the RESTBase Cassandra cluster for a bigger-picture discussion.

Yeah, T92680 was submitted in the spirit of a simpler alternative to this problem, (and several other (as of yet )unfiled issues). Are we settled on that approach then, and if so, should I close this ticket (and abandon the corresponding gerrit review)?

GWicke moved this task from Backlog to In progress on the RESTBase board.Mar 17 2015, 8:16 PM
Eevans added a comment.Apr 1 2015, 2:16 PM

For additional reference: http://article.gmane.org/gmane.comp.db.cassandra.user/45442

This advisory only applies to us in the sense that it underlines the need to secure RMI (remote execution of code). We still want the ability to issue JMX requests to all of the nodes in the cluster, from any of the nodes in the cluster (for example, to coordinate repair jobs). We need it exposed, only in a secure fashion.

GWicke added a comment.EditedApr 1 2015, 7:02 PM

The list of changes in Cassandra 2.1.4:

2.1.4
 * If no JMX flags are set start a localhost only JMX service

Here's the diff & the bug.

Lets upgrade?

Eevans added a comment.Apr 1 2015, 7:22 PM

If we are now eschewing the approach discussed in https://phabricator.wikimedia.org/T92355, then sure.

Eevans added a comment.Apr 1 2015, 7:38 PM

In the interest of full disclosure:

Having this bound to localhost still exposes a locally exploitable vulnerability; If left unauthenticated, any local user will be able to execute arbitrary code as the cassandra user. If that is deemed acceptable, and if it is preferable to coordinate nodetool invocations like the one proposed in T92355 using SSH instead, then using the localhost-only listener introduced in 2.1.4 should be fine.

Dzahn added a subscriber: Dzahn.Apr 1 2015, 8:12 PM

We should probably treat this as an AND rather than an OR and have both, network level separation and also use a password.

Some additional information:

The new local-only option introduced in 2.1.4 does not support authentication, or encryption. So the choices are: unauthenticated, unencrypted, local-only access (if we upgrade to 2.1.4), or, remotely accessible with or without any of encrypted and authenticated.

Setting up password authentication is relatively straightforward, TLS a little less so, (and they would needed to be created by ops, and stored in the private puppet repo).

And of course, as @GWicke says, once in place this will likely make casual use of nodetool a pain. I think the best we could do here is install a wrapper script (say nt) that either a) uses a per-user password file ~/.nodetool, or b) uses the same jmxremote.passwords file that Cassandra does (/etc/cassandra/jmxremote.passwords), and set group permissions on that file accordingly.

TL;DR

Easy/awesome, or totally secure?

sharing the passwords file with cassandra seems easy enough, instead of group permissions I'd go for limited sudo nodetool as user cassandra for said unix group. reason being that we get free auditing and consistency across invocations (e.g. if nodetool would create files on the local fs owner would vary). what do you think @csteipp @MoritzMuehlenhoff ?

Running nodetool through sudo also seems preferable to me.

The upstream documentation https://wiki.apache.org/cassandra/JmxSecurity suggests passing "-pw" to nodetool, but I suppose we should rather use "-pwf", otherwise we'd leak the password into the process environment (unless hidepid as used when mounting procfs)

What Moritz said sounds sane.

We can enforce localhost-only access from a specific group or root with iptables. @Dzahn already looked into that, and it seems that ferm supports it quite well.

We can enforce localhost-only access from a specific group or root with iptables. @Dzahn already looked into that, and it seems that ferm supports it quite well.

I think that'd be less obvious (and less secure) than e.g. calling this via sudo -u cassandra cassandra-nodetool

$ cat cassandra-nodetool 
#!/bin/sh
set -e
set -u

password_file=${PASSWORD_FILE:-/etc/cassandra/jmxremote.passwords}
wrap_args="--password-file $password_file"

case "$@" in
  -pwf*|--password-file*)
    wrap_args=""
  ;;
esac

exec nodetool $wrap_args "$@"

We can enforce localhost-only access from a specific group or root with iptables. @Dzahn already looked into that, and it seems that ferm supports it quite well.

I think that'd be less obvious (and less secure) than e.g. calling this via sudo -u cassandra cassandra-nodetool

$ cat cassandra-nodetool 
#!/bin/sh
set -e
set -u
password_file=${PASSWORD_FILE:-/etc/cassandra/jmxremote.passwords}
wrap_args="--password-file $password_file"
case "$@" in
  -pwf*|--password-file*)
    wrap_args=""
  ;;
esac
exec nodetool $wrap_args "$@"

Anyway that could be called nt, and have it invoke nodetool with sudo -u cassandra? This command is run a lot, and sudo -u cassandra cassandra-nodetool is cumbersome.

@fgiunchedi, the point is that we are making this user / group-based in any case: In one case by making the password file readable to that group / user, in the other by locking down the network connection to JMX to a user / group. The latter avoids the need for wrapper scripts and password files, so might be less complex.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJun 26 2015, 5:57 PM
Eevans added a project: Cassandra.
Eevans set Security to None.
Eevans moved this task from Backlog to Next on the Cassandra board.

@Eevans @fgiunchedi is there something left to be done here?

Eevans added a comment.Mar 2 2018, 5:56 PM

@Eevans @fgiunchedi is there something left to be done here?

This issue has been open for a long time, during which it has been something of a moving target, but here goes:

The status quo was supposed to be that JMX is now bound to loopback (not true when this ticket was opened, but since made possible by upstream changes), but that anyone still able to access that network port (so, anyone who could obtain a shell), would be able to execute arbitrary code as the cassandra user. I say that was supposed to be the status quo, because I went and had a look to make sure everything would be as I said it was after leaving a comment, and this no longer appears to be the case:

tcp        0      0 0.0.0.0:7189            0.0.0.0:*               LISTEN      -

I know this was working (that it was bound to 127.0.0.1). I do not know why it is not anymore.

Remote access is still blocked by ferm, but we're without that redundancy anymore; We're a fat-fingering away from a remote exploit.

Now, even if the status quo were as expected -even if this were fixed and the RMI port were bound to loopback-, there didn't seem to be a lot of certainty about what else ought to be done. I think we can scratch SSL off the list if we can ensure the agent is bound to loopback. I think consensus was that another layer of defense was warranted though, i.e. that we password protect it, even if in the end that still boils down to file permissions (we wrap the invocation of nodetool and pass a -pf argument for a file containing the credentials).

So:

  1. Figure out why the RMI port is no-longer bound to loopback, and fix it
  2. Setup a Puppet template for the credentials file, w/ corresponding secrets
  3. Modify our nodetool wrapper to use the file

Comments welcome.