Page MenuHomePhabricator

Evaluate options for non-root operations with cumin and spicerack cookbooks
Open, MediumPublic

Description

In order to reduce the requirement of being root to be able to run cumin or spicerack cookbooks as part of this Q OKRs, this task is focused on the auditing part of the existing requirements and then on the proposal of possible improvements.

Current Requirements

Cumin

Some preconditions:

  • To execute commands we need a local user, so we we need to tweak the admin module to create a user without deploying an SSH key

We can however use group-based matching in the OpenSSH config to control _who_ is allowed to log into a server via Kerberos The users which are locally created by the admin module can be added to a local Unix group like "krblogin". Users in that group can then be matched and redirected to a ForceCommand execution, e.g.

Match Group krblogin
     ForceCommand /usr/bin/cumin-login-wrapper

The $COMMAND passed by the user is accessible by the cumin-login-wrapper as $SSH_ORIGINAL_COMMAND. The wrapper can perform a check whether a user
is allowed to execute the given command and do so if permitted.

This allows for a setup where people SSH to the Cumin hosts and then run e.g.

cumin-nonpriv install1002.wikimedia.org 'sudo systemctl restart dhcp'

This would internally execute a SSH login using Kerberos running "ssh -K $DESTINATIONHOST $COMMAND"

(The use of the wrapper isn't strictly necessary, but still seems useful to allow full logs of executed commands, better error reporting to the users (e.g. in case a ticket has expired) and it provides a method to restrict kerberized SSH logins to a whitelisted list of commands (which we can generate from a common Puppet structure along with the sudo rules) and restricting a user from logging into the host itself.)

Spicerack

Read access
  • /etc/spicerack
  • /srv/deployment/spicerack
  • /etc/phabricator_ops-monitoring-bot.conf
  • /etc/cumin/
  • /etc/conftool/
  • /etc/debmonitor.conf
  • /root/.ssh/new_install # SSH key to ssh into the debian-installer
Write access
  • /var/log/spicerack
Accounts/network access
  • AuthDNS
    • A:dns-auth hosts on port 5353
  • Ganeti
    • Ganeti RO API account (ro_user user)
    • ganeti01.svc.$DC.wmnet:5080
  • Debmonitor
    • Debmonitor RW on all hosts access (implicit for spicerack being run from a proxy host, hard to make RO only for some)
    • debmonitor.discovery.wmnet:443
  • Netbox
    • Netbox RW API account (sre_bot user)
  • Phabricator
    • Phabricator RW API account, no special privileges (ops-monitoring-bot user)
  • Logging (tcpircbot for !log)
    • icinga.wikimedia.org:9200
  • Conftool (etcd)
    • etcd RW account (root user)
    • conf*.$DC.wmnet:4001 # SRV targets for _etcd._tcp.conftool.$DC.wmnet
  • Prometheus
    • prometheus.svc.$DC.wmnet/ops/api/v1/query
  • Elasticsearch
    • search.svc.$DC.wmnet:{9243,9443,9643} # Search clusters elasticsearch API
    • relforge*.$DC.wmnet:{924,9443} # Relforge elasticsearch API
  • Redis (to be removed soon?)
    • Redis password
    • mc*.$DC.wmnet:6379 # Redis clusters
  • Selected cookbooks (reimage/decommission/ipmi-password-reset)
    • management password

Event Timeline

Volans triaged this task as Medium priority.Feb 11 2020, 9:43 AM
Volans created this task.
Possible improvements
  • Netbox
    • Have 2 config files in /etc/spicerack/netbox/, one RW and one RO with different permissions.
    • Change Spicerack.netbox() to accept a write=False param and load the appropriate file based on that.
    • Update the cookbooks that require to write on netbox to use write=True.
  • Various spicerack modules that use Remote (cumin) and require root might self-detect if they are running with less privileges and bail out at instantiation time instead of failing in the middle of the execution.
  • conftool/etcd: implement T97972 and adapt spicerack accordingly, possibly having multiple config files with different permissions like the above proposal for Netbox.

Change 664780 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Initial stub role for rootless Cumin

https://gerrit.wikimedia.org/r/664780

Change 664780 merged by Muehlenhoff:
[operations/puppet@production] Initial stub role for rootless Cumin

https://gerrit.wikimedia.org/r/664780

Change 664813 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] profile::base::cuminunpriv: Allow SSH access from unprivileged Cumin masters

https://gerrit.wikimedia.org/r/664813

Change 664812 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add unprivileged Cumin master(s) to network constants

https://gerrit.wikimedia.org/r/664812

Change 664812 merged by Muehlenhoff:
[operations/puppet@production] Add unprivileged Cumin master(s) to Hiera

https://gerrit.wikimedia.org/r/664812

Change 664813 merged by Muehlenhoff:
[operations/puppet@production] profile::base::cuminunpriv: Allow SSH access from unprivileged Cumin masters

https://gerrit.wikimedia.org/r/664813

Change 666313 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Make GSSAPI configurable in sshd config and enable for sretest1001

https://gerrit.wikimedia.org/r/666313

Change 666313 merged by Muehlenhoff:
[operations/puppet@production] Make GSSAPI configurable in sshd config and enable for sretest1001

https://gerrit.wikimedia.org/r/666313

Change 666347 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Set a symlink for the host keytab

https://gerrit.wikimedia.org/r/666347

Change 666349 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Drop host keytab from the cuminunpriv profile

https://gerrit.wikimedia.org/r/666349

Change 666349 merged by Muehlenhoff:
[operations/puppet@production] Drop host keytab from the cuminunpriv profile

https://gerrit.wikimedia.org/r/666349

Change 666347 merged by Muehlenhoff:
[operations/puppet@production] Set a symlink for the host keytab

https://gerrit.wikimedia.org/r/666347

Change 666609 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Initial Cumin parts for the unprivileged Cumin master profile

https://gerrit.wikimedia.org/r/666609

Change 666609 merged by Muehlenhoff:
[operations/puppet@production] Initial Cumin parts for the unprivileged Cumin master profile

https://gerrit.wikimedia.org/r/666609

Change 666672 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] puppetdb: Also allow access for unprivileged Cumin masters

https://gerrit.wikimedia.org/r/666672

Change 666845 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] profile::cumin::unprivmaster: Log to user's home

https://gerrit.wikimedia.org/r/666845

Change 666855 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/cumin@master] cli: expand user's home directory for logging

https://gerrit.wikimedia.org/r/666855

Change 666855 merged by jenkins-bot:
[operations/software/cumin@master] cli: expand user's home directory for logging

https://gerrit.wikimedia.org/r/666855

Change 666845 merged by Muehlenhoff:
[operations/puppet@production] profile::cumin::unprivmaster: Log to user's home

https://gerrit.wikimedia.org/r/666845

Change 666672 abandoned by Muehlenhoff:
[operations/puppet@production] puppetdb: Also allow access for unprivileged Cumin masters

Reason:
puppetdb will be queried through a proxy which sanitises the results

https://gerrit.wikimedia.org/r/666672

Change 667549 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] puppetdb microservice: refactor prior to expand it

https://gerrit.wikimedia.org/r/667549

Change 667550 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] puppetdb microservice: add support for cumin

https://gerrit.wikimedia.org/r/667550

Change 667549 merged by Volans:
[operations/puppet@production] puppetdb microservice: refactor prior to expand it

https://gerrit.wikimedia.org/r/667549

Change 667550 merged by Volans:
[operations/puppet@production] puppetdb microservice: add support for cumin

https://gerrit.wikimedia.org/r/667550

Change 668017 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] puppetdb microservice: fix API paths

https://gerrit.wikimedia.org/r/668017

Change 668017 merged by Volans:
[operations/puppet@production] puppetdb microservice: fix API paths

https://gerrit.wikimedia.org/r/668017

Cumin has been adapted to be usable for non-privileged users with Kerberos (sans a final patch for the logging config to land in the next Cumin release):

  1. There is a new unprivileged Cumin master cuminunpriv1001.eqiad.wmnet using the new role cluster::unprivmanagement
  2. There's a new profile::base::cuminunpriv (currently applied to sretest1001.eqiad.wmnet) which enables kerberised access:
jmm@cuminunpriv1001:~$ whoami
jmm
jmm@cuminunpriv1001:~$ kinit
Password for jmm@WIKIMEDIA:
jmm@cuminunpriv1001:~$ cumin A:sretestunpriv 'uname -a'
1 hosts will be targeted:
sretest1001.eqiad.wmnet
Confirm to continue [y/n]? y
----- OUTPUT of 'uname -a' -----
Linux sretest1001 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux
================
PASS |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00,  1.83hosts/s]
FAIL |                                                                                                                                                |   0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'uname -a'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.