Page MenuHomePhabricator

Evaluate options for non-root operations with cumin and spicerack cookbooks
Open, MediumPublic

Description

In order to reduce the requirement of being root to be able to run cumin or spicerack cookbooks as part of this Q OKRs, this task is focused on the auditing part of the existing requirements and then on the proposal of possible improvements.

Current Requirements

Cumin

Some preconditions:

  • To execute commands we need a local user, so we we need to tweak the admin module to create a user without deploying an SSH key

We can however use group-based matching in the OpenSSH config to control _who_ is allowed to log into a server via Kerberos The users which are locally created by the admin module can be added to a local Unix group like "krblogin". Users in that group can then be matched and redirected to a ForceCommand execution, e.g.

Match Group krblogin
     ForceCommand /usr/bin/cumin-login-wrapper

The $COMMAND passed by the user is accessible by the cumin-login-wrapper as $SSH_ORIGINAL_COMMAND. The wrapper can perform a check whether a user
is allowed to execute the given command and do so if permitted.

This allows for a setup where people SSH to the Cumin hosts and then run e.g.

cumin-nonpriv install1002.wikimedia.org 'sudo systemctl restart dhcp'

This would internally execute a SSH login using Kerberos running "ssh -K $DESTINATIONHOST $COMMAND"

(The use of the wrapper isn't strictly necessary, but still seems useful to allow full logs of executed commands, better error reporting to the users (e.g. in case a ticket has expired) and it provides a method to restrict kerberized SSH logins to a whitelisted list of commands (which we can generate from a common Puppet structure along with the sudo rules) and restricting a user from logging into the host itself.)

Spicerack

Read access
  • /etc/spicerack
  • /srv/deployment/spicerack
  • /etc/phabricator_ops-monitoring-bot.conf
  • /etc/cumin/
  • /etc/conftool/
  • /etc/debmonitor.conf
  • /root/.ssh/new_install # SSH key to ssh into the debian-installer
Write access
  • /var/log/spicerack
Accounts/network access
  • AuthDNS
    • A:dns-auth hosts on port 5353
  • Ganeti
    • Ganeti RO API account (ro_user user)
    • ganeti01.svc.$DC.wmnet:5080
  • Debmonitor
    • Debmonitor RW on all hosts access (implicit for spicerack being run from a proxy host, hard to make RO only for some)
    • debmonitor.discovery.wmnet:443
  • Netbox
    • Netbox RW API account (sre_bot user)
  • Phabricator
    • Phabricator RW API account, no special privileges (ops-monitoring-bot user)
  • Logging (tcpircbot for !log)
    • icinga.wikimedia.org:9200
  • Conftool (etcd)
    • etcd RW account (root user)
    • conf*.$DC.wmnet:4001 # SRV targets for _etcd._tcp.conftool.$DC.wmnet
  • Prometheus
    • prometheus.svc.$DC.wmnet/ops/api/v1/query
  • Elasticsearch
    • search.svc.$DC.wmnet:{9243,9443,9643} # Search clusters elasticsearch API
    • relforge*.$DC.wmnet:{924,9443} # Relforge elasticsearch API
  • Redis (to be removed soon?)
    • Redis password
    • mc*.$DC.wmnet:6379 # Redis clusters
  • Selected cookbooks (reimage/decommission/ipmi-password-reset)
    • management password

Event Timeline

Volans triaged this task as Medium priority.Feb 11 2020, 9:43 AM
Volans created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 11 2020, 9:43 AM
Possible improvements
  • Netbox
    • Have 2 config files in /etc/spicerack/netbox/, one RW and one RO with different permissions.
    • Change Spicerack.netbox() to accept a write=False param and load the appropriate file based on that.
    • Update the cookbooks that require to write on netbox to use write=True.
  • Various spicerack modules that use Remote (cumin) and require root might self-detect if they are running with less privileges and bail out at instantiation time instead of failing in the middle of the execution.
  • conftool/etcd: implement T97972 and adapt spicerack accordingly, possibly having multiple config files with different permissions like the above proposal for Netbox.
Volans moved this task from Backlog to In Progress on the SRE-tools board.Feb 11 2020, 10:01 AM
Dzahn added a subscriber: Dzahn.Jun 27 2020, 6:28 PM
Volans moved this task from In Progress to Up next on the SRE-tools board.Jul 27 2020, 9:08 AM
Gehel added a subscriber: Gehel.Aug 27 2020, 1:16 PM
dcausse added a subscriber: dcausse.
CDanis added a subscriber: CDanis.Fri, Oct 16, 1:16 PM