In order to reduce the requirement of being root to be able to run cumin or spicerack cookbooks as part of this Q OKRs, this task is focused on the auditing part of the existing requirements and then on the proposal of possible improvements.
Current Requirements
Cumin
Some preconditions:
- The target server on which command should be executed needs to join the existing WIKIMEDIA Kerberos realm (profile::kerberos::client) (and the Cumin hosts)
- Each kerberos-enabled sshd needs a "host" keytab deployed (https://unix.stackexchange.com/questions/339149/can-keytab-file-location-be-configured-in-openssh)
- We need a few config changes on the SSHD config to enable GSSAPI logins
- To execute commands we need a local user, so we we need to tweak the admin module to create a user without deploying an SSH key
- The access for the Cumin root key is controlled via the authorized_keys file in /etc/ssh/userkeys/root.d/cumin, but that mechanism doesn't exist for kerberized logins (There is k5login, but it was originally meant for krsh and seems to brittle to rely on: https://web.mit.edu/kerberos/krb5-1.13/doc/user/user_config/k5login.html)
We can however use group-based matching in the OpenSSH config to control _who_ is allowed to log into a server via Kerberos The users which are locally created by the admin module can be added to a local Unix group like "krblogin". Users in that group can then be matched and redirected to a ForceCommand execution, e.g.
Match Group krblogin
ForceCommand /usr/bin/cumin-login-wrapperThe $COMMAND passed by the user is accessible by the cumin-login-wrapper as $SSH_ORIGINAL_COMMAND. The wrapper can perform a check whether a user
is allowed to execute the given command and do so if permitted.
This allows for a setup where people SSH to the Cumin hosts and then run e.g.
cumin-nonpriv install1002.wikimedia.org 'sudo systemctl restart dhcp'
This would internally execute a SSH login using Kerberos running "ssh -K $DESTINATIONHOST $COMMAND"
(The use of the wrapper isn't strictly necessary, but still seems useful to allow full logs of executed commands, better error reporting to the users (e.g. in case a ticket has expired) and it provides a method to restrict kerberized SSH logins to a whitelisted list of commands (which we can generate from a common Puppet structure along with the sudo rules) and restricting a user from logging into the host itself.)
Spicerack
Read access
- /etc/spicerack
- /srv/deployment/spicerack
- /etc/phabricator_ops-monitoring-bot.conf
- /etc/cumin/
- /etc/conftool/
- /etc/debmonitor.conf
- /root/.ssh/new_install # SSH key to ssh into the debian-installer
Write access
- /var/log/spicerack
Accounts/network access
- AuthDNS
- A:dns-auth hosts on port 5353
- Ganeti
- Ganeti RO API account (ro_user user)
- ganeti01.svc.$DC.wmnet:5080
- Debmonitor
- Debmonitor RW on all hosts access (implicit for spicerack being run from a proxy host, hard to make RO only for some)
- debmonitor.discovery.wmnet:443
- Netbox
- Netbox RW API account (sre_bot user)
- Phabricator
- Phabricator RW API account, no special privileges (ops-monitoring-bot user)
- Logging (tcpircbot for !log)
- icinga.wikimedia.org:9200
- Conftool (etcd)
- etcd RW account (root user)
- conf*.$DC.wmnet:4001 # SRV targets for _etcd._tcp.conftool.$DC.wmnet
- Prometheus
- prometheus.svc.$DC.wmnet/ops/api/v1/query
- Elasticsearch
- search.svc.$DC.wmnet:{9243,9443,9643} # Search clusters elasticsearch API
- relforge*.$DC.wmnet:{924,9443} # Relforge elasticsearch API
- Redis (to be removed soon?)
- Redis password
- mc*.$DC.wmnet:6379 # Redis clusters
- Selected cookbooks (reimage/decommission/ipmi-password-reset)
- management password