Whenever an instance is added to the beta cluster and switched to the local salt master, we might forget to sign the key on the salt master. We should get a monitoring for any unsigned or rejected keys:
Example:
root@deployment-salt:~# salt-key --list rejected Rejected Keys: root@deployment-salt:~# salt-key --list unsigned Unaccepted Keys: i-000004f8.eqiad.wmflabs i-000005ba.eqiad.wmflabs root@deployment-salt:~#
Version: unspecified
Severity: enhancement
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | ArielGlenn | T102039 Disabling agent forwarding breaks dsh based restarts for Parsoid (required for deployments) | |||
| Declined | ArielGlenn | T63882 [Trebuchet] Salt times out on parsoid restarts | |||
| Open | None | T53494 Use Beta cluster as a true canary for code deployments (epic) | |||
| Stalled | None | T53497 Setup monitoring for Beta Cluster (tracking) | |||
| Declined | yuvipanda | T87803 Cannot use dsh-based restart of parsoid from tin anymore | |||
| Resolved | ArielGlenn | T88971 Upgrade salt to 2014.7 (investigating) | |||
| Declined | yuvipanda | T72862 monitor unsigned salt keys |
Yuvi, I am not sure how familiar you are with diamond. Would it make sense to write a basic collector that list the rejected/unsigned keys on the salt master, send that to graphite and alert on them?
Indeed, that seems ok to do. *Ideally* we would just do this in icinga instead of with diamond, but considering icinga status on labs I'd say go ahead with doing it in diamond. We already have written some custom collectors for us (see minimalpuppetagent.py), and it should be fairly trivial to copy that and use it here.
Do you want to give it a shot? I can help with the diamond bits :)
I already have too many things to complete which are long overdue. So I am unlikely to look at writing a diamond collector anytime soon. If you have some spare bandwidth, please step in :-D
Alright, I'll put it on my 'spare bandwidth TODO' list :)
In the meantime, if anyone else wants to step in, please do! I'll be happy to help.
http://docs.saltstack.com/en/latest/ref/configuration/master.html#auto-accept salt has auto-accept now.
Should just be fixed with the next version upgrade that will have auto accepting keys.