Page MenuHomePhabricator

Make it possible for wmf-reimage to work seamlessly with a non-local salt master
Closed, ResolvedPublic

Description

At the moment, wmf-reimage relied on having a local salt master to add the key of a specific server to the salt ring of trust. This isn't possible anymore now that the salt master is on a separate server than the puppet master.

I have had a few ideas:

  1. Create a simple script running on neodymium that just looks at the unaccepted keys locally, checks the puppet master for accepted certs with the same name, and in case signs the salt key
  2. Use some keyholder mechanism to allow wmf-reimage to connect to neodymium (but this is highly suboptimal)

Details

Related Gerrit Patches:

Event Timeline

Joe created this task.Jan 26 2016, 10:30 AM
Joe assigned this task to ArielGlenn.
Joe claimed this task.
Joe raised the priority of this task from to Medium.
Joe updated the task description. (Show Details)
Joe added projects: Operations, Salt, Puppet.
Joe set Security to None.
Joe added subscribers: Joe, MoritzMuehlenhoff, Aklapper and 2 others.

We could write a runner for the salt master that accepts a key after checking the puppet accepted cert, and we could configure the salt master to accept this command from the puppet masters, much as we allow it to accept git deploy commands from the deployment servers. There's already management.safe_accept() which accepts keys after a check via salt-ssh.

Joe added a comment.Jan 27 2016, 10:33 AM

@ArielGlenn it seems like a good idea.

ArielGlenn moved this task from Backlog to active on the Salt board.Feb 1 2016, 12:53 PM

Change 267670 had a related patch set uploaded (by ArielGlenn):
new salt runner to sign key for a specific minion

https://gerrit.wikimedia.org/r/267670

Tested with a print in place of the key acceptance line.

Joe added a comment.Feb 2 2016, 8:58 AM

BTW, wmf-reimage need to be able to both sign the key and delete it, as the process it follows is:

  1. clean puppet certs
  2. clean salt keys
  3. reboot in pxe mode && reimage
  4. accept puppet key when it's available
  5. run puppet on the host
  6. accept salt key when it's available

so we need that the sign salt key runner returns a value that can be evaluated by wmf-reimage too.

I've got that going now, have a look at the code; you can ask for status of a key, accept or delete, and you get some output back depending on whether the key was found to be accepted/deleted or not, etc.

elukey added a subscriber: elukey.Feb 8 2016, 2:40 PM

Change 267670 merged by ArielGlenn:
new salt runner to accept/delete/check status of salt key

https://gerrit.wikimedia.org/r/267670

root@palladium:~# salt-call publish.runner keys.status dataset1001.wikimedia.org
[INFO ] Publishing runner 'keys.status' to tcp://10.64.32.20:4506
local:


status:
    accepted

root@palladium:~# salt-call publish.runner keys.status dataset1001.wikimedia.o
[INFO ] Publishing runner 'keys.status' to tcp://10.64.32.20:4506
local:


status:
    missing

over to you now, @Joe, let me know what else you need as you come across it.

Change 269437 had a related patch set uploaded (by Giuseppe Lavagetto):
puppetmaster: adapt wmf-reimage to use remote salt calls

https://gerrit.wikimedia.org/r/269437

Change 269437 merged by Giuseppe Lavagetto:
puppetmaster: adapt wmf-reimage to use remote salt calls

https://gerrit.wikimedia.org/r/269437

ArielGlenn moved this task from active to testing needed on the Salt board.Feb 15 2016, 11:30 AM
Joe added a comment.Mar 16 2016, 10:02 AM

I tested this, finally, yesterday:

while the key gets correctly deleted, it seemed not able to find the key to sign afterwards; will need to investigate before we start heavily reimaging next quarter.

OK, I'll double-check my end of the code again just in case.

Joe added a comment.Mar 31 2016, 10:41 AM

@ArielGlenn the problem lies solely with wmf-reimage, I'm going to fix it now.

Change 280647 had a related patch set uploaded (by Giuseppe Lavagetto):
wmf-reimage: fix salt signing

https://gerrit.wikimedia.org/r/280647

Change 280647 merged by Giuseppe Lavagetto:
wmf-reimage: fix salt signing

https://gerrit.wikimedia.org/r/280647

Joe closed this task as Resolved.Apr 4 2016, 8:39 AM
ArielGlenn moved this task from Blocked/Stalled to Done on the Salt board.Apr 4 2016, 9:24 AM