Page MenuHomePhabricator

Eliminate sudo rm -rf /var/lib/puppet/ssl step in new deployment-prep WMCS project (and others)
Closed, ResolvedPublic

Description

Right now, whenever a new VM is created in the deployment-prep WMCS project, the following manual steps are required for puppet to work correctly:

  • On the new node: sudo rm -rf /var/lib/puppet/ssl
  • On the puppetmaster: sudo puppetserver ca sign --certname <fqdn-of-new-instance>

This is unpleasant and inefficient and something computers should be handling.

Event Timeline

@Andrew Do you anticipate any issues if we set profile::puppetserver::autosign to true (or something) in the deployment-prep project? profile::puppetserver::autosign was mentioned in T421244#11847986.

Change #1302978 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] modules/profile/files/puppet/bin: cleanup puppet SSL on CA server mismatch

https://gerrit.wikimedia.org/r/1302978

@Andrew Do you anticipate any issues if we set profile::puppetserver::autosign to true (or something) in the deployment-prep project? profile::puppetserver::autosign was mentioned in T421244#11847986.

That should be fine. You'll still have to do part of the dance to clear out the old certs on clients unfortunately.

Mentioned in SAL (#wikimedia-releng) [2026-06-24T17:26:18Z] <dancy> Set profile::puppetserver::autosign: /usr/local/sbin/validatecloudvpsfqdn.py in hiera config for deployment-puppetserver prefix (T429413)

• bd808 changed the task status from Open to In Progress.Jun 25 2026, 5:59 PM
• bd808 assigned this task to dancy.
• bd808 moved this task from To Triage to Future on the Beta-Cluster-Infrastructure board.

@Andrew Do you anticipate any issues if we set profile::puppetserver::autosign to true (or something) in the deployment-prep project? profile::puppetserver::autosign was mentioned in T421244#11847986.

That should be fine. You'll still have to do part of the dance to clear out the old certs on clients unfortunately.

This is handled by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302978 which has been tested successfully in deployment-prep.

Change #1302978 merged by Andrew Bogott:

[operations/puppet@production] modules/profile/files/puppet/bin: cleanup puppet SSL on CA server change

https://gerrit.wikimedia.org/r/1302978

  • On the new node: sudo rm -rf /var/lib/puppet/ssl

Resolved via https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302978

  • On the puppetmaster: sudo puppetserver ca sign --certname <fqdn-of-new-instance>

Resolved by setting profile::puppetserver::autosign: /usr/local/sbin/validatecloudvpsfqdn.py in deployment-puppetserver Prefix Puppet Hiera Config. Noting that https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver#Autosigning discourages the use of autosigning if there are local secrets in puppet, which there are in deployment-prep. It would be nice to address this in some way in the future (for example by allowing the puppetserver to be informed about newly created VMs so it can be more selective about what names it will autosign for).

Resolved by setting profile::puppetserver::autosign: /usr/local/sbin/validatecloudvpsfqdn.py in deployment-puppetserver Prefix Puppet Hiera Config. Noting that https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver#Autosigning discourages the use of autosigning if there are local secrets in puppet, which there are in deployment-prep. It would be nice to address this in some way in the future (for example by allowing the puppetserver to be informed about newly created VMs so it can be more selective about what names it will autosign for).

One possible improvement here could be a replacement/config for validatecloudvpsfqdn.py that validates that the host FQDN is associated with the same Cloud VPS project as the puppetserver. The current script use the OpenStack APIs to verify that the FQDN is associated with any Cloud VPS project, but not a specific project. I think that pinning the validated project is functionally what @dancy said as "allowing the puppetserver to be informed about newly created VMs".

Change #1306745 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] profile::puppet::agent: write pinned CA cert with a trailing newline

https://gerrit.wikimedia.org/r/1306745

Change #1307202 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] profile::base::certificates: drop unused puppet_ca_content CA-swap support

https://gerrit.wikimedia.org/r/1307202

Change #1306745 merged by RLazarus:

[operations/puppet@production] profile::puppet::agent: write pinned CA cert with a trailing newline

https://gerrit.wikimedia.org/r/1306745