Review puppetmaster SSL configuration
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jbond
	Nov 17 2020, 4:48 PM

Description

the following change cause all of the puppet backends to fail with an error simlar to the folloowig

ov 17 16:29:03 puppetmaster2003 puppet-master[11348]: Could not run: The CSR retrieved from the master does not match the agent's public key.
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]: CSR fingerprint: E2:EB:87:A4:70:9B:18:5A:CA:0D:3F:CC:58:58:2B:37:98:58:3F:85:7F:EA:60:73:D4:15:48:ED:5A:36:B4:39
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]: CSR public key: RSA Public-Key: (4096 bit)
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]: Modulus:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     00:c6:e6:16:57:1d:a1:06:7a:77:8d:0b:9a:16:b8:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     c1:28:8f:a9:5e:dc:e7:65:65:51:07:84:a7:f6:17:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     54:ae:15:7a:0d:06:c6:ad:3c:ec:a5:5a:27:a9:6a:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     0a:22:6c:c2:6e:a7:d6:6f:5b:ca:dd:cf:bd:0c:3c:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     f1:e7:2e:91:08:85:67:70:72:de:be:ad:7f:f6:87:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     d4:a7:0d:24:48:dc:d7:ce:00:3e:30:9d:35:42:d6:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     1e:8d:7c:5a:ff:0a:b6:b0:b2:85:dc:be:7b:7a:e9:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     24:c2:ca:f2:f4:1b:32:26:c9:a4:e2:98:78:a4:83:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     d1:91:fc:53:77:30:be:2f:12:a4:de:49:f4:3e:0b:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     ee:01:43:0c:80:d7:cf:d8:a2:12:ff:ed:49:5b:44:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     55:8a:8d:41:9c:06:77:69:be:b4:3b:00:52:26:ff:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     e5:87:19:72:b9:e6:ba:6c:2e:ee:79:4e:6b:94:b8:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     c8:ee:a2:e5:58:92:f8:ab:5b:9a:2e:1f:14:1c:af:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     b7:72:02:90:c5:d7:1e:56:5f:45:dc:58:b9:70:79:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     f7:0c:42:2e:9b:23:a7:48:36:0e:4f:b7:88:3d:e5:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     dc:59:9f:9b:3c:09:4a:71:0f:7d:da:87:69:1f:58:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     ab:a6:cb:d1:38:a0:51:ce:b5:2d:8b:12:3b:79:d4:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     e7:2f:57:7d:97:59:5e:b4:71:7f:eb:8c:bc:3e:09:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     b8:e8:fa:a8:95:92:2c:03:d2:92:63:2d:29:ca:f6:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     1a:73:85:e6:dc:4b:d0:23:85:47:14:04:12:91:13:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     09:ee:57:48:81:47:52:c2:c2:a5:dc:8c:c3:50:5b:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     ea:35:1e:70:fd:6e:a4:12:02:b8:18:e8:36:e5:eb:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     84:0b:7a:57:97:15:26:d4:8e:8b:6c:8e:75:d0:55:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     9b:4b:f8:b9:6d:bd:ee:1a:d4:af:36:00:f3:55:d5:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     47:c5:a1:e2:f6:eb:19:de:8d:75:3c:19:d5:03:9f:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     a7:12:a8:84:24:62:c1:63:cc:fe:9d:42:6e:91:06:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     ef:be:1f:25:0f:d7:97:3a:7e:f5:90:13:db:a1:0e:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     ce:6e:1a:0d:b8:59:36:2d:e4:2a:df:c4:d3:3f:ae:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     92:e0:95:7b:e5:59:de:1d:15:1d:a1:a6:85:24:31:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     de:ec:f7:8b:b0:55:7e:87:07:86:02:51:e2:ee:ce:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     5f:eb:da:6d:ea:21:fa:f1:4f:70:68:f7:20:42:8d:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     b2:8f:81:88:d8:6b:df:2d:42:87:e1:9a:22:fa:e7:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     3c:e3:cb:8b:0c:74:46:aa:1f:89:50:10:9b:00:ae:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     cf:5c:59:52:4b:5c:96:7e:68:2f:50:48:60:2f:33:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     f9:e6:ef
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]: Exponent: 65537 (0x10001)
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]: 
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]: Agent public key: RSA Public-Key: (4096 bit)
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]: Modulus:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     00:c6:a9:f7:97:46:5a:5c:28:4d:03:8c:d2:5c:90:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     a7:d0:c1:08:bd:87:8c:94:2b:76:61:9f:96:b8:8e:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     08:b8:58:bd:59:a0:c5:13:d6:ad:91:c1:37:a5:eb:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     c9:99:93:2e:89:17:63:3d:ce:15:7e:03:2b:a2:08:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     34:f2:68:0d:03:6b:2a:91:e1:1c:16:c2:b2:9a:a9:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     6e:d2:33:fb:98:89:09:2a:22:ae:2b:a9:46:23:43:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     3d:bc:05:20:8a:3e:d8:9b:a9:8e:06:e0:91:22:2d:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     54:dd:68:29:fe:49:09:af:e5:05:4c:ed:cb:d5:97:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     1b:ea:d9:f6:76:6c:8f:61:2b:dd:2e:e9:6f:76:58:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     fe:0d:4b:28:e8:df:6e:15:62:0a:ff:84:fd:c8:b6:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     97:fa:b9:ff:52:7d:3f:5d:31:5c:24:32:0e:e7:2b:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     96:4d:29:08:ed:9f:7a:1f:b9:80:9a:b7:49:b7:0e:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     c6:43:ee:24:28:bc:f6:96:3c:ca:f2:ab:fc:9a:38:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     72:f4:6c:d6:a1:d3:5e:6c:a6:58:90:7f:78:fa:9a:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     89:fe:cc:d1:4d:8e:1b:99:1b:41:4c:a2:c8:07:7d:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     b7:26:9b:8d:28:f4:02:fc:af:da:40:7e:14:bc:99:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     01:49:ca:3f:50:bf:55:34:a6:46:50:52:a4:c1:68:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     af:e3:ef:74:e2:66:c0:04:08:2a:20:86:6f:a3:2d:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     3d:c4:8a:4f:03:3e:a1:5d:eb:62:a8:2d:97:80:b0:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     e9:95:ce:3c:6e:6c:89:9e:c3:48:2c:0f:f6:d7:7e:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     56:94:5d:3a:42:7c:ed:90:6b:af:e1:69:a6:b7:53:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     89:e2:88:f8:a2:87:1f:b3:01:73:64:be:62:1b:82:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     14:08:dc:6a:7e:cc:2e:b5:c6:04:07:2e:8b:b0:9f:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     07:00:9b:17:4d:42:d9:6d:ee:86:cb:5d:44:5b:13:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     63:30:8e:0d:43:1d:73:16:c1:26:37:72:ea:b4:38:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     86:da:67:eb:30:05:c3:ef:e2:68:51:c6:7c:b1:09:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     96:70:a5:4c:4e:cf:49:79:42:0d:37:00:be:dd:d6:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     1e:1d:c2:6b:0e:79:db:99:21:4c:5c:f2:da:85:c4:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     f2:e6:c8:21:d1:64:61:20:47:13:50:84:da:a0:77:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     cf:00:fc:2e:9a:6a:e2:ff:cc:5b:d9:ea:44:fb:0d:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     47:66:5a:2d:fa:e8:55:e4:63:ca:42:a3:1b:24:1a:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     6c:d3:ba:09:c3:f6:13:16:99:23:9c:72:a6:2d:3b:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     c2:63:01:5f:5d:2a:0b:3e:9d:df:72:37:e4:c4:6d:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     1d:72:a6:9a:48:32:bc:d2:6c:71:e9:43:a8:43:90:
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:     73:d9:45
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]: Exponent: 65537 (0x10001)
Nov 17 16:29:03 puppetmaster2003 puppet-master[11348]:

It seems this is related to the following files which are automaticly created by the puppet master if they dont exist

1315220      4 -rw-r--r--   1 puppet   puppet       1582 Jan 20  2020 /var/lib/puppet/server/ssl/certificate_requests/puppet.pem
1315213      4 -rw-r--r--   1 puppet   puppet        800 Nov 17 15:28 /var/lib/puppet/server/ssl/public_keys/puppet.pem
1315212      4 -rw-r-----   1 puppet   puppet       3243 Nov 17 15:28 /var/lib/puppet/server/ssl/private_keys/puppet.pem

The change refrenced above create a new private key which likley had a missmatch with the pkey automaticly create by the puppet-master process

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	puppetmaster: only configuere a separate ssl dir for CA puppetmasters	operations/puppet	production	+6 -2

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T294906 Puppet Improvements
		Resolved		jbond	T268040 Review puppetmaster SSL configuration

Event Timeline

jbond created this task.Nov 17 2020, 4:48 PM

jbond triaged this task as Medium priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 17 2020, 4:48 PM

Tagging https://gerrit.wikimedia.org/r/c/operations/puppet/+/386666 as although its slightly different it seems to be around the same bit of code

checking the following one the backends shows that the keys are all different which points to the puppet master process generating theses keys when it first needs them as appose to reciving them from the configuered puppet master

$ openssl pkey -in /var/lib/puppet/server/ssl/public_keys/puppet.pem  -pubin -noout -text
$ openssl req -in ssl/certificate_requests/puppet.pem -noout -text

/var/lib/puppet/server/ssl/public_keys/puppet.pem dosn't exist on the frontend servers and is never generated

I have done some initial testing an i think we should just drop the ssl config from the puppetmaster backend servers and let it use the default. the backends dont ,ever do CA operations so they dont need to worry about that and the frontends have the CA dir rsynced

Change 641472 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: only configuere a separate ssl dir for CA puppetmasters

https://gerrit.wikimedia.org/r/641472

gerritbot added a project: Patch-For-Review.Nov 17 2020, 6:07 PM

Aklapper renamed this task from Investigate why the exictence of files under the server ssl dir foobars puppet to Investigate the existence of files under the server ssl dir foobars puppet.Nov 17 2020, 8:14 PM

Change 641472 abandoned by Jbond:
[operations/puppet@production] puppetmaster: only configuere a separate ssl dir for CA puppetmasters

Reason:
I think this needs a bit more thought and maybe changes else where will update the task

https://gerrit.wikimedia.org/r/641472

jbond renamed this task from Investigate the existence of files under the server ssl dir foobars puppet to Review puppetmaster SSL configueration.Nov 19 2020, 12:54 PM

In considering this task more I think one complication is an SSL certificate for CN=puppet which is copied to every server so that it can be used to provide SSL Client authentication to the https://puppet:8140/ endpoint. Note this is not the Root certificate which has a CN of Puppet CA: palladium.eqiad.wmnet.

I wonder if we should instead make use of the dns_alt_name puppet config paramter to add the puppet dns name to the puppetmaster agent certificates which would remove the need to maintain the additional puppet.pem certificate in the private repo

I also wonder if the production environment needs to configure a separate ssldir in the master section CA. A separate master section is useful when a puppetmaster is also an agent to a separate puppetmaster infrastructure e.g. puppetmaster::standalone and the cloud puppet masters. however it feels like this just adds unnecessary complications in the production environment.

jbond added subscribers: colewhite, Volans, akosiaris and 2 others.Nov 19 2020, 1:21 PM

jbond added a project: User-jbond.Jun 1 2021, 2:56 PM

jbond moved this task from Unsorted 💣 to Back Burner 🏛️ on the User-jbond board.Jun 1 2021, 2:59 PM

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2021, 3:10 PM

Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM

jbond added a parent task: T294906: Puppet Improvements.Nov 3 2021, 11:40 AM

• joanna_borun renamed this task from Review puppetmaster SSL configueration to Review puppetmaster SSL configuration.Nov 18 2021, 3:35 PM

@jbond: Removing task assignee as this open task has been assigned for more than two years - See the email sent to task assignee on Feburary 22nd, 2023.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome! :)
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!

This will all change in puppet7

Review puppetmaster SSL configurationClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Review puppetmaster SSL configuration
Closed, ResolvedPublic
Actions

Related Objects
Search...