Replace deployment-cumin-3 (Bullseye deprecation)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Southparkfan
	Aug 30 2026, 8:31 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

Southparkfan

Description

New hostname should match deployment-cumin0[0-9]. No special requirements known.

Details

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
scaps-installed-in-beta: Switch to new Cumin host	repos/releng/scap!1332	southparkfan	T436370	master
cumin: wrap up Puppet config	cloudvps-repos/deployment-prep/tofu-provisioning!39	southparkfan	T436470-finish	main
cumin: add host hiera	cloudvps-repos/deployment-prep/tofu-provisioning!38	southparkfan	T436470-add-host-hiera	main
cumin: remove profile::openstack::eqiad1::cumin::aliases	cloudvps-repos/deployment-prep/tofu-provisioning!37	southparkfan	T436470-remove-hiera	main
cumin: add deployment-cumin04	cloudvps-repos/deployment-prep/tofu-provisioning!36	southparkfan	T436470-initial	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T401839 Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie
		Resolved		Southparkfan	T436470 Replace deployment-cumin-3 (Bullseye deprecation)

Event Timeline

Southparkfan created this task.Aug 30 2026, 8:31 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 30 2026, 8:31 PM

Southparkfan updated the task description. (Show Details)Aug 30 2026, 8:32 PM

Southparkfan mentioned this in T401839: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie.Aug 30 2026, 8:52 PM

Southparkfan changed the task status from Open to In Progress.Sat, Sep 12, 5:35 PM

Southparkfan claimed this task.

southparkfan opened https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/36

cumin: add deployment-cumin04

southparkfan merged https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/36

cumin: add deployment-cumin04

southparkfan opened https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/37

cumin: remove profile::openstack::eqiad1::cumin::aliases

southparkfan merged https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/37

cumin: remove profile::openstack::eqiad1::cumin::aliases

southparkfan opened https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/38

cumin: add host hiera

southparkfan merged https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/38

cumin: add host hiera

Puppet run fails:

# run-puppet-agent
Local puppet cert no longer verifies against the pinned puppetserver CA; re-anchoring SSL
Notice: Removed private key /var/lib/puppet/ssl/private_keys/deployment-cumin04.deployment-prep.eqiad1.wikimedia.cloud.pem
Notice: Removed certificate request /var/lib/puppet/ssl/certificate_requests/deployment-cumin04.deployment-prep.eqiad1.wikimedia.cloud.pem
Error: Could not run: Could not submit certificate request for 'deployment-cumin04.deployment-prep.eqiad1.wikimedia.cloud' to https://deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:8140/puppet-ca/v1 due to a conflict on the server
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for deployment-cumin04.deployment-prep.eqiad1.wikimedia.cloud
Info: Certificate Request fingerprint (SHA256): 08:32:B8:C7:14:3D:F8:49:DE:9E:6B:88:3A:24:12:E3:50:A2:A4:97:CE:A6:E1:95:EB:60:2A:B4:22:69:1F:A6
Info: Downloaded certificate for deployment-cumin04.deployment-prep.eqiad1.wikimedia.cloud from https://deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:8140/puppet-ca/v1
Error: The certificate for 'CN=deployment-cumin04.deployment-prep.eqiad1.wikimedia.cloud' does not match its private key
Error: The certificate for 'CN=deployment-cumin04.deployment-prep.eqiad1.wikimedia.cloud' does not match its private key

Somehow, there's a mismatch:

root@deployment-cumin04:/var/lib/puppet/ssl# openssl rsa -in ./private_keys/deployment-cumin04.deployment-prep.eqiad1.wikimedia.cloud.pem -noout -modulus | openssl sha256
SHA2-256(stdin)= 6611000d8568f2d37951a0c7770acad3482ea49f665bdeda1e82f29911c8b8b6
root@deployment-cumin04:/var/lib/puppet/ssl# openssl x509 -in ./certs/deployment-cumin04.deployment-prep.eqiad1.wikimedia.cloud.pem -noout -modulus | openssl sha256
SHA2-256(stdin)= ba60cde2e6f485105915e3581579b41edfd47068b719355a810da4e732c36390

Fixed after wiping /var/lib/puppet/ssl on deployment-cumin04 and running puppetserver ca clean --certname deployment-cumin04.deployment-prep.eqiad1.wikimedia.cloud on the Puppetserver. Couldn't find an existing, open Phabricator task at first glance.

Maintenance_bot removed a project: Patch-For-Review.Sat, Sep 12, 7:30 PM

southparkfan opened https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/39

cumin: wrap up Puppet config

southparkfan merged https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/39

cumin: wrap up Puppet config

Manual steps, not in Puppet:

Copy /root/.ssh/cumin{,.pub} to new host
Arm the keyholder (keyholder arm) using the passphrase found at deployment-puppetserver-1:/srv/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
Add the primary IPv4 and IPv6 addresses to profile::openstack::eqiad1::cumin::project_masters

To do: