Found non-revoked Puppet certificates for 3 deleted instances on deployment-puppetserver-1
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	wmcs-alerts
	Sat, Oct 3, 2:49 PM

Project Tags

Referenced Files

None

Subscribers

Description

Common information

runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates
summary: Found non-revoked Puppet certificates for 3 deleted instances on deployment-puppetserver-1

alertname: PuppetStaleCertificates
instance: deployment-puppetserver-1
project: deployment-prep
severity: warning

Firing alerts

runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates
summary: Found non-revoked Puppet certificates for 3 deleted instances on deployment-puppetserver-1
alertname: PuppetStaleCertificates
instance: deployment-puppetserver-1
project: deployment-prep
severity: warning
Source

Related Objects

Mentioned Here: T429497: Upgrade (replace) beta|deployment-prep event platform VPS instances

Event Timeline

wmcs-alerts created this task.Sat, Oct 3, 2:49 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSat, Oct 3, 2:49 PM

Hi @tchin, I think this alert is related to your work on T429497. From the Prometheus metrics:

puppetmaster_stale_cert{cert_instance="deployment-eventstreams-2",cert_name="deployment-eventstreams-2.deployment-prep.eqiad1.wikimedia.cloud",cert_project="deployment-prep"} 1.0
puppetmaster_stale_cert{cert_instance="deployment-eventgate-4",cert_name="deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloud",cert_project="deployment-prep"} 1.0
puppetmaster_stale_cert{cert_instance="deployment-schema-3",cert_name="deployment-schema-3.deployment-prep.eqiad1.wikimedia.cloud",cert_project="deployment-prep"} 1.0

Could you confirm the instances have been decommissioned? Revoking the Puppet certificates is part of the decommission checklist, although it's definitely not optional. The checklist is a tad confusing.

Let me know if I can help!

Ooo yes sorry they have been decommissioned! I don't think I have the permissions to log into puppetmaster though, can you do it?

~~Or is puppetmaster that the page refers to deployment-cumin-3.deployment-prep.eqiad1.wikimedia.cloud? Or I guess it's 04 now?~~ nevermind I just tried

Mentioned in SAL (#wikimedia-releng) [2026-10-03T19:03:44Z] <Southparkfan> clean Puppetserver references to deployment-eventstreams-2 deployment-eventgate-4 deployment-schema-3 - T440085

Thanks, @tchin! I have performed the required actions on deployment-puppetserver-1:

for host in deployment-eventstreams-2 deployment-eventgate-4 deployment-schema-3; do fqdn="${host}.deployment-prep.eqiad1.wikimedia.cloud"; puppet node deactivate "${fqdn}" && puppet node clean "${fqdn}"; done

Found non-revoked Puppet certificates for 3 deleted instances on deployment-puppetserver-1Closed, ResolvedPublicActions