This meta task tracks remaining steps which need to happen before we can retire the Puppet 5 servers
Description
Details
Event Timeline
Change #1224687 merged by Muehlenhoff:
[operations/puppet@production] Rename stale_certs_exporter and move under puppetserver
Change #1224689 merged by Muehlenhoff:
[operations/puppet@production] Rename puppetmaster::gitsync and move under puppetserver
Change #1224722 merged by Muehlenhoff:
[operations/puppet@production] Remove Puppet 5 settings from late_command.sh
Change #1224605 merged by Muehlenhoff:
[operations/puppet@production] puppet: Remove the force_puppet7 parameter
Change #1225502 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from insetup roles
Change #1225502 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from insetup roles
Change #1225519 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from backup hosts
Change #1225522 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from swift/ceph hosts
Change #1225524 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from traffic hosts
Change #1225525 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from Cassandra hosts
Mentioned in SAL (#wikimedia-operations) [2026-01-12T12:17:30Z] <moritzm> revoked legacy default-staging-certificate certificate T365798
Mentioned in SAL (#wikimedia-operations) [2026-01-12T12:19:16Z] <moritzm> revoked legacy ganeti02.svc.esams certificate T365798
Mentioned in SAL (#wikimedia-operations) [2026-01-12T12:26:37Z] <moritzm> revoked legacy linkrecommendation discovery certificate T365798
Change #1225519 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from backup hosts
Change #1225522 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from swift/ceph hosts
Change #1225576 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from DB hosts
Change #1225525 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from Cassandra hosts
Change #1225576 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from DB hosts
Change #1226175 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from collab roles
Change #1226176 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from ML roles
Change #1226178 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from observability roles
Mentioned in SAL (#wikimedia-operations) [2026-01-13T10:06:50Z] <moritzm> revoked legacy similar-users discovery certificate T365798
Change #1226176 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from ML roles
Change #1226175 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from collab roles
Mentioned in SAL (#wikimedia-operations) [2026-01-13T15:11:31Z] <moritzm> revoked legacy restbase discovery certificate T365798
Change #1226178 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from observability roles
Change #1227261 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from serviceops roles
Change #1227264 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from cloud roles
Change #1227270 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from search roles
Change #1227264 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from cloud roles
Change #1227261 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from serviceops roles
Change #1227292 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from IF roles
Change #1227292 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from IF roles
Change #1227313 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from Data Platform roles
Change #1227322 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 for Cloud VPS
Change #1227322 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 for Cloud VPS
Change #1227270 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from search roles
Change #1227313 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from Data Platform roles
Change #1225524 merged by Muehlenhoff:
[operations/puppet@production] Remove profile::puppet::agent::force_puppet7 from traffic hosts
Change #1227616 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove remaining traces of profile::puppet::agent::force_puppet7
Change #1227618 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Rename enc_client and move under puppetserver
Change #1227694 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Move validatecloudvpsfqdn.py out of the puppetmaster module
Change #1227698 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster spec files
Change #1227702 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Copy yamllint into the puppetserver module and use it
Change #1227734 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] pcc_update_facts: Rename variables
Change #1227698 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster spec files
Change #1227618 merged by Muehlenhoff:
[operations/puppet@production] Rename enc_client and move under puppetserver
Change #1227694 merged by Muehlenhoff:
[operations/puppet@production] Move validatecloudvpsfqdn.py out of the puppetmaster module
Change #1227616 merged by Muehlenhoff:
[operations/puppet@production] Remove remaining traces of profile::puppet::agent::force_puppet7
Change #1227702 merged by Muehlenhoff:
[operations/puppet@production] Copy yamllint into the puppetserver module and use it
Change #1229107 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster::updatenetboot
Change #1229110 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove Puppet 5 volatile directory from backups
Change #1229110 merged by Muehlenhoff:
[operations/puppet@production] Remove Puppet 5 volatile directory from backups
Change #1229107 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster::updatenetboot
Change #1230331 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster role from puppetmaster2001
Change #1230332 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster2001 from active Puppet 5 servers
Change #1230912 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Stop running the IP reputation dump on the Puppet 5 servers
Change #1230913 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove ip_reputation_vendors from Puppet 5 servers
Change #1230914 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove ip_reputation_vendors from puppetmasters
Change #1230913 abandoned by Muehlenhoff:
[operations/puppet@production] Remove ip_reputation_vendors from Puppet 5 servers
Reason:
dupe
Mentioned in SAL (#wikimedia-operations) [2026-02-04T12:53:00Z] <moritzm> remove legacy eventstreams-internal discovery certificate T365798
Mentioned in SAL (#wikimedia-operations) [2026-02-04T13:00:21Z] <moritzm> remove legacy wdqs-internal discovery certificate T365798
Mentioned in SAL (#wikimedia-operations) [2026-02-04T14:27:52Z] <moritzm> remove legacy kibana discovery certificate T365798
Change #1230331 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster role from puppetmaster2001
cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: puppetmaster2001.codfw.wmnet
- puppetmaster2001.codfw.wmnet (PASS)
- Downtimed host on Icinga/Alertmanager
- Found physical host
- Downtimed management interface on Alertmanager
- Wiped all swraid, partition-table and filesystem signatures
- Powered off
- [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
- Configured the linked switch interface(s)
- Removed from DebMonitor
- Removed from Puppet master and PuppetDB
Change #1230332 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster2001 from active Puppet 5 servers
Change #1237942 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/cookbooks@master] sre.ganeti.makevm: Stop passing the puppetversion
Change #1237942 merged by Muehlenhoff:
[operations/cookbooks@master] sre.ganeti.makevm: Stop passing the puppetversion
Change #1239314 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/cookbooks@master] sre.hosts.decommission: Hack to allow decommission of puppetmaster1001
cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: puppetmaster1001.eqiad.wmnet
- puppetmaster1001.eqiad.wmnet (PASS)
- Downtimed host on Icinga/Alertmanager
- Found physical host
- Downtimed management interface on Alertmanager
- Wiped all swraid, partition-table and filesystem signatures
- Powered off
- [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
- Configured the linked switch interface(s)
- Removed from DebMonitor
- Removed from Puppet master and PuppetDB
- COMMON_STEPS (FAIL)
- Failed to run the sre.dns.netbox cookbook, run it manually
ERROR: some step on some host failed, check the bolded items above
Change #1239638 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/cookbooks@master] sre.puppet.sync-netbox-hiera: Remove support for Puppet 5
Change #1239638 merged by Muehlenhoff:
[operations/cookbooks@master] sre.puppet.sync-netbox-hiera: Remove support for Puppet 5
Change #1239647 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] puppetdb: Drop firewall rule for access to Puppet 5 servers
Change #1239648 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove now obsolete Cumin aliases for Buster and Puppet 5
Change #1239676 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster::frontend role
Change #1239676 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster::frontend role
Change #1239314 abandoned by Muehlenhoff:
[operations/cookbooks@master] sre.hosts.decommission: Hack to allow decommission of puppetmaster1001
Change #1239749 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] profile::puppet::agent: Remove support for Buster
Change #1239648 merged by Muehlenhoff:
[operations/puppet@production] Remove now obsolete Cumin aliases for Buster and Puppet 5
Change #1239749 merged by Muehlenhoff:
[operations/puppet@production] profile::puppet::agent: Remove support for Buster
Change #1239891 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster::monitoring and related classes
Change #1239895 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster::gitclone and related classes
Change #1239897 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster::r10k
Change #1239898 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster::rsync and related classes
Change #1239899 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster::web_frontend and related classes
Change #1239907 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster::passenger and related files
Change #1239908 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] This was only used with Puppet 5.
Change #1239897 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster::r10k
Change #1239891 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster::monitoring and related classes
Change #1239898 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster::rsync and related classes
Change #1239908 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster:ssl
Change #1239907 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster::passenger and related files
Change #1239899 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster::web_frontend and related classes
Change #1239895 merged by Muehlenhoff:
[operations/puppet@production] Remove puppetmaster::gitclone and related classes
Change #1240242 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Revert "Remove puppetmaster::gitclone and related classes"
Change #1240242 merged by Muehlenhoff:
[operations/puppet@production] Revert "Remove puppetmaster::gitclone and related classes"
Change #1240268 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Remove puppetmaster::ca_server and related classes
Change #1240278 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Move the puppetmaster puppetdb client class under puppet_compiler