Page MenuHomePhabricator

Next steps for Puppet 7
Open, MediumPublic

Description

T184564: Plan Puppet 5 upgrade | Not yet


The Puppet maintainers in Debian have done amazing work and Bookworm will introduce a current stack:

Based on a discussion with John and myself the plan to move towards Puppet 7 would leads towards installing the new setup in parallel:

  • We already have a pair of puppetdb servers running on Bookworm on new hardware (puppetdb1003/puppetdb2003). With this there's also general compability of our Puppet tree with current facter and puppet-agent (there will be a need for more fine-tuning and some uses cases used by other roles, but the general support is present)
  • There's new hardware (puppetmaster[12]004/[12]005) meant to replace the old puppetmaster[12]00[12] servers. We can install puppetmaster[12]005 with Bookworm as the new puppetserver 7 frontends and puppetmaster[12]004 as the new puppetserver backends,
  • Create a separate Ganeti VM for puppetboard and serve is under a distinct URL like puppetboard7.wikimedia.org
  • Modify puppet-merge on puppetmaster[12]001 to sync every puppet change also towards puppetmaster[12]00[45]
  • Configure an additional/separate puppetdb 7 backend in Cumin
  • Modify the agent configuration to use a separate puppet7 CNAME (and create it)
  • Write a cookbook to remove a host from the Puppet 5 masters, add them for the Puppet 7 masters and modify agent config to use the puppet7 CNAME
  • Modify PCC to also compile every patch with Puppet 7 in parallel

Once there is two setups in parallel we can:

  • Use the cookbook to move the sretest* servers (and some other hosts like idp-test and some other canaries towards Puppet 7 (Buster/Bullseye/Bookworm on the agent side), test and fix up all issues
  • Use the cookbook to move ulsfo towards the Puppet 7 setup (and observe/fix for some time)
  • Use the cookbook to move all the remaining servers to Puppet 7
  • Migrate config-master over to puppetmaster1005
  • Reimage puppetmaster[12]003 with Bookworm
  • Decom puppetmaster[12]00[12] and the old puppetboard VM

Blockers To an upgrade

puppet forge core type

when puppet version 6 was released a bunch of core resource types where removed from the puppet code base and spun of as external forge modules. We should be aware of any resource we are using which fall into this catagory and either migrate to a different module/type or add the core modules to our code base when we upgrade

  • migrate all cron types to systemd::timer::job
  • if still using nagios migrate to puppetlabs/nagios_core as nagios types are no longer core puppet
  • Check if which other native types need to be migrated

Benefits to explore (some of theses benefits have be realized by updating to the latest 5.5 branch)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
operations/puppetproduction+3 -2
operations/puppetproduction+9 -2
operations/puppetproduction+34 -24
integration/configmaster+7 -1
operations/puppetproduction+6 -12
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+6 -2
operations/puppetproduction+3 -21
operations/puppetproduction+2 -1
operations/puppetproduction+1 -0
operations/puppetproduction+8 -2
operations/puppetproduction+16 -0
operations/puppetproduction+29 -8
operations/puppetproduction+60 -315
operations/puppetproduction+15 -5
operations/puppetproduction+78 -23
operations/puppetproduction+30 -79
operations/puppetproduction+63 -26
operations/puppetproduction+25 -10
operations/puppetproduction+347 -0
operations/puppetproduction+7 -0
operations/puppetproduction+2 -2
operations/puppetproduction+58 -5
operations/puppetproduction+2 -2
operations/puppetproduction+6 -1
operations/puppetproduction+125 -15
operations/puppetproduction+2 K -0
operations/puppetproduction+133 -0
operations/puppetproduction+14 -0
operations/puppetproduction+2 -0
operations/puppetproduction+5 -1
operations/puppetproduction+50 -25
operations/puppetproduction+44 -0
operations/puppetproduction+1 -0
operations/puppetproduction+3 -9
operations/puppetproduction+0 -10
operations/puppetproduction+4 -1
operations/puppetproduction+5 -0
operations/puppetproduction+1 -0
operations/dnsmaster+3 -0
operations/puppetproduction+25 -0
operations/puppetproduction+1 -1
operations/puppetproduction+2 -0
operations/puppetproduction+3 -1
operations/puppetproduction+19 -10
operations/puppetproduction+7 -5
operations/puppetproduction+535 -1
operations/puppetproduction+20 -18
operations/puppetproduction+15 K -7
operations/puppetproduction+129 -110
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
OpenNone
Resolvedjbond
OpenNone
Resolvedjbond
InvalidNone
Resolvedjbond
Resolvedjbond
ResolvedMoritzMuehlenhoff
Resolvedjbond
Resolvedjbond
Resolvedjbond
ResolvedABran-WMF
DeclinedBTullis
ResolvedLadsgroup
Resolveddcaro
Resolveddcaro
ResolvedAndrew
ResolvedNone
Resolvedbking
ResolvedNone
OpenNone
OpenNone
OpenNone
Resolvedjbond
Resolvedjbond
Openjhathaway
OpenNone
Resolvedjbond
DeclinedNone
Resolvedjbond
OpenNone
ResolvedVolans
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
StalledNone
OpenNone
OpenNone
OpenNone
Resolvedjbond
Resolvedjhathaway
ResolvedVRiley-WMF
ResolvedVRiley-WMF
ResolvedJhancock.wm
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
InvalidNone
ResolvedVolans
ResolvedVolans
Resolvedjbond
Resolvedjbond
DuplicateNone
Resolvedjbond
Resolvedjbond
OpenNone
StalledNone
DuplicateNone
ResolvedMoritzMuehlenhoff
ResolvedJhancock.wm
ResolvedVRiley-WMF
ResolvedMoritzMuehlenhoff
ResolvedBTullis
ResolvedDzahn
ResolvedDzahn
Resolvedandrea.denisse
ResolvedBTullis
Resolvedelukey
ResolvedCDobbins
ResolvedMoritzMuehlenhoff
ResolvedMoritzMuehlenhoff
ResolvedJgreen
ResolvedMoritzMuehlenhoff
ResolvedRequesthashar
Resolvedjhathaway
Resolvedjhathaway

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 935731 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetmaster::puppetdb: add documentation

https://gerrit.wikimedia.org/r/935731

Change 935731 merged by Jbond:

[operations/puppet@production] puppetmaster::puppetdb: add documentation

https://gerrit.wikimedia.org/r/935731

Change 937049 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetmaster: move source scripts under the puppetserver name space

https://gerrit.wikimedia.org/r/937049

Change 937049 merged by Jbond:

[operations/puppet@production] puppetmaster: move source scripts under the puppetserver name space

https://gerrit.wikimedia.org/r/937049

jbond changed the status of subtask T342125: puppetserver monitoring from Open to In Progress.Jul 18 2023, 2:19 PM

Change 939643 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: do not notify puppetserver service on changes

https://gerrit.wikimedia.org/r/939643

jbond changed the status of subtask T342125: puppetserver monitoring from In Progress to Stalled.

Change 940366 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] motd: Add motd indicating services which need restarting

https://gerrit.wikimedia.org/r/940366

Change 940365 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: Add a file to track when a service restart or reload is

https://gerrit.wikimedia.org/r/940365

Change 939643 merged by Jbond:

[operations/puppet@production] puppetserver: make notifying configurable

https://gerrit.wikimedia.org/r/939643

Change 940365 merged by Jbond:

[operations/puppet@production] puppetserver: Add a file to track when a service restart or reload is

https://gerrit.wikimedia.org/r/940365

Change 940366 merged by Jbond:

[operations/puppet@production] motd: Add motd indicating services which need restarting

https://gerrit.wikimedia.org/r/940366

Change 949980 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: dont auto restart puppet server

https://gerrit.wikimedia.org/r/949980

Change 949980 merged by Jbond:

[operations/puppet@production] puppetserver: dont auto restart puppet server

https://gerrit.wikimedia.org/r/949980

Change 919291 merged by Jbond:

[operations/puppet@production] httpyaml: replace URI.escape

https://gerrit.wikimedia.org/r/919291

Change 955731 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetmaster::servers: remove puppetserveres from puppetmaster::servers hash

https://gerrit.wikimedia.org/r/955731

Change 955731 merged by Jbond:

[operations/puppet@production] puppetmaster::servers: remove puppetservers from puppetmaster::servers hash

https://gerrit.wikimedia.org/r/955731

Change 955911 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetmaster: add pupetserveres back to git private

https://gerrit.wikimedia.org/r/955911

Change 955911 merged by Jbond:

[operations/puppet@production] puppetmaster: add pupetserveres back to git private

https://gerrit.wikimedia.org/r/955911

Change 961153 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] cumin: Add puppetserver alias

https://gerrit.wikimedia.org/r/961153

Change 961153 merged by Jbond:

[operations/puppet@production] cumin: Add puppetserver alias

https://gerrit.wikimedia.org/r/961153

Change 962638 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: add logstash reporter

https://gerrit.wikimedia.org/r/962638

Change 962638 merged by Jbond:

[operations/puppet@production] puppetserver: add logstash reporter

https://gerrit.wikimedia.org/r/962638

@jbond I think that the decommission cookbook needs some adjustment too, both because it checks some git checkout on the puppetmaster's CA and also because it does remove the certificate.

Change #1073906 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] ci: upgrade to puppet 7

https://gerrit.wikimedia.org/r/1073906

Change #1073906 merged by JHathaway:

[operations/puppet@production] ci: upgrade to puppet 7

https://gerrit.wikimedia.org/r/1073906

Change #1074534 had a related patch set uploaded (by JHathaway; author: JHathaway):

[integration/config@master] operations-puppet: re-bundle to grab the updates in the Gemfile for Puppet 7

https://gerrit.wikimedia.org/r/1074534

Change #1074534 merged by jenkins-bot:

[integration/config@master] operations-puppet: re-bundle to grab the updates in the Gemfile for Puppet 7

https://gerrit.wikimedia.org/r/1074534

Mentioned in SAL (#wikimedia-operations) [2024-09-23T08:25:49Z] <hashar> Updated CI job operations-puppet-tests-bullseye to image rebuild for Puppet 7 # T330490

Change #1100161 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] puppet 7: fix facter.conf location

https://gerrit.wikimedia.org/r/1100161

Change #1100161 merged by JHathaway:

[operations/puppet@production] puppet 7: fix facter.conf location

https://gerrit.wikimedia.org/r/1100161

Change #1100490 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] facter: fix facter conf location

https://gerrit.wikimedia.org/r/1100490

Change #1100490 merged by JHathaway:

[operations/puppet@production] facter: fix facter conf location

https://gerrit.wikimedia.org/r/1100490

Change #1100861 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] facter: fix facter conf location, attempt 3

https://gerrit.wikimedia.org/r/1100861

Change #1100861 merged by JHathaway:

[operations/puppet@production] facter: fix facter conf location, attempt 3

https://gerrit.wikimedia.org/r/1100861