Page MenuHomePhabricator

Next steps for Puppet 7
Open, MediumPublic

Description

The Puppet maintainers in Debian have done amazing work and Bookworm will introduce a current stack:

Based on a discussion with John and myself the plan to move towards Puppet 7 would leads towards installing the new setup in parallel:

  • We already have a pair of puppetdb servers running on Bookworm on new hardware (puppetdb1003/puppetdb2003). With this there's also general compability of our Puppet tree with current facter and puppet-agent (there will be a need for more fine-tuning and some uses cases used by other roles, but the general support is present)
  • There's new hardware (puppetmaster[12]004/[12]005) meant to replace the old puppetmaster[12]00[12] servers. We can install puppetmaster[12]005 with Bookworm as the new puppetserver 7 frontends and puppetmaster[12]004 as the new puppetserver backends,
  • Create a separate Ganeti VM for puppetboard and serve is under a distinct URL like puppetboard7.wikimedia.org
  • Modify puppet-merge on puppetmaster[12]001 to sync every puppet change also towards puppetmaster[12]00[45]
  • Configure an additional/separate puppetdb 7 backend in Cumin
  • Modify the agent configuration to use a separate puppet7 CNAME (and create it)
  • Write a cookbook to remove a host from the Puppet 5 masters, add them for the Puppet 7 masters and modify agent config to use the puppet7 CNAME
  • Modify PCC to also compile every patch with Puppet 7 in parallel

Once there is two setups in parallel we can:

  • Use the cookbook to move the sretest* servers (and some other hosts like idp-test and some other canaries towards Puppet 7 (Buster/Bullseye/Bookworm on the agent side), test and fix up all issues
  • Use the cookbook to move ulsfo towards the Puppet 7 setup (and observe/fix for some time)
  • Use the cookbook to move all the remaining servers to Puppet 7
  • Migrate config-master over to puppetmaster1005
  • Reimage puppetmaster[12]003 with Bookworm
  • Decom puppetmaster[12]00[12] and the old puppetboard VM

Blockers To an upgrade

puppet forge core type

when puppet version 6 was released a bunch of core resource types where removed from the puppet code base and spun of as external forge modules. We should be aware of any resource we are using which fall into this catagory and either migrate to a different module/type or add the core modules to our code base when we upgrade

  • migrate all cron types to systemd::timer::job
  • if still using nagios migrate to puppetlabs/nagios_core as nagios types are no longer core puppet
  • Check if which other native types need to be migrated

Benefits to explore (some of theses benefits have be realized by updating to the latest 5.5 branch)

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+6 -2
operations/puppetproduction+3 -21
operations/puppetproduction+2 -1
operations/puppetproduction+1 -0
operations/puppetproduction+8 -2
operations/puppetproduction+16 -0
operations/puppetproduction+29 -8
operations/puppetproduction+60 -315
operations/puppetproduction+15 -5
operations/puppetproduction+78 -23
operations/puppetproduction+30 -79
operations/puppetproduction+63 -26
operations/puppetproduction+25 -10
operations/puppetproduction+347 -0
operations/puppetproduction+7 -0
operations/puppetproduction+2 -2
operations/puppetproduction+58 -5
operations/puppetproduction+2 -2
operations/puppetproduction+6 -1
operations/puppetproduction+125 -15
operations/puppetproduction+2 K -0
operations/puppetproduction+133 -0
operations/puppetproduction+14 -0
operations/puppetproduction+2 -0
operations/puppetproduction+5 -1
operations/puppetproduction+50 -25
operations/puppetproduction+44 -0
operations/puppetproduction+1 -0
operations/puppetproduction+3 -9
operations/puppetproduction+0 -10
operations/puppetproduction+4 -1
operations/puppetproduction+5 -0
operations/puppetproduction+1 -0
operations/dnsmaster+3 -0
operations/puppetproduction+25 -0
operations/puppetproduction+1 -1
operations/puppetproduction+2 -0
operations/puppetproduction+3 -1
operations/puppetproduction+19 -10
operations/puppetproduction+7 -5
operations/puppetproduction+535 -1
operations/puppetproduction+20 -18
operations/puppetproduction+15 K -7
operations/puppetproduction+129 -110
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
OpenNone
Resolvedjbond
OpenNone
Resolvedjbond
InvalidNone
Resolvedjbond
Resolvedjbond
In ProgressNone
Resolvedjbond
Resolvedjbond
Resolvedjbond
OpenNone
Resolvedfgiunchedi
OpenNone
OpenNone
ResolvedABran-WMF
DeclinedBTullis
ResolvedLadsgroup
In Progressdcaro
Resolvedbking
In ProgressNone
OpenNone
OpenNone
OpenNone
Resolvedjbond
Resolvedjbond
OpenNone
OpenNone
Resolvedjbond
DeclinedNone
Resolvedjbond
OpenNone
OpenNone
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
StalledNone
In ProgressNone
Resolvedjbond
In Progressjhathaway
ResolvedVRiley-WMF
ResolvedVRiley-WMF
ResolvedJhancock.wm
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
Resolvedjbond
InvalidNone
ResolvedVolans
ResolvedVolans
Resolvedjbond
Resolvedjbond
DuplicateNone
Resolvedjbond
Resolvedjbond
OpenNone
StalledNone
DuplicateNone
ResolvedMoritzMuehlenhoff
ResolvedJhancock.wm
ResolvedVRiley-WMF
OpenMoritzMuehlenhoff
ResolvedBTullis
ResolvedDzahn
In Progressandrea.denisse
OpenNone
OpenBTullis
OpenNone
OpenNone
OpenNone
Openjijiki
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 928628 merged by Jbond:

[operations/puppet@production] puppetserver: Add private repo configurations

https://gerrit.wikimedia.org/r/928628

Change 931659 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: use sudo to run g10k from post-merge

https://gerrit.wikimedia.org/r/931659

Change 931659 merged by Jbond:

[operations/puppet@production] puppetserver: use sudo to run g10k from post-merge

https://gerrit.wikimedia.org/r/931659

Change 931661 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: fix hook sources

https://gerrit.wikimedia.org/r/931661

Change 931661 merged by Jbond:

[operations/puppet@production] puppetserver: fix hook sources

https://gerrit.wikimedia.org/r/931661

Change 931673 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: add merge_cli

https://gerrit.wikimedia.org/r/931673

Change 931673 merged by Jbond:

[operations/puppet@production] puppetserver: add merge_cli as a seperate module

https://gerrit.wikimedia.org/r/931673

Change 931681 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] merge_cli: fix source paths

https://gerrit.wikimedia.org/r/931681

Change 931681 merged by Jbond:

[operations/puppet@production] merge_cli: fix source paths

https://gerrit.wikimedia.org/r/931681

Change 933440 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: open up firewall rules

https://gerrit.wikimedia.org/r/933440

Change 933457 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: add pcc facts upload functionality

https://gerrit.wikimedia.org/r/933457

Change 933440 merged by Jbond:

[operations/puppet@production] puppetserver: open up firewall rules

https://gerrit.wikimedia.org/r/933440

Change 933457 merged by Jbond:

[operations/puppet@production] puppetserver: add pcc facts upload functionality

https://gerrit.wikimedia.org/r/933457

Change 933459 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: Add support for providing hardcoded list of servers

https://gerrit.wikimedia.org/r/933459

Change 933459 merged by Jbond:

[operations/puppet@production] puppetserver: Add support for providing hardcoded list of servers

https://gerrit.wikimedia.org/r/933459

Change 933485 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] merge_cli: Make the paths a parameter and add them to a config file

https://gerrit.wikimedia.org/r/933485

Change 933485 merged by Jbond:

[operations/puppet@production] merge_cli: Make the paths a parameter and add them to a config file

https://gerrit.wikimedia.org/r/933485

Change 933422 had a related patch set uploaded (by Jbond; author: Jbond):

[operations/puppet@production] merge_cli: Make the paths a parameter and add them to a config file

https://gerrit.wikimedia.org/r/933422

Change 933608 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetdb: Add ability to configure secondary proxies

https://gerrit.wikimedia.org/r/933608

Change 933887 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] merge_cli: migrate puppetmasteres to module version of merge_cli

https://gerrit.wikimedia.org/r/933887

Change 933887 merged by Jbond:

[operations/puppet@production] merge_cli: migrate puppetmasteres to module version of merge_cli

https://gerrit.wikimedia.org/r/933887

Change 933422 merged by Jbond:

[operations/puppet@production] merge_cli: Make the paths a parameter and add them to a config file

https://gerrit.wikimedia.org/r/933422

Change 935731 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetmaster::puppetdb: add documentation

https://gerrit.wikimedia.org/r/935731

Change 935731 merged by Jbond:

[operations/puppet@production] puppetmaster::puppetdb: add documentation

https://gerrit.wikimedia.org/r/935731

Change 937049 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetmaster: move source scripts under the puppetserver name space

https://gerrit.wikimedia.org/r/937049

Change 937049 merged by Jbond:

[operations/puppet@production] puppetmaster: move source scripts under the puppetserver name space

https://gerrit.wikimedia.org/r/937049

jbond changed the status of subtask T342125: puppetserver monitoring from Open to In Progress.Jul 18 2023, 2:19 PM

Change 939643 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: do not notify puppetserver service on changes

https://gerrit.wikimedia.org/r/939643

jbond changed the status of subtask T342125: puppetserver monitoring from In Progress to Stalled.

Change 940366 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] motd: Add motd indicating services which need restarting

https://gerrit.wikimedia.org/r/940366

Change 940365 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: Add a file to track when a service restart or reload is

https://gerrit.wikimedia.org/r/940365

Change 939643 merged by Jbond:

[operations/puppet@production] puppetserver: make notifying configurable

https://gerrit.wikimedia.org/r/939643

Change 940365 merged by Jbond:

[operations/puppet@production] puppetserver: Add a file to track when a service restart or reload is

https://gerrit.wikimedia.org/r/940365

Change 940366 merged by Jbond:

[operations/puppet@production] motd: Add motd indicating services which need restarting

https://gerrit.wikimedia.org/r/940366

Change 949980 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: dont auto restart puppet server

https://gerrit.wikimedia.org/r/949980

Change 949980 merged by Jbond:

[operations/puppet@production] puppetserver: dont auto restart puppet server

https://gerrit.wikimedia.org/r/949980

Change 919291 merged by Jbond:

[operations/puppet@production] httpyaml: replace URI.escape

https://gerrit.wikimedia.org/r/919291

Change 955731 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetmaster::servers: remove puppetserveres from puppetmaster::servers hash

https://gerrit.wikimedia.org/r/955731

Change 955731 merged by Jbond:

[operations/puppet@production] puppetmaster::servers: remove puppetservers from puppetmaster::servers hash

https://gerrit.wikimedia.org/r/955731

Change 955911 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetmaster: add pupetserveres back to git private

https://gerrit.wikimedia.org/r/955911

Change 955911 merged by Jbond:

[operations/puppet@production] puppetmaster: add pupetserveres back to git private

https://gerrit.wikimedia.org/r/955911

Change 961153 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] cumin: Add puppetserver alias

https://gerrit.wikimedia.org/r/961153

Change 961153 merged by Jbond:

[operations/puppet@production] cumin: Add puppetserver alias

https://gerrit.wikimedia.org/r/961153

Change 962638 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: add logstash reporter

https://gerrit.wikimedia.org/r/962638

Change 962638 merged by Jbond:

[operations/puppet@production] puppetserver: add logstash reporter

https://gerrit.wikimedia.org/r/962638

@jbond I think that the decommission cookbook needs some adjustment too, both because it checks some git checkout on the puppetmaster's CA and also because it does remove the certificate.