Page MenuHomePhabricator
Create Task
Maniphest T321783

Setup an initial bookworm host pair with Puppetdb 7
Closed, ResolvedPublic
Actions

Description

We want to move to puppetdb 7 and initially run it on Bookworm. This will still see quite a bit of changes given it's not frozen yet, but backporting puppetdb 7 to the Clojure stack in Bullseye is also a significant undertaking (and this way we can also help with getting puppetdb in shape in Debian).

The Debian installer isn't yet adapted to install Bookworm in our infra, so initially a bullseye node will be dist-upgraded (and minimal puppet changes deployed to support the core base classes).

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
puppetdb: add classification backoperations/puppetproduction+1 -1
puppetmaster: add new puppetsrveroperations/puppetproduction+1 -0
puppetserver: codfw update puppetdb urloperations/puppetproduction+1 -1
pupetserver::git: Ensure we build the git repo if using initoperations/puppetproduction+8 -4
pupetserver::git: ensure we build all parent directoriesoperations/puppetproduction+3 -2
puppetdb2003: move to new roleoperations/puppetproduction+1 -6
package_builder: add hooks for bookwormoperations/puppetproduction+6 -0
httpd: Let Puppet pick the init provideroperations/puppetproduction+0 -1
Enable command_broadcast to the new puppetdb 7 hostsoperations/puppetproduction+5 -1
nginx: Drop require for the nginx packageoperations/puppetproduction+2 -4
Only set profile::nginx::variant to custom for the new bookworm nodesoperations/puppetproduction+3 -1
Switch role::puppetdb to Nginx custom flavouroperations/puppetproduction+1 -1
Reapply puppetdb roleoperations/puppetproduction+1 -1
Reset puppetdb1003/2003 to insetupoperations/puppetproduction+2 -2
puppetdb: use new allowed_hosts paramater to postgresql:useroperations/puppetproduction+27 -16
postgres::user: add hostname support to postgres user defineoperations/puppetproduction+38 -19
postgrers::user::hba: drop hba_label and use title insteadoperations/puppetproduction+3 -8
postgresql::user: add documentation and fix minor lint errorsoperations/puppetproduction+18 -19
monitoring: convert prometheus-puppet-agent-stats to pathliboperations/puppetproduction+23 -16
augeas_core: add augeas core module to the vendor modulesoperations/puppetproduction+4 K -0
puppet: install puppet-module-puppetlabs-augeas-core on puppet >= 6operations/puppetproduction+4 -0
puppetdb: add auth.conf fileoperations/puppetproduction+61 -0
Decom puppetdb-test2001operations/puppetproduction+0 -11
puppetdb/bookworm: One more typo in the configoperations/puppetproduction+2 -2
Properly name replication records for second puppetdb pairoperations/puppetproduction+6 -6
monitoring: update monitoring files to dynamically discover configoperations/puppetproduction+49 -15
Mask uwsgi on puppetdb hostsoperations/puppetproduction+9 -0
monitoring: update monitoring files to dynamically discover configoperations/puppetproduction+22 -13
nginx: let puppet pick the correct provideroperations/puppetproduction+0 -1
Don't install quickstack on Bookworm, revisit lateroperations/puppetproduction+6 -1
uwsgi: Add support for bookwormoperations/puppetproduction+1 -0
postgresql::server: Add bookworm supportoperations/puppetproduction+3 -0
Add Hiera settings for second bookworm puppetdb pairoperations/puppetproduction+79 -1
postgresql: Add bookworm supportoperations/puppetproduction+2 -0
profile::java: Add support for bookwormoperations/puppetproduction+1 -0
puppetdb::database: Add support for bookwormoperations/puppetproduction+1 -0
puppetdb: On bookworm install from component/puppetdb7operations/puppetproduction+10 -1
Add component/puppetdb7 for bookworm-wikimediaoperations/puppetproduction+1 -0
prometheus: use default for ignored_devicesoperations/puppetproduction+1 -8
Add support for bookworm to apt.wikimedia.orgoperations/puppetproduction+31 -0
debian: Add bookwormoperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT330490 Next steps for Puppet 7
 ResolvedjbondT321783 Setup an initial bookworm host pair with Puppetdb 7

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Nov 29 2022, 9:17 AM

Mentioned in SAL (#wikimedia-operations) [2022-11-29T09:17:13Z] <moritzm> update component/puppetdb7 to puppetdb 7.11.2-3 (fixing Postgres 15 compat) T321783

Stashbot added a comment.Nov 29 2022, 10:15 AM

Mentioned in SAL (#wikimedia-operations) [2022-11-29T10:15:49Z] <moritzm> upgrading puppetdb2003 to bookworm T321783

Stashbot added a comment.Nov 29 2022, 11:37 AM

Mentioned in SAL (#wikimedia-operations) [2022-11-29T11:37:43Z] <moritzm> uploaded ferm 2.5.1-1.1+wmf11u1 to apt.wikimedia.org/bookworm (rebasing our systemd startup fixes to what's in bookworm) T321783

Stashbot added a comment.Nov 30 2022, 11:19 AM

Mentioned in SAL (#wikimedia-operations) [2022-11-30T11:19:53Z] <moritzm> upgrade puppetdb1003 to bookworm T321783

gerritbot added a comment.Nov 30 2022, 1:14 PM

Change 862256 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add Hiera settings for second bookworm puppetdb pair

https://gerrit.wikimedia.org/r/862256

gerritbot added a comment.Nov 30 2022, 1:27 PM

Change 862260 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] postgresql: Add bookworm support

https://gerrit.wikimedia.org/r/862260

gerritbot added a comment.Dec 1 2022, 12:03 PM

Change 862260 merged by Muehlenhoff:

[operations/puppet@production] postgresql: Add bookworm support

https://gerrit.wikimedia.org/r/862260

gerritbot added a comment.Dec 1 2022, 12:30 PM

Change 862256 merged by Muehlenhoff:

[operations/puppet@production] Add Hiera settings for second bookworm puppetdb pair

https://gerrit.wikimedia.org/r/862256

gerritbot added a comment.Dec 2 2022, 12:38 PM

Change 863286 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] postgresql::server: Add bookworm support

https://gerrit.wikimedia.org/r/863286

gerritbot added a comment.Dec 5 2022, 8:25 AM

Change 863286 merged by Muehlenhoff:

[operations/puppet@production] postgresql::server: Add bookworm support

https://gerrit.wikimedia.org/r/863286

gerritbot added a comment.Dec 5 2022, 9:16 AM

Change 864664 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] uwsgi: Add support for bookworm

https://gerrit.wikimedia.org/r/864664

gerritbot added a comment.Dec 5 2022, 11:23 AM

Change 864664 merged by Muehlenhoff:

[operations/puppet@production] uwsgi: Add support for bookworm

https://gerrit.wikimedia.org/r/864664

gerritbot added a comment.Dec 14 2022, 12:56 PM

Change 868078 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Don't install quickstack on Bookworm, revisit later

https://gerrit.wikimedia.org/r/868078

gerritbot added a comment.Dec 15 2022, 7:53 AM

Change 868078 merged by Muehlenhoff:

[operations/puppet@production] Don't install quickstack on Bookworm, revisit later

https://gerrit.wikimedia.org/r/868078

Stashbot added a comment.Dec 15 2022, 3:05 PM

Mentioned in SAL (#wikimedia-operations) [2022-12-15T15:05:02Z] <moritzm> imported prometheus-jmx-exporter for bookworm-wikimedia T321783

jbond added a comment.Dec 15 2022, 3:18 PM

upstream bug re java 11

gerritbot added a comment.Dec 15 2022, 3:39 PM

Change 868431 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] nginx: let puppet pick the correct provider

https://gerrit.wikimedia.org/r/868431

gerritbot added a comment.Dec 15 2022, 9:40 PM

Change 868471 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] monitoring: update monitoring files to dynamically discover config

https://gerrit.wikimedia.org/r/868471

gerritbot added a comment.Dec 19 2022, 12:06 PM

Change 868431 merged by Muehlenhoff:

[operations/puppet@production] nginx: let puppet pick the correct provider

https://gerrit.wikimedia.org/r/868431

gerritbot added a comment.Dec 19 2022, 12:16 PM

Change 869199 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] httpd: Let Puppet pick the init provider

https://gerrit.wikimedia.org/r/869199

gerritbot added a comment.Dec 20 2022, 10:59 AM

Change 868471 merged by Jbond:

[operations/puppet@production] monitoring: update monitoring files to dynamically discover config

https://gerrit.wikimedia.org/r/868471

gerritbot added a comment.Dec 21 2022, 11:47 AM

Change 869716 had a related patch set uploaded (by Jbond; author: Jbond):

[operations/puppet@production] monitoring: update monitoring files to dynamically discover config

https://gerrit.wikimedia.org/r/869716

gerritbot added a comment.Jan 3 2023, 2:15 PM

Change 874852 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Mask uwsgi on puppetdb hosts

https://gerrit.wikimedia.org/r/874852

gerritbot added a comment.Jan 3 2023, 3:16 PM

Change 874891 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] monitoring: convert prometheus-puppet-agent-stats to pathlib

https://gerrit.wikimedia.org/r/874891

gerritbot added a comment.Jan 4 2023, 8:42 AM

Change 874852 merged by Muehlenhoff:

[operations/puppet@production] Mask uwsgi on puppetdb hosts

https://gerrit.wikimedia.org/r/874852

gerritbot added a comment.Jan 4 2023, 10:17 AM

Change 875272 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Properly name replication records for second puppetdb pair

https://gerrit.wikimedia.org/r/875272

gerritbot added a comment.Jan 4 2023, 10:25 AM

Change 869716 merged by Jbond:

[operations/puppet@production] monitoring: update monitoring files to dynamically discover config

https://gerrit.wikimedia.org/r/869716

gerritbot added a comment.Jan 4 2023, 11:43 AM

Change 875272 merged by Muehlenhoff:

[operations/puppet@production] Properly name replication records for second puppetdb pair

https://gerrit.wikimedia.org/r/875272

gerritbot added a comment.Jan 4 2023, 12:29 PM

Change 875301 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] puppetdb/bookworm: One more typo in the config

https://gerrit.wikimedia.org/r/875301

gerritbot added a comment.Jan 4 2023, 3:04 PM

Change 875301 merged by Muehlenhoff:

[operations/puppet@production] puppetdb/bookworm: One more typo in the config

https://gerrit.wikimedia.org/r/875301

gerritbot added a comment.Jan 10 2023, 12:17 PM

Change 878001 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Decom puppetdb-test2001

https://gerrit.wikimedia.org/r/878001

gerritbot added a comment.Jan 10 2023, 12:30 PM

Change 878001 merged by Muehlenhoff:

[operations/puppet@production] Decom puppetdb-test2001

https://gerrit.wikimedia.org/r/878001

ops-monitoring-bot added a comment.Jan 10 2023, 1:08 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: puppetdb-test2001.codfw.wmnet

  • puppetdb-test2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
gerritbot added a comment.Jan 24 2023, 2:27 PM

Change 883184 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppetdb: add auth.conf file

https://gerrit.wikimedia.org/r/883184

gerritbot added a comment.Jan 24 2023, 2:31 PM

Change 883184 merged by Jbond:

[operations/puppet@production] puppetdb: add auth.conf file

https://gerrit.wikimedia.org/r/883184

gerritbot added a comment.Jan 24 2023, 3:03 PM

Change 883189 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppet: install puppet-module-puppetlabs-augeas-core on puppet >= 6

https://gerrit.wikimedia.org/r/883189

gerritbot added a comment.Jan 24 2023, 3:30 PM

Change 883189 merged by Jbond:

[operations/puppet@production] puppet: install puppet-module-puppetlabs-augeas-core on puppet >= 6

https://gerrit.wikimedia.org/r/883189

gerritbot added a comment.Jan 24 2023, 4:40 PM

Change 883233 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] augeas_core: add augeas core module to the vendor modules

https://gerrit.wikimedia.org/r/883233

gerritbot added a comment.Jan 25 2023, 12:59 PM

Change 883233 merged by Jbond:

[operations/puppet@production] augeas_core: add augeas core module to the vendor modules

https://gerrit.wikimedia.org/r/883233

Stashbot added a comment.Jan 30 2023, 2:47 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-30T14:47:40Z] <moritzm> updating puppetdb 7 hosts to 7.12.1 T321783

gerritbot added a comment.Feb 6 2023, 11:22 AM

Change 874891 merged by Jbond:

[operations/puppet@production] monitoring: convert prometheus-puppet-agent-stats to pathlib

https://gerrit.wikimedia.org/r/874891

gerritbot added a comment.Feb 6 2023, 1:58 PM

Change 886895 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] postgresql::user: add documentation and fix minor lint errors

https://gerrit.wikimedia.org/r/886895

gerritbot added a comment.Feb 6 2023, 2:11 PM

Change 886897 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] postgres::user: add hostname support to postgres user define

https://gerrit.wikimedia.org/r/886897

gerritbot added a comment.Feb 6 2023, 2:12 PM

Change 886895 merged by Jbond:

[operations/puppet@production] postgresql::user: add documentation and fix minor lint errors

https://gerrit.wikimedia.org/r/886895

gerritbot added a comment.Feb 6 2023, 2:45 PM

Change 886900 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppetdb: use new allowed_hosts paramater to postgresql:user

https://gerrit.wikimedia.org/r/886900

gerritbot added a comment.Feb 6 2023, 3:41 PM

Change 886912 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] postgrers::user::hba: drop hba_label and use title instead

https://gerrit.wikimedia.org/r/886912

taavi mentioned this in T185815: The Rack Puppet master server is deprecated and will be removed in a future release. Please use Puppet Server instead. .Feb 7 2023, 9:38 AM
gerritbot added a comment.Feb 7 2023, 11:41 AM

Change 886912 merged by Jbond:

[operations/puppet@production] postgrers::user::hba: drop hba_label and use title instead

https://gerrit.wikimedia.org/r/886912

gerritbot added a comment.Feb 7 2023, 12:08 PM

Change 886897 merged by Jbond:

[operations/puppet@production] postgres::user: add hostname support to postgres user define

https://gerrit.wikimedia.org/r/886897

gerritbot added a comment.Feb 7 2023, 1:37 PM

Change 886900 merged by Jbond:

[operations/puppet@production] puppetdb: use new allowed_hosts paramater to postgresql:user

https://gerrit.wikimedia.org/r/886900

ops-monitoring-bot added a comment.Feb 9 2023, 10:59 AM

Icinga downtime and Alertmanager silence (ID=b1f3dbef-467c-49de-8608-5ba564efbe81) set by jmm@cumin2002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: master is being reimaged

puppetdb2003.codfw.wmnet
gerritbot added a comment.Feb 9 2023, 11:08 AM

Change 887971 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Reset puppetdb1003/2003 to insetup

https://gerrit.wikimedia.org/r/887971

gerritbot added a comment.Feb 9 2023, 11:11 AM

Change 887971 merged by Muehlenhoff:

[operations/puppet@production] Reset puppetdb1003/2003 to insetup

https://gerrit.wikimedia.org/r/887971

ops-monitoring-bot added a comment.Feb 9 2023, 11:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host puppetdb1003.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 12:10 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host puppetdb1003.eqiad.wmnet with OS bullseye completed:

  • puppetdb1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091140_jmm_991175_puppetdb1003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 9 2023, 12:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host puppetdb2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 12:46 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host puppetdb2003.codfw.wmnet with OS bullseye completed:

  • puppetdb2003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091212_jmm_998112_puppetdb2003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 10 2023, 11:05 AM

Mentioned in SAL (#wikimedia-operations) [2023-02-10T11:05:20Z] <moritzm> upgrade puppetdb[12]003 to bookworm T321783

gerritbot added a comment.Feb 10 2023, 1:34 PM

Change 888218 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Reapply puppetdb role

https://gerrit.wikimedia.org/r/888218

gerritbot added a comment.Feb 10 2023, 1:46 PM

Change 888218 merged by Muehlenhoff:

[operations/puppet@production] Reapply puppetdb role

https://gerrit.wikimedia.org/r/888218

gerritbot added a comment.Feb 16 2023, 4:14 PM

Change 889817 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Enable command_broadcast to the new puppetdb 7 hosts

https://gerrit.wikimedia.org/r/889817

gerritbot added a comment.Feb 20 2023, 2:40 PM

Change 890439 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch role::puppetdb to Nginx custom flavour

https://gerrit.wikimedia.org/r/890439

gerritbot added a comment.Feb 23 2023, 8:39 AM

Change 890439 merged by Muehlenhoff:

[operations/puppet@production] Switch role::puppetdb to Nginx custom flavour

https://gerrit.wikimedia.org/r/890439

gerritbot added a comment.Feb 23 2023, 8:42 AM

Change 891487 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Only set profile::nginx::variant to custom for the new bookworm nodes

https://gerrit.wikimedia.org/r/891487

gerritbot added a comment.Feb 23 2023, 8:44 AM

Change 891487 merged by Muehlenhoff:

[operations/puppet@production] Only set profile::nginx::variant to custom for the new bookworm nodes

https://gerrit.wikimedia.org/r/891487

gerritbot added a comment.Feb 23 2023, 8:55 AM

Change 891489 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] nginx: Drop require for the nginx package

https://gerrit.wikimedia.org/r/891489

gerritbot added a comment.Feb 23 2023, 11:20 AM

Change 891489 merged by Muehlenhoff:

[operations/puppet@production] nginx: Drop require for the nginx package

https://gerrit.wikimedia.org/r/891489

MoritzMuehlenhoff added a parent task: T330490: Next steps for Puppet 7.Feb 24 2023, 9:48 AM
gerritbot added a comment.Mar 8 2023, 11:23 AM

Change 889817 abandoned by Muehlenhoff:

[operations/puppet@production] Enable command_broadcast to the new puppetdb 7 hosts

Reason:

No longer needed, new plan outlined at https://phabricator.wikimedia.org/T330490

https://gerrit.wikimedia.org/r/889817

gerritbot added a comment.Apr 19 2023, 8:41 AM

Change 869199 abandoned by Muehlenhoff:

[operations/puppet@production] httpd: Let Puppet pick the init provider

Reason:

Already merged in 11feac29ed0910a7b54ad506826dc2568281ff7d

https://gerrit.wikimedia.org/r/869199

Maintenance_bot removed a project: Patch-For-Review.Apr 19 2023, 9:11 AM
gerritbot added a comment.Apr 27 2023, 11:46 AM

Change 912835 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] package_builder: add hooks for bookworm

https://gerrit.wikimedia.org/r/912835

gerritbot added a project: Patch-For-Review.Apr 27 2023, 11:46 AM
gerritbot added a comment.Apr 27 2023, 11:55 AM

Change 912835 merged by Jbond:

[operations/puppet@production] package_builder: add hooks for bookworm

https://gerrit.wikimedia.org/r/912835

ops-monitoring-bot added a comment.May 30 2023, 7:49 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host puppetdb2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.May 30 2023, 9:29 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host puppetdb2003.codfw.wmnet with OS bookworm completed:

  • puppetdb2003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202305300811_jmm_3834209_puppetdb2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.May 30 2023, 9:30 AM
ops-monitoring-bot added a comment.May 30 2023, 9:42 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host puppetdb1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.May 30 2023, 2:16 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host puppetdb1003.eqiad.wmnet with OS bookworm executed with errors:

  • puppetdb1003 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202305300958_jmm_3953981_puppetdb1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Jul 4 2023, 4:32 PM

Change 935478 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetdb2003: move to new role

https://gerrit.wikimedia.org/r/935478

gerritbot added a project: Patch-For-Review.Jul 4 2023, 4:32 PM

Change 935478 merged by Jbond:

[operations/puppet@production] puppetdb2003: move to new role

https://gerrit.wikimedia.org/r/935478

gerritbot added a comment.Jul 4 2023, 4:54 PM

Change 935480 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] pupetserver::git: ensure we build all parent directories

https://gerrit.wikimedia.org/r/935480

gerritbot added a comment.Jul 4 2023, 5:13 PM

Change 935480 merged by Jbond:

[operations/puppet@production] pupetserver::git: ensure we build all parent directories

https://gerrit.wikimedia.org/r/935480

gerritbot added a comment.Jul 4 2023, 5:20 PM

Change 935483 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] pupetserver::git: Ensure we build the git repo if using init

https://gerrit.wikimedia.org/r/935483

gerritbot added a comment.Jul 4 2023, 5:41 PM

Change 935483 merged by Jbond:

[operations/puppet@production] pupetserver::git: Ensure we build the git repo if using init

https://gerrit.wikimedia.org/r/935483

gerritbot added a comment.Jul 4 2023, 5:59 PM

Change 935507 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetserver: codfw update puppetdb url

https://gerrit.wikimedia.org/r/935507

gerritbot added a comment.Jul 4 2023, 6:02 PM

Change 935507 merged by Jbond:

[operations/puppet@production] puppetserver: codfw update puppetdb url

https://gerrit.wikimedia.org/r/935507

gerritbot added a comment.Jul 4 2023, 6:16 PM

Change 935509 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetmaster: add new puppetsrver

https://gerrit.wikimedia.org/r/935509

gerritbot added a comment.Jul 4 2023, 6:21 PM

Change 935509 merged by Jbond:

[operations/puppet@production] puppetmaster: add new puppetsrver

https://gerrit.wikimedia.org/r/935509

Maintenance_bot removed a project: Patch-For-Review.Jul 4 2023, 6:30 PM
gerritbot added a comment.Jul 4 2023, 7:33 PM

Change 935514 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] puppetdb: add classification back

https://gerrit.wikimedia.org/r/935514

gerritbot added a project: Patch-For-Review.Jul 4 2023, 7:33 PM
gerritbot added a comment.Jul 4 2023, 7:38 PM

Change 935514 merged by Jbond:

[operations/puppet@production] puppetdb: add classification back

https://gerrit.wikimedia.org/r/935514

jbond closed this task as Resolved.Jul 7 2023, 3:33 PM
jbond claimed this task.

We now have puppetserver, db and puppetboard running on both codfw and eqiad

Maintenance_bot removed a project: Patch-For-Review.Jul 7 2023, 4:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits