Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • The reimage failed, see the cookbook logs for the details

Change 908789 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Use signed-by notation for component/puppet5

https://gerrit.wikimedia.org/r/908789

Change 908789 merged by Muehlenhoff:

[operations/puppet@production] Use signed-by notation for component/puppet5

https://gerrit.wikimedia.org/r/908789

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • The reimage failed, see the cookbook logs for the details

Change 908799 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Pass -y --force-yes to puppet installation on bookworm

https://gerrit.wikimedia.org/r/908799

Change 908799 merged by Muehlenhoff:

[operations/puppet@production] Pass -y --force-yes to puppet installation on bookworm

https://gerrit.wikimedia.org/r/908799

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • The reimage failed, see the cookbook logs for the details

Change 908833 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Install ruby-sorted-set on Bookworm

https://gerrit.wikimedia.org/r/908833

Change 908833 merged by Muehlenhoff:

[operations/puppet@production] Install ruby-sorted-set on Bookworm

https://gerrit.wikimedia.org/r/908833

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304170713_jmm_720569_sretest1002.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • The reimage failed, see the cookbook logs for the details

late-setup.sh has been modified to integrate the Puppet 5.5 forward port during the Debian installer and that makes the installation work!

There's still the caveat that the reimage cookbook doesn't fully complete until the very end since the Puppet run of the rebooted system fails due to the YAML issue mentioned in https://phabricator.wikimedia.org/T330495#8771692, but the installation itself (and the host cert renewal) are working fine now.

In T330495#8771692, @MoritzMuehlenhoff wrote:

This leaves one issue related to YAML parsing which got stricter, similar to https://tickets.puppetlabs.com/browse/PUP-9506. As an affect /var/lib/puppet/state/state.yaml s considered corrupted and recreated with every Puppet run. Given that this setup is an interim until we're on Puppet 7 in general, we can live with that for now. I'll import this forward port to component/puppet5, next step is to fixup late-setup.sh to use it during Bookworm installs instead of the agent from Bookworm.

We can either leave it or add one of two patches on top of the foward port.

1. Use YAML.unsafe_load_file, which shouldn't be any worse than our present yaml safety
2. Backport Puppet's safe_load which selectively unmarshal's classes

Both patches are attached and were tested on puppetdb1003. I think the unsafe load is probably fine for our interim state?

In T330495#8787624, @jhathaway wrote:

We can either leave it or add one of two patches on top of the foward port.

1. Use YAML.unsafe_load_file, which shouldn't be any worse than our present yaml safety
2. Backport Puppet's safe_load which selectively unmarshal's classes

Both patches are attached and were tested on puppetdb1003. I think the unsafe load is probably fine for our interim state?

Excellent, thanks! I'll add the unsafe.patch version on top of the current component. As you said it's not a difference to the status quo and with the migration to Puppet 7 we'll have it move to the new unmarshalling anyway.

Mentioned in SAL (#wikimedia-operations) [2023-04-18T12:39:48Z] <moritzm> imported puppet 5.5.22-2+deb12u2 for bookworm-wikimedia T330495

Change 909663 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update puppet version to be installed on bookworm

https://gerrit.wikimedia.org/r/909663

Change 909663 merged by Muehlenhoff:

[operations/puppet@production] Update puppet version to be installed on bookworm

https://gerrit.wikimedia.org/r/909663

In T330495#8789136, @MoritzMuehlenhoff wrote:

Both patches are attached and were tested on puppetdb1003. I think the unsafe load is probably fine for our interim state?

Excellent, thanks! I'll add the unsafe.patch version on top of the current component. As you said it's not a difference to the status quo and with the migration to Puppet 7 we'll have it move to the new unmarshalling anyway.

I have pushed the new build as 5.5.22-2+deb12u2 to the component.

In T330495#8789702, @MoritzMuehlenhoff wrote:

I have pushed the new build as 5.5.22-2+deb12u2 to the component.

tested on puppetdb1003, works great!

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304190822_jmm_2721130_sretest1002.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • The reimage failed, see the cookbook logs for the details

Change 910438 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/cookbooks@master] sre.hosts.reimage/sre.ganeti.reimage: Delete Puppet state file before reimage

https://gerrit.wikimedia.org/r/910438

Change 910438 merged by Muehlenhoff:

[operations/cookbooks@master] sre.hosts.reimage/sre.ganeti.reimage: Delete Puppet state file before reimage

https://gerrit.wikimedia.org/r/910438

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Change 912232 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Temporarily stop using udebs from unstable

https://gerrit.wikimedia.org/r/912232

Change 912232 merged by Muehlenhoff:

[operations/puppet@production] Temporarily stop using udebs from unstable

https://gerrit.wikimedia.org/r/912232

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304260822_jmm_1328859_sretest1002.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Change 912311 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/cookbooks@master] Revert "sre.hosts.reimage/sre.ganeti.reimage: Delete Puppet state file before reimage"

https://gerrit.wikimedia.org/r/912311

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304261252_jmm_1515402_sretest1002.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • The reimage failed, see the cookbook logs for the details

Mentioned in SAL (#wikimedia-operations) [2023-04-27T07:23:38Z] <moritzm> uploaded debmonitor-client 0.3.2-1+deb12u1 to bookworm-wikimedia T330495

Change 912773 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Adapt sources.list for bookworm

https://gerrit.wikimedia.org/r/912773

On a fresh bookworm installation I'm seeing a few Puppet failures like those:

Error: Could not set 'link' on ensure: wrong number of arguments (given 3, expected 2)

After some digging I tend to believe this is caused by https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/ and backporting the following patch should fix it?
https://github.com/puppetlabs/puppet/commit/6af09225b3b962547a

@jbond, @jhathaway Does that make sense to you? If so, I'd update out Puppet 5.5. agent backport for bookworm with that patch.

Mentioned in SAL (#wikimedia-operations) [2023-04-27T09:06:15Z] <moritzm> uploaded debdeploy 0.0.99.13+deb12u1 to bookworm-wikimedia T330495

And one more question for @jbond and @jhathaway : We're installing the ruby-safe-yaml package via the monitoring profile, where it seems to be used in check_puppetrun.rb only. Can we fix this script to use different YAML parsing without ruby-safe-yaml? The reason I'm asking is that in bookworm ruby-safe-yaml no longer exists: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019665 and it is lined up for removal from the archive.

In T330495#8809860, @MoritzMuehlenhoff wrote:

After some digging I tend to believe this is caused by https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/ and backporting the following patch should fix it?
https://github.com/puppetlabs/puppet/commit/6af09225b3b962547a

@jbond, @jhathaway Does that make sense to you? If so, I'd update out Puppet 5.5. agent backport for bookworm with that patch.

Yes that looks correct to me

Can we fix this script to use different YAML parsing without ruby-safe-yaml?

Yes we should be able to use yaml.safe_load ill take a look today

Mentioned in SAL (#wikimedia-operations) [2023-04-27T09:29:02Z] <moritzm> imported wmf-certificates to bookworm-wikimedia T330495

Mentioned in SAL (#wikimedia-operations) [2023-04-27T09:29:57Z] <moritzm> imported prometheus-rsyslog-exporter to bookworm-wikimedia T330495

Change 912798 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] check_puppetrun: Drop safe_yaml

https://gerrit.wikimedia.org/r/912798

Change 912773 merged by Muehlenhoff:

[operations/puppet@production] Adapt sources.list for bookworm

https://gerrit.wikimedia.org/r/912773

In T330495#8809997, @jbond wrote:

In T330495#8809860, @MoritzMuehlenhoff wrote:

After some digging I tend to believe this is caused by https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/ and backporting the following patch should fix it?
https://github.com/puppetlabs/puppet/commit/6af09225b3b962547a

@jbond, @jhathaway Does that make sense to you? If so, I'd update out Puppet 5.5. agent backport for bookworm with that patch.

Yes that looks correct to me

I can confirm that this works. With the following patch applied setting symlinks works again on Bookworm with Puppet 5.5 and Ruby 3.1. I'm going to roll this into an updated puppet package and upload to apt,wikimedia.org:

Partial Backport of the following upstream commit:

From 6af09225b3b962547a2564a9d34ccd6832e60558 Mon Sep 17 00:00:00 2001
From: Melissa Stone <melissa@puppet.com>
Date: Wed, 27 May 2020 12:24:40 -0700
Subject: [PATCH] (PUP-10537) Ruby 2.7 warning: keyword param as last arg

`warning: Using the last argument as keyword parameters is deprecated;
maybe ** should be added to the call`

In preparation for Ruby 3, Ruby 2.7 has added warnings around how
keyword and positional arguments are used. Details can be found at:
https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/
---
 lib/puppet/file_system/file_impl.rb  | 2 +-

--- puppet-5.5.22.orig/lib/puppet/file_system/file_impl.rb
+++ puppet-5.5.22/lib/puppet/file_system/file_impl.rb
@@ -121,7 +121,7 @@ class Puppet::FileSystem::FileImpl
   end
 
   def symlink(path, dest, options = {})
-    FileUtils.symlink(path, dest, options)
+    FileUtils.symlink(path, dest, **options)
   end
 
   def symlink?(path)

Mentioned in SAL (#wikimedia-operations) [2023-04-27T12:12:15Z] <moritzm> imported puppet 5.5.22-2+deb13u3 to bookworm-wikimedia T330495

Change 912840 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Install 5.5.22-2+deb12u3 in late-setup.sh

https://gerrit.wikimedia.org/r/912840

Change 912840 merged by Muehlenhoff:

[operations/puppet@production] Install 5.5.22-2+deb12u3 in late-setup.sh

https://gerrit.wikimedia.org/r/912840

Change 912798 merged by Jbond:

[operations/puppet@production] check_puppetrun: Drop safe_yaml

https://gerrit.wikimedia.org/r/912798

Change 912864 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Stop installing ruby-safe-yaml

https://gerrit.wikimedia.org/r/912864

Change 912864 merged by Muehlenhoff:

[operations/puppet@production] Stop installing ruby-safe-yaml

https://gerrit.wikimedia.org/r/912864

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm executed with errors:

• sretest1002 (FAIL)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304280730_jmm_3276541_sretest1002.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • The reimage failed, see the cookbook logs for the details

Change 913121 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Use signed-by to include the Wikimedia repo starting with Bookworm

https://gerrit.wikimedia.org/r/913121

Change 913132 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Use signed-by to in apt::package_from_component on Bookworm

https://gerrit.wikimedia.org/r/913132

Change 913132 merged by Muehlenhoff:

[operations/puppet@production] Use signed-by to in apt::package_from_component on Bookworm

https://gerrit.wikimedia.org/r/913132

Change 913121 merged by Muehlenhoff:

[operations/puppet@production] Use signed-by to include the Wikimedia repo starting with Bookworm

https://gerrit.wikimedia.org/r/913121

Change 912311 merged by Muehlenhoff:

[operations/cookbooks@master] Revert "sre.hosts.reimage/sre.ganeti.reimage: Delete Puppet state file before reimage"

https://gerrit.wikimedia.org/r/912311

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1002.eqiad.wmnet with OS bookworm completed:

• sretest1002 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202305040551_jmm_2214008_sretest1002.out
  • Checked BIOS boot parameters are back to normal
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Change 916434 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Move duplicity check for apt keyrings to !defined

https://gerrit.wikimedia.org/r/916434

Change 916434 merged by Muehlenhoff:

[operations/puppet@production] Move duplicity check for apt keyrings to !defined

https://gerrit.wikimedia.org/r/916434

Mentioned in SAL (#wikimedia-operations) [2023-05-08T07:19:11Z] <moritzm> updated bookworm installer to RC2 T330495

Mentioned in SAL (#wikimedia-operations) [2023-05-09T13:27:06Z] <moritzm> updated bookworm d-i image to 2022-05-09 daily build T330495

The installer is working fine for baremetal and VM installations, but there will be a few more RC releases before the final release, so keeping the task open for now.

One important note is that the memory requirements for the kernel have raised and installations with just 1G RAM will fail, I didn't track down the exact lower minimum, but 1.5G are working perfectly fine.

Mentioned in SAL (#wikimedia-operations) [2023-05-16T11:00:03Z] <moritzm> updated bookworm image to RC3 T330495

Change 921174 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Drop Boost packages from legacy package removal list for Bookworm

https://gerrit.wikimedia.org/r/921174

Change 921174 merged by Muehlenhoff:

[operations/puppet@production] Drop Boost packages from legacy package removal list for Bookworm

https://gerrit.wikimedia.org/r/921174

Mentioned in SAL (#wikimedia-operations) [2023-05-30T07:16:33Z] <moritzm> update bookworm installer to rc4 T330495

Change 925862 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] expand_path, regex_data: use yaml safe_load when available

https://gerrit.wikimedia.org/r/925862

Change 925878 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] bookworm: Change to deb822 format for sources.list

https://gerrit.wikimedia.org/r/925878