Page MenuHomePhabricator
Create Task
Maniphest T357748

Migrate CAS to Bookworm
Closed, ResolvedPublic
Actions

Description

We're using the idp-test role to build CAS; but that no longer works: CAS 6.6 can only be built with Java 11 and Bookworm comes with 17. As such, we're creating a new idp_build role for that on idp-test1003, the server we used to test whether CAS 6.6 works fine with Bookworm.

Details

SubjectRepoBranchLines +/-
Remove obsolete grantoperations/puppetproduction+0 -3
R:idp decommision Bullseye IDP hosts.operations/puppetproduction+0 -8
P:idp Disable Bullseye hosts.operations/puppetproduction+1 -7
IDP: Switch to new Bookworm host.operations/dnsmaster+1 -1
R:idp enable new Bookworm hosts.operations/puppetproduction+5 -1
P:acme_chief::certificates Add new IDP hosts.operations/puppetproduction+4 -0
site: Add new IDP production hosts.operations/puppetproduction+4 -0
P:idp-test remove build host from list of valid IDP hosts.operations/puppetproduction+0 -1
idp-test - Switch to upgraded Bookworm host.operations/dnsmaster+1 -1
Configure idp-test as build hostoperations/puppetproduction+1 -1
IDP: Switchback to Bullseye host.operations/dnsmaster+1 -1
Add a stub role to build Apereo CASoperations/puppetproduction+13 -0
P:idp Use Tomcat9 build for Bookwormoperations/puppetproduction+16 -11
Switch idp-test to new Bookworm server.operations/dnsmaster+1 -1
R:idp Add new node to IDP node set.operations/puppetproduction+1 -0
cas-overlay: Revert Tomcat10-related changesoperations/software/cas-overlay-templatemaster+6 -0
C:tomcat Pin Tomcat9 packages for Bookworm.operations/puppetproduction+11 -9
Sync web.xml to default template from Tomcat 10/Bookwormoperations/puppetproduction+627 -36
C:tomcat10 Sync server.xml with default from Tomcat10/Bookworm.operations/puppetproduction+143 -66
P:idp switch final prodution host to Java 17.operations/puppetproduction+4 -3
Sync context.xml to default template from Tomcat 10/Bookwormoperations/puppetproduction+11 -8
C:tomcat10 New modules for Tomcat10operations/puppetproduction+4 K -2
C:tomcat Allow users to specify which version of Tomcat to install.operations/puppetproduction+19 -6
More postinst changes to cope with Tomcat 9->10 changesoperations/software/cas-overlay-templatemaster+17 -12
Rebuild cas for Bookworm, and depend on tomcat 10 nowoperations/software/cas-overlay-templatemaster+7 -1
idp::build: Remove explicit dependency on openjdk-11-jdk-headlessoperations/puppetproduction+2 -1
acmechief: Add idp-test[12]003operations/puppetproduction+4 -0
Set acmechief_host for idp-test[12]003operations/puppetproduction+3 -0
P:IDP New Bookworm IDP servers.operations/puppetproduction+8 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedSLyngshede-WMFT357748 Migrate CAS to Bookworm
 ResolvedSLyngshede-WMFT357749 Move CAS to Java 17
 ResolvedMoritzMuehlenhoffT359333 Build Tomcat 9 for Bookworm

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Feb 28 2024, 12:47 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-28T12:47:38Z] <moritzm> import cas 6.6.12+wmf12u2 to bookworm-wikimedia T357748

gerritbot added a comment.Feb 28 2024, 12:52 PM

Change 1007324 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/software/cas-overlay-template@master] More Tomcat 10 changes T357748

https://gerrit.wikimedia.org/r/1007324

gerritbot added a comment.Feb 28 2024, 3:55 PM

Change 1007324 merged by Muehlenhoff:

[operations/software/cas-overlay-template@master] More Tomcat 10 changes T357748

https://gerrit.wikimedia.org/r/1007324

Stashbot added a comment.Feb 28 2024, 4:17 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-28T16:17:35Z] <moritzm> import cas 6.6.12+wmf12u3 to bookworm-wikimedia T357748

Maintenance_bot removed a project: Patch-For-Review.Feb 28 2024, 4:31 PM
ops-monitoring-bot added a comment.Feb 29 2024, 7:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp-test1003.wikimedia.org with OS bookworm

gerritbot added a comment.Feb 29 2024, 7:23 AM

Change 1007487 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:idp switch final prodution host to Java 17.

https://gerrit.wikimedia.org/r/1007487

gerritbot added a project: Patch-For-Review.Feb 29 2024, 7:23 AM
ops-monitoring-bot added a comment.Feb 29 2024, 7:51 AM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp-test1003.wikimedia.org with OS bookworm completed:

  • idp-test1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402290735_slyngshede_3618694_idp-test1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Feb 29 2024, 10:09 AM

Change 1007569 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] C:tomcat10 New modules for Tomcat10

https://gerrit.wikimedia.org/r/1007569

gerritbot added a comment.Feb 29 2024, 11:15 AM

Change 1007569 merged by Slyngshede:

[operations/puppet@production] C:tomcat10 New modules for Tomcat10

https://gerrit.wikimedia.org/r/1007569

gerritbot added a comment.Feb 29 2024, 11:39 AM

Change rMW1007585a9cd5 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Sync context.xml to default template from 10/Bookworm

https://gerrit.wikimedia.org/r/1007585

gerritbot added a comment.Feb 29 2024, 12:15 PM

Change rMW1007585a9cd5 merged by Muehlenhoff:

[operations/puppet@production] Sync context.xml to default template from Tomcat 10/Bookworm

https://gerrit.wikimedia.org/r/1007585

gerritbot added a comment.Feb 29 2024, 12:20 PM

Change 1007596 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] C:tomcat10 Sync server.xml with default from Tomcat10/Bookworm.

https://gerrit.wikimedia.org/r/1007596

gerritbot added a comment.Feb 29 2024, 12:38 PM

Change 1007603 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Sync web.xml to default template from Tomcat 10/Bookworm

https://gerrit.wikimedia.org/r/1007603

gerritbot added a comment.Feb 29 2024, 2:26 PM

Change 1007487 merged by Slyngshede:

[operations/puppet@production] P:idp switch final prodution host to Java 17.

https://gerrit.wikimedia.org/r/1007487

SLyngshede-WMF closed subtask T357749: Move CAS to Java 17 as Resolved.Feb 29 2024, 2:30 PM
gerritbot added a comment.Mar 6 2024, 8:49 AM

Change 1007596 merged by Slyngshede:

[operations/puppet@production] C:tomcat10 Sync server.xml with default from Tomcat10/Bookworm.

https://gerrit.wikimedia.org/r/1007596

gerritbot added a comment.Mar 6 2024, 8:58 AM

Change 1007603 merged by Muehlenhoff:

[operations/puppet@production] Sync web.xml to default template from Tomcat 10/Bookworm

https://gerrit.wikimedia.org/r/1007603

Maintenance_bot removed a project: Patch-For-Review.Mar 6 2024, 9:30 AM
gerritbot added a comment.Mar 8 2024, 8:21 AM

Change 1009709 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:idp Use Tomcat9 build for Bookworm

https://gerrit.wikimedia.org/r/1009709

gerritbot added a project: Patch-For-Review.Mar 8 2024, 8:21 AM
gerritbot added a comment.Mar 8 2024, 9:10 AM

Change 1009709 merged by Slyngshede:

[operations/puppet@production] P:idp Use Tomcat9 build for Bookworm

https://gerrit.wikimedia.org/r/1009709

Maintenance_bot removed a project: Patch-For-Review.Mar 8 2024, 9:31 AM
ops-monitoring-bot added a comment.Mar 18 2024, 1:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp-test1003.wikimedia.org with OS bookworm

gerritbot added a comment.Mar 18 2024, 1:25 PM

Change 1012374 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] C:tomcat Pin Tomcat9 packages for Bookworm.

https://gerrit.wikimedia.org/r/1012374

gerritbot added a project: Patch-For-Review.Mar 18 2024, 1:25 PM
ops-monitoring-bot added a comment.Mar 18 2024, 1:40 PM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp-test1003.wikimedia.org with OS bookworm completed:

  • idp-test1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403181324_slyngshede_1731844_idp-test1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mar 18 2024, 1:41 PM

Change 1012374 merged by Slyngshede:

[operations/puppet@production] C:tomcat Pin Tomcat9 packages for Bookworm.

https://gerrit.wikimedia.org/r/1012374

Maintenance_bot removed a project: Patch-For-Review.Mar 18 2024, 2:31 PM
gerritbot added a comment.Mar 18 2024, 2:51 PM

Change 1012394 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/software/cas-overlay-template@master] cas-overlay: Revert Tomcat10-related changes

https://gerrit.wikimedia.org/r/1012394

gerritbot added a project: Patch-For-Review.Mar 18 2024, 2:51 PM
gerritbot added a comment.Mar 18 2024, 2:57 PM

Change 1012394 merged by Muehlenhoff:

[operations/software/cas-overlay-template@master] cas-overlay: Revert Tomcat10-related changes

https://gerrit.wikimedia.org/r/1012394

Maintenance_bot removed a project: Patch-For-Review.Mar 18 2024, 3:32 PM
Stashbot added a comment.Mar 18 2024, 3:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-18T15:57:33Z] <moritzm> uploaded cas 6.6.12+wmf12u4 (rebuild with/for tomcat9) T357748

ops-monitoring-bot added a comment.Mar 19 2024, 8:09 AM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp-test1003.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Mar 19 2024, 8:42 AM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp-test1003.wikimedia.org with OS bookworm completed:

  • idp-test1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403190824_slyngshede_1873028_idp-test1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mar 19 2024, 9:51 AM

Change 1012612 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] R:idp Add new node to IDP node set.

https://gerrit.wikimedia.org/r/1012612

gerritbot added a project: Patch-For-Review.Mar 19 2024, 9:51 AM
gerritbot added a comment.Mar 19 2024, 10:13 AM

Change 1012612 merged by Slyngshede:

[operations/puppet@production] R:idp Add new node to IDP node set.

https://gerrit.wikimedia.org/r/1012612

Maintenance_bot removed a project: Patch-For-Review.Mar 19 2024, 10:31 AM
gerritbot added a comment.Mar 19 2024, 10:33 AM

Change 1012620 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/dns@master] Switch idp-test to new Bookworm server.

https://gerrit.wikimedia.org/r/1012620

gerritbot added a project: Patch-For-Review.Mar 19 2024, 10:33 AM
gerritbot added a comment.Mar 19 2024, 12:00 PM

Change 1012620 merged by Slyngshede:

[operations/dns@master] Switch idp-test to new Bookworm server.

https://gerrit.wikimedia.org/r/1012620

Maintenance_bot removed a project: Patch-For-Review.Mar 19 2024, 12:31 PM
gerritbot added a comment.Mar 20 2024, 8:12 AM

Change 1012990 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add a stub role to build Apereo CAS

https://gerrit.wikimedia.org/r/1012990

gerritbot added a project: Patch-For-Review.Mar 20 2024, 8:12 AM
gerritbot added a comment.Mar 20 2024, 9:06 AM

Change 1012990 merged by Muehlenhoff:

[operations/puppet@production] Add a stub role to build Apereo CAS

https://gerrit.wikimedia.org/r/1012990

gerritbot added a comment.Mar 20 2024, 9:13 AM

Change 1013001 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/dns@master] IDP: Switchback to Bullseye host.

https://gerrit.wikimedia.org/r/1013001

MoritzMuehlenhoff updated the task description. (Show Details)Mar 20 2024, 9:19 AM
SLyngshede-WMF updated the task description. (Show Details)Mar 20 2024, 9:22 AM
gerritbot added a comment.Mar 20 2024, 9:24 AM

Change 1013003 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Configure idp-test as build host

https://gerrit.wikimedia.org/r/1013003

gerritbot added a comment.Mar 20 2024, 9:28 AM

Change 1013001 merged by Slyngshede:

[operations/dns@master] IDP: Switchback to Bullseye host.

https://gerrit.wikimedia.org/r/1013001

gerritbot added a comment.Mar 20 2024, 9:33 AM

Change 1013003 merged by Muehlenhoff:

[operations/puppet@production] Configure idp-test as build host

https://gerrit.wikimedia.org/r/1013003

ops-monitoring-bot added a comment.Mar 20 2024, 9:39 AM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp-test1003.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Mar 20 2024, 10:07 AM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp-test1003.wikimedia.org with OS bullseye completed:

  • idp-test1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403200954_slyngshede_2070529_idp-test1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Mar 20 2024, 10:31 AM
ops-monitoring-bot added a comment.Mar 20 2024, 12:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp-test2002.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Mar 20 2024, 12:59 PM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp-test2002.wikimedia.org with OS bookworm completed:

  • idp-test2002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403201240_slyngshede_2095771_idp-test2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mar 20 2024, 1:06 PM

Change 1013043 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/dns@master] idp-test - Switch to upgraded Bookworm host.

https://gerrit.wikimedia.org/r/1013043

gerritbot added a project: Patch-For-Review.Mar 20 2024, 1:06 PM
gerritbot added a comment.Mar 20 2024, 1:08 PM

Change 1013043 merged by Slyngshede:

[operations/dns@master] idp-test - Switch to upgraded Bookworm host.

https://gerrit.wikimedia.org/r/1013043

gerritbot added a comment.Mar 20 2024, 1:20 PM

Change 1013046 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:idp-test remove build host from list of valid IDP hosts.

https://gerrit.wikimedia.org/r/1013046

gerritbot added a comment.Mar 20 2024, 1:34 PM

Change 1013046 merged by Slyngshede:

[operations/puppet@production] P:idp-test remove build host from list of valid IDP hosts.

https://gerrit.wikimedia.org/r/1013046

MoritzMuehlenhoff closed subtask T359333: Build Tomcat 9 for Bookworm as Resolved.Mar 20 2024, 1:58 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 20 2024, 2:30 PM
gerritbot added a comment.Mar 21 2024, 7:50 AM

Change 1013166 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] site: Add new IDP production hosts.

https://gerrit.wikimedia.org/r/1013166

gerritbot added a project: Patch-For-Review.Mar 21 2024, 7:50 AM
SLyngshede-WMF added a comment.EditedMar 21 2024, 8:13 AM

VM creation:

sudo cookbook sre.ganeti.makevm --vcpus 2 --memory 4 --disk 40 --network public --os bookworm --cluster codfw --group A -t T357748 -p 7 idp2003
sudo cookbook sre.ganeti.makevm --vcpus 2 --memory 4 --disk 40 --network public --os bookworm --cluster eqiad --group C -t T357748 -p 7 idp1003
gerritbot added a comment.Mar 21 2024, 8:57 AM

Change 1013166 merged by Slyngshede:

[operations/puppet@production] site: Add new IDP production hosts.

https://gerrit.wikimedia.org/r/1013166

ops-monitoring-bot added a comment.Mar 21 2024, 9:22 AM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp2003.wikimedia.org with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Mar 21 2024, 9:30 AM
ops-monitoring-bot added a comment.Mar 21 2024, 9:59 AM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp2003.wikimedia.org with OS bookworm completed:

  • idp2003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403210940_slyngshede_2261129_idp2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Mar 21 2024, 10:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp1003.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Mar 21 2024, 10:34 AM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp1003.wikimedia.org with OS bookworm completed:

  • idp1003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403211020_slyngshede_2269692_idp1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mar 21 2024, 10:50 AM

Change 1013237 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] R:idp enable new Bookworm hosts.

https://gerrit.wikimedia.org/r/1013237

gerritbot added a project: Patch-For-Review.Mar 21 2024, 10:50 AM
gerritbot added a comment.Mar 21 2024, 10:58 AM

Change 1013241 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:acme_chief::certificates Add new IDP hosts.

https://gerrit.wikimedia.org/r/1013241

gerritbot added a comment.Mar 21 2024, 11:56 AM

Change 1013241 merged by Slyngshede:

[operations/puppet@production] P:acme_chief::certificates Add new IDP hosts.

https://gerrit.wikimedia.org/r/1013241

gerritbot added a comment.Mar 22 2024, 7:52 AM

Change #1013237 merged by Slyngshede:

[operations/puppet@production] R:idp enable new Bookworm hosts.

https://gerrit.wikimedia.org/r/1013237

gerritbot added a comment.Mar 25 2024, 9:22 AM

Change #1013949 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/dns@master] IDP: Switch to new Bookworm host.

https://gerrit.wikimedia.org/r/1013949

gerritbot added a comment.Mar 25 2024, 12:23 PM

Change #1013949 merged by Slyngshede:

[operations/dns@master] IDP: Switch to new Bookworm host.

https://gerrit.wikimedia.org/r/1013949

gerritbot added a comment.Mon, Apr 8, 6:45 AM

Change #1017602 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:idp Disable Bullseye hosts.

https://gerrit.wikimedia.org/r/1017602

SLyngshede-WMF changed the task status from Open to In Progress.Mon, Apr 8, 6:51 AM
SLyngshede-WMF claimed this task.
gerritbot added a comment.Mon, Apr 8, 7:11 AM

Change #1017602 merged by Slyngshede:

[operations/puppet@production] P:idp Disable Bullseye hosts.

https://gerrit.wikimedia.org/r/1017602

SLyngshede-WMF added a comment.Wed, Apr 10, 12:02 PM

Logs from idp1002 / idp2002 from 2023 and 2024 have been copied to idp1003 and idp2003 respectively, and are stored in /srv/logs

ops-monitoring-bot added a comment.Wed, Apr 10, 12:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp-test1002.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Wed, Apr 10, 12:38 PM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp-test1002.wikimedia.org with OS bookworm completed:

  • idp-test1002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404101220_slyngshede_1801157_idp-test1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Apr 10, 12:59 PM

cookbooks.sre.hosts.decommission executed by slyngshede@cumin1002 for hosts: idp-test2003.wikimedia.org

  • idp-test2003.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
ops-monitoring-bot added a comment.Thu, Apr 11, 7:10 AM

cookbooks.sre.hosts.decommission executed by slyngshede@cumin1002 for hosts: idp1002.wikimedia.org

  • idp1002.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
gerritbot added a comment.Thu, Apr 11, 7:10 AM

Change #1018871 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] R:idp decommision Bullseye IDP hosts.

https://gerrit.wikimedia.org/r/1018871

ops-monitoring-bot added a comment.Thu, Apr 11, 8:06 AM

cookbooks.sre.hosts.decommission executed by slyngshede@cumin1002 for hosts: idp2002.wikimedia.org

  • idp2002.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
gerritbot added a comment.Thu, Apr 11, 8:07 AM

Change #1018871 merged by Slyngshede:

[operations/puppet@production] R:idp decommision Bullseye IDP hosts.

https://gerrit.wikimedia.org/r/1018871

SLyngshede-WMF closed this task as Resolved.Thu, Apr 11, 8:13 AM

Old Bullseye servers have been decommissioned.

Note that one Bullseye server remains idp-test1003 remains as a build host for CAS.
This will need to stay in place until we upgrade CAS, as Java 11 is required for building the CAS 6.6 overlay (it can run on Java 17, but not build).

gerritbot added a comment.Thu, Apr 11, 8:47 AM

Change #1018941 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove obsolete grant

https://gerrit.wikimedia.org/r/1018941

gerritbot added a comment.Thu, Apr 11, 2:30 PM

Change #1018941 abandoned by Muehlenhoff:

[operations/puppet@production] Remove obsolete grant

Reason:

Obsoleted by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1019014

https://gerrit.wikimedia.org/r/1018941

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL