Page MenuHomePhabricator
Create Task
Maniphest T343515

Migrate bastions to Bookworm
Closed, ResolvedPublic
Actions

Description

Migrate the SSH bastions to Bookworm:

  • eqiad
  • codfw
  • esams
  • ulsfo
  • eqsin
  • drmrs

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove bast4004/bast5003/bast6002operations/puppetproduction+0 -12
new bastions in ulsfo/eqsin/drmrsoperations/puppetproduction+9 -3
Add new bastions to site.ppoperations/puppetproduction+16 -0
Customize query in gerrit

Event Timeline

MoritzMuehlenhoff created this task.Aug 4 2023, 10:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2023, 10:30 AM
MoritzMuehlenhoff triaged this task as Medium priority.Aug 4 2023, 10:30 AM
MoritzMuehlenhoff claimed this task.Aug 4 2023, 10:38 AM
MoritzMuehlenhoff updated the task description. (Show Details)
ops-monitoring-bot added a comment.Aug 4 2023, 11:02 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host bast3007.wikimedia.org with OS bookworm

gerritbot added a comment.Aug 4 2023, 11:36 AM

Change 945767 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add new bastions to site.pp

https://gerrit.wikimedia.org/r/945767

gerritbot added a project: Patch-For-Review.Aug 4 2023, 11:36 AM
gerritbot added a comment.Aug 4 2023, 11:42 AM

Change 945767 merged by Muehlenhoff:

[operations/puppet@production] Add new bastions to site.pp

https://gerrit.wikimedia.org/r/945767

ops-monitoring-bot added a comment.Aug 4 2023, 12:06 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host bast3007.wikimedia.org with OS bookworm completed:

  • bast3007 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202308041151_jmm_2421825_bast3007.out, asking the operator what to do
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202308041156_jmm_2421825_bast3007.out, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308041158_jmm_2421825_bast3007.out
    • Unable to run puppet on config-master2001.codfw.wmnet,config-master1001.eqiad.wmnet,puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Aug 4 2023, 12:11 PM
ops-monitoring-bot added a comment.Aug 4 2023, 1:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host bast4005.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Aug 4 2023, 2:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host bast4005.wikimedia.org with OS bookworm completed:

  • bast4005 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308041402_jmm_2577329_bast4005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Aug 10 2023, 5:32 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host bast5004.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Aug 10 2023, 7:19 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host bast5004.wikimedia.org with OS bookworm executed with errors:

  • bast5004 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Aug 10 2023, 8:00 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: bast5004.wikimedia.org

  • bast5004.wikimedia.org (WARN)
    • Host not found on Icinga, unable to downtime it
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqsin to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqsin to Netbox
ops-monitoring-bot added a comment.Aug 10 2023, 11:58 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: bast3007.wikimedia.org

  • bast3007.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
MoritzMuehlenhoff updated the task description. (Show Details)Aug 21 2023, 10:00 AM
ops-monitoring-bot added a comment.Aug 25 2023, 7:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host bast5004.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Aug 25 2023, 8:04 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host bast5004.wikimedia.org with OS bookworm executed with errors:

  • bast5004 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Aug 25 2023, 8:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host bast6003.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Aug 25 2023, 9:07 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host bast6003.wikimedia.org with OS bookworm executed with errors:

  • bast6003 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Aug 25 2023, 10:06 AM

Change 952320 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] new bastions in ulsfo/eqsin/drmrs

https://gerrit.wikimedia.org/r/952320

gerritbot added a project: Patch-For-Review.Aug 25 2023, 10:06 AM
gerritbot added a comment.Aug 25 2023, 10:12 AM

Change 952320 merged by Muehlenhoff:

[operations/puppet@production] new bastions in ulsfo/eqsin/drmrs

https://gerrit.wikimedia.org/r/952320

Maintenance_bot removed a project: Patch-For-Review.Aug 25 2023, 10:30 AM
ops-monitoring-bot added a comment.Sep 4 2023, 8:17 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: bast4004.wikimedia.org

  • bast4004.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ulsfo to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ulsfo to Netbox
ops-monitoring-bot added a comment.Sep 4 2023, 8:31 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: bast5003.wikimedia.org

  • bast5003.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqsin to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqsin to Netbox
ops-monitoring-bot added a comment.Sep 4 2023, 8:45 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: bast6002.wikimedia.org

  • bast6002.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster drmrs02 to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster drmrs02 to Netbox
gerritbot added a comment.Sep 4 2023, 8:53 AM

Change 954597 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove bast4004/bast5003/bast6002

https://gerrit.wikimedia.org/r/954597

gerritbot added a project: Patch-For-Review.Sep 4 2023, 8:54 AM
MoritzMuehlenhoff updated the task description. (Show Details)Sep 4 2023, 8:54 AM
gerritbot added a comment.Sep 4 2023, 8:56 AM

Change 954597 merged by Muehlenhoff:

[operations/puppet@production] Remove bast4004/bast5003/bast6002

https://gerrit.wikimedia.org/r/954597

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2023, 9:11 AM
MoritzMuehlenhoff closed this task as Resolved.Jun 7 2024, 9:38 AM
MoritzMuehlenhoff updated the task description. (Show Details)

All bastions are on bookworm now

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits