Page MenuHomePhabricator
Create Task
Maniphest T349286

Upgrade schema hosts to bookworm
Closed, ResolvedPublic
Actions

Description

Reference ticket for when they were upgrade to buster: T255026: Upgrade schema[12]00[12] to Debian Buster

These are 4 VMs, 2 in eqiad and 2 in codfw

  • schema1003.eqiad.wmnet
  • schema1004.eqiad.wmnet
  • schema2003.codfw.wmnet
  • schema2004.codfw.wmnet

Since the last time these were upgraded, our reimaging cookbook has been adapted so that it works with VMs and this means that they can be upgraded in place.

This should make it even easier to upgrade them. There are no custom applications running on them, it is just nginx and a git::clone which is all managed by puppet.
Tagging Event-Platform and Data-Engineering for visibility, but I would expect that Data-Platform-SRE will be carrying out the upgrades.

Related Objects
Search...

Event Timeline

BTullis created this task.Oct 19 2023, 11:07 AM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptOct 19 2023, 11:07 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
BTullis added a parent task: T288804: Upgrade the Data Engineering infrastructure to Debian Bullseye.Oct 23 2023, 8:40 AM
Gehel moved this task from Incoming to Ready for Work on the Data-Platform-SRE board.Nov 3 2023, 10:34 AM
Gehel triaged this task as High priority.Nov 15 2023, 9:44 AM
BTullis claimed this task.Nov 28 2023, 2:59 PM
BTullis moved this task from Ready for Work to In Progress on the Data-Platform-SRE board.
Stashbot added a comment.Nov 29 2023, 2:04 PM

Mentioned in SAL (#wikimedia-analytics) [2023-11-29T14:04:40Z] <btullis> depooling schema1003 for reimage T349286

Stashbot added a comment.Nov 29 2023, 2:10 PM

Mentioned in SAL (#wikimedia-analytics) [2023-11-29T14:10:29Z] <btullis> reimaging schema1003 to bookworm for T349286

ops-monitoring-bot added a comment.Nov 29 2023, 2:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host schema1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 29 2023, 2:38 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host schema1003.eqiad.wmnet with OS bookworm completed:

  • schema1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311291424_btullis_1645562_schema1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Nov 29 2023, 2:43 PM

Mentioned in SAL (#wikimedia-analytics) [2023-11-29T14:43:02Z] <btullis> depooling schema1004 for reimage T349286

ops-monitoring-bot added a comment.Nov 29 2023, 2:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host schema1004.eqiad.wmnet with OS bookworm

Stashbot added a comment.Nov 29 2023, 2:44 PM

Mentioned in SAL (#wikimedia-analytics) [2023-11-29T14:44:09Z] <btullis> reimaging schema1004 to bookworm for T349286

ops-monitoring-bot added a comment.Nov 29 2023, 3:15 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host schema1004.eqiad.wmnet with OS bookworm completed:

  • schema1004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311291502_btullis_1661207_schema1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Nov 29 2023, 3:24 PM

Mentioned in SAL (#wikimedia-analytics) [2023-11-29T15:24:51Z] <btullis> pooled schema1004 after upgrade to bookworm for T349286

ops-monitoring-bot added a comment.Nov 29 2023, 3:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host schema2003.codfw.wmnet with OS bookworm

BTullis renamed this task from Upgrade schema hosts to bullseye to Upgrade schema hosts to bookworm.Nov 29 2023, 3:31 PM
ops-monitoring-bot added a comment.Nov 29 2023, 4:16 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host schema2003.codfw.wmnet with OS bookworm completed:

  • schema2003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311291603_btullis_1684882_schema2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Nov 29 2023, 5:10 PM

Mentioned in SAL (#wikimedia-analytics) [2023-11-29T17:10:28Z] <btullis> depool schema2004 for reimage to bookworm for T349286

ops-monitoring-bot added a comment.Nov 29 2023, 5:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host schema2004.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 29 2023, 5:50 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host schema2004.codfw.wmnet with OS bookworm completed:

  • schema2004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311291732_btullis_1750826_schema2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BTullis closed this task as Resolved.Nov 29 2023, 6:12 PM
BTullis updated the task description. (Show Details)
BTullis moved this task from In Progress to Done on the Data-Platform-SRE board.
BTullis mentioned this in T347421: [NEEDS GROOMING] schema services should be moved to k8s.Dec 20 2023, 5:33 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL