Page MenuHomePhabricator
Create Task
Maniphest T332400

Migrate cloudweb, cloudbackup, cloudmetrics physical servers off buster
Closed, ResolvedPublic
Actions

Description

According to https://os-reports.wikimedia.org/buster.html, the following cloud roles need migrated off buster:

role::wmcs::ceph::mon (6 host(s))
role::wmcs::ceph::osd (20 host(s))
role::wmcs::monitoring (2 host(s))
role::wmcs::openstack::codfw1dev::cloudweb (1 host(s))
role::wmcs::openstack::eqiad1::backy (2 host(s))
role::wmcs::openstack::eqiad1::labweb (2 host(s))

This equates to these physical hosts:

  • cloudbackup1003.eqiad.wmnet
  • cloudbackup1004.eqiad.wmnet
  • cloudmetrics1003.eqiad.wmnet (reclaimed T351077)
  • cloudmetrics1004.eqiad.wmnet (reclaimed T351077)
  • cloudweb1003.wikimedia.org (now bullseye)
  • cloudweb1004.wikimedia.org (now bullseye)
  • cloudweb2002-dev.wikimedia.org (now bullseye)

Ceph hosts are already covered / tracked by T309789.

Details

SubjectRepoBranchLines +/-
Revert "wmcs VM backups: move all backups to one host"operations/puppetproduction+3 -3
wmcs VM backups: move all backups to one host, cloudbackup1003operations/puppetproduction+29 -29
wmcs VM backups: move all backups to one host, cloudbackup1004operations/puppetproduction+28 -28
Customize query in gerrit

Related Objects

Event Timeline

• nskaggs created this task.Mar 17 2023, 2:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 17 2023, 2:53 PM
• nskaggs added a comment.Apr 5 2023, 2:42 PM

Note, WMCS would like to migrate directly to bookworm if possible.

gerritbot added a comment.Apr 23 2024, 4:36 PM

Change #1023466 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] wmcs VM backups: move all backups to one host, cloudbackup1004

https://gerrit.wikimedia.org/r/1023466

gerritbot added a project: Patch-For-Review.Apr 23 2024, 4:36 PM

Change #1023467 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] wmcs VM backups: move all backups to one host, cloudbackup1003

https://gerrit.wikimedia.org/r/1023467

gerritbot added a comment.Apr 23 2024, 4:36 PM

Change #1023468 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Revert "wmcs VM backups: move all backups to one host, cloudbackup1003" Revert "wmcs VM backups: move all backups to one host, cloudbackup1004"

https://gerrit.wikimedia.org/r/1023468

Andrew updated the task description. (Show Details)Apr 23 2024, 4:41 PM
gerritbot added a comment.Apr 23 2024, 4:42 PM

Change #1023466 merged by Andrew Bogott:

[operations/puppet@production] wmcs VM backups: move all backups to one host, cloudbackup1004

https://gerrit.wikimedia.org/r/1023466

Andrew subscribed.Apr 23 2024, 4:43 PM

It is safe to reimage cloudbackup1003 on April 30.

ops-monitoring-bot added a comment.Apr 29 2024, 1:42 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudbackup1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 29 2024, 3:11 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudbackup1003.eqiad.wmnet with OS bookworm completed:

  • cloudbackup1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404290217_andrew_2941912_cloudbackup1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 29 2024, 3:19 AM

Change #1023467 merged by Andrew Bogott:

[operations/puppet@production] wmcs VM backups: move all backups to one host, cloudbackup1003

https://gerrit.wikimedia.org/r/1023467

ops-monitoring-bot added a comment.May 6 2024, 1:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudbackup1004.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.May 6 2024, 2:16 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudbackup1004.eqiad.wmnet with OS bookworm completed:

  • cloudbackup1004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405061353_andrew_839804_cloudbackup1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.May 6 2024, 3:47 PM

Change #1023468 merged by Andrew Bogott:

[operations/puppet@production] Revert "wmcs VM backups: move all backups to one host"

https://gerrit.wikimedia.org/r/1023468

Andrew closed this task as Resolved.May 6 2024, 3:51 PM
Andrew updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.May 6 2024, 4:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits