Page MenuHomePhabricator
Create Task
Maniphest T341201

rename cloudswift1002 as cloudlb1002
Closed, ResolvedPublic
Actions

Description

Netbox device: https://netbox.wikimedia.org/dcim/devices/3525

Procedure: https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Rename_while_reimaging

  • decomission
  • netbox: edit the device name, and set its status from DECOMMISSIONING to PLANNED.
  • readd the DNS Name field for the management interface
  • run sre.dns.netbox cookbook
  • run sre.network.configure-switch-interfaces cookbook
  • reimage server with new name

Details

SubjectRepoBranchLines +/-
cloudlb: eqiad: bootstrap hiera dataoperations/puppetproduction+18 -0
cloudlb1001/1002: add roleoperations/puppetproduction+3 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero changed the task status from Open to In Progress.Jul 6 2023, 11:09 AM
aborrero triaged this task as Medium priority.
aborrero created this task.
aborrero moved this task from Backlog to Doing on the User-aborrero board.
aborrero updated the task description. (Show Details)Jul 6 2023, 11:28 AM
ops-monitoring-bot added a comment.Jul 6 2023, 11:30 AM

cookbooks.sre.hosts.decommission executed by aborrero@cumin1001 for hosts: cloudswift1002.eqiad.wmnet

  • cloudswift1002.eqiad.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
aborrero updated the task description. (Show Details)Jul 6 2023, 11:40 AM
aborrero added a comment.Jul 6 2023, 11:45 AM

saving this here for later:

image.png (619×1 px, 78 KB)

gerritbot added a comment.Jul 6 2023, 11:58 AM

Change 936019 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudlb1001/1002: add role

https://gerrit.wikimedia.org/r/936019

gerritbot added a project: Patch-For-Review.Jul 6 2023, 11:58 AM
gerritbot added a comment.Jul 6 2023, 12:00 PM

Change 936019 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudlb1001/1002: add role

https://gerrit.wikimedia.org/r/936019

gerritbot added a comment.Jul 6 2023, 12:08 PM

Change 936022 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudlb: eqiad: bootstrap hiera data

https://gerrit.wikimedia.org/r/936022

aborrero updated the task description. (Show Details)Jul 6 2023, 12:17 PM
aborrero changed the task status from In Progress to Stalled.Jul 6 2023, 2:32 PM
aborrero moved this task from Doing to Blocked on the User-aborrero board.

blocked on T341223: Configure eqiad cloudsw devices to support cloud-private

aborrero moved this task from Blocked to Doing on the User-aborrero board.Jul 7 2023, 9:45 AM
aborrero changed the task status from Stalled to In Progress.Jul 7 2023, 3:59 PM

No longer blocked!

gerritbot added a comment.Jul 10 2023, 8:59 AM

Change 936022 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudlb: eqiad: bootstrap hiera data

https://gerrit.wikimedia.org/r/936022

Maintenance_bot removed a project: Patch-For-Review.Jul 10 2023, 9:10 AM
ops-monitoring-bot added a comment.Jul 10 2023, 10:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudlb1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 10 2023, 10:11 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudlb1002.eqiad.wmnet with OS bullseye executed with errors:

  • cloudlb1002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 10 2023, 10:13 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudlb1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 10 2023, 10:49 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudlb1002.eqiad.wmnet with OS bullseye completed:

  • cloudlb1002 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307101028_aborrero_4018022_cloudlb1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
aborrero closed this task as Resolved.Jul 10 2023, 3:42 PM
aborrero updated the task description. (Show Details)
fnegri moved this task from Backlog to Done on the cloud-services-team (FY2022/2023-Q4) board.Jul 27 2023, 3:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits