Page MenuHomePhabricator
Create Task
Maniphest T346892

cloudcontrol1007: move to new network setup
Closed, ResolvedPublic
Actions

Description

The cloudcontrol1007 server should move to a new network setup.

We should:

  • drop wikimedia.org domain in favor of .eqiad.wmnet.
  • drop connection to asw
  • add private.eqiad.wikimedia.cloud address

Following procedure at https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Rename_while_reimaging

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
site: re-assign role for cloudcontrol1007operations/puppetproduction+13 -6
Remove most references to cloudcontrol1007operations/puppetproduction+3 -14
Customize query in gerrit

Related Objects
Search...

Event Timeline

• aborrero created this task.Sep 20 2023, 12:16 PM
• aborrero updated the task description. (Show Details)
• aborrero reassigned this task from Jclark-ctr to taavi.Sep 20 2023, 12:19 PM
• aborrero mentioned this in T341494: cloud @ eqiad: hardware re-racking plan.
• aborrero updated the task description. (Show Details)
• aborrero moved this task from Backlog to Radar/observer on the User-aborrero board.Sep 20 2023, 12:23 PM
• aborrero mentioned this in T341060: openstack eqiad1: introduce cloud-private and cloudlb.
taavi removed a project: Goal.Sep 21 2023, 8:27 AM
taavi updated the task description. (Show Details)
• aborrero mentioned this in T346875: cookbook: sre.hosts.decommission: also remove logical volumes (to allow rename while reimage).Sep 21 2023, 8:35 AM
gerritbot added a comment.Sep 21 2023, 8:41 AM

Change 959677 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Remove most references to cloudcontrol1007

https://gerrit.wikimedia.org/r/959677

gerritbot added a project: Patch-For-Review.Sep 21 2023, 8:41 AM

Change 959677 merged by Majavah:

[operations/puppet@production] Remove most references to cloudcontrol1007

https://gerrit.wikimedia.org/r/959677

ops-monitoring-bot added a comment.Sep 21 2023, 9:00 AM

cookbooks.sre.hosts.decommission executed by taavi@cumin1001 for hosts: cloudcontrol1007.wikimedia.org

  • cloudcontrol1007.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
taavi updated the task description. (Show Details)Sep 21 2023, 9:00 AM
Maintenance_bot removed a project: Patch-For-Review.Sep 21 2023, 9:10 AM
taavi moved this task from Backlog to In progress on the cloud-services-team (FY2023/2024-Q1-Q2) board.Sep 21 2023, 10:13 AM
taavi reassigned this task from taavi to Jclark-ctr.Sep 21 2023, 10:30 AM
taavi subscribed.

hi @Jclark-ctr! This server has been powered off and can be moved at any time to E4. thanks!

• aborrero updated the task description. (Show Details)Sep 21 2023, 10:35 AM
Jclark-ctr added a comment.Sep 25 2023, 2:12 PM

@taavi Relovated to rack E 4. updated netbox with location. switch port is #9

Jclark-ctr updated the task description. (Show Details)Sep 25 2023, 2:13 PM
Jclark-ctr reassigned this task from Jclark-ctr to taavi.Sep 25 2023, 2:21 PM
gerritbot added a comment.Sep 25 2023, 3:26 PM

Change 960642 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] site: re-assign role for cloudcontrol1007

https://gerrit.wikimedia.org/r/960642

gerritbot added a project: Patch-For-Review.Sep 25 2023, 3:26 PM
taavi updated the task description. (Show Details)Sep 25 2023, 3:29 PM
gerritbot added a comment.Sep 26 2023, 6:41 AM

Change 960642 merged by Majavah:

[operations/puppet@production] site: re-assign role for cloudcontrol1007

https://gerrit.wikimedia.org/r/960642

ops-monitoring-bot added a comment.Sep 26 2023, 6:42 AM

Cookbook cookbooks.sre.hosts.reimage was started by taavi@cumin1001 for host cloudcontrol1007.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2023, 7:11 AM
taavi updated the task description. (Show Details)Sep 26 2023, 7:20 AM
ops-monitoring-bot added a comment.Sep 26 2023, 7:55 AM

Cookbook cookbooks.sre.hosts.reimage started by taavi@cumin1001 for host cloudcontrol1007.eqiad.wmnet with OS bullseye completed:

  • cloudcontrol1007 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309260708_taavi_1741001_cloudcontrol1007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
taavi updated the task description. (Show Details)Sep 26 2023, 8:11 AM
taavi closed this task as Resolved.Sep 26 2023, 8:28 AM
fnegri moved this task from In progress to Done on the cloud-services-team (FY2023/2024-Q1-Q2) board.Oct 3 2023, 5:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits