Page MenuHomePhabricator
Create Task
Maniphest T346891

cloudcontrol1006: move to new network setup
Closed, ResolvedPublic
Actions

Description

The cloudcontrol1006 server should move to a new network setup.

We should:

  • drop wikimedia.org domain in favor of .eqiad.wmnet.
  • drop connection to asw
  • add private.eqiad.wikimedia.cloud address

Following procedure at https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Move_existing_server_between_rows/racks,_changing_IPs

Details

SubjectRepoBranchLines +/-
site: Put cloudcontrol1006 back into serviceoperations/puppetproduction+11 -1
Take cloudcontrol1006 out of serviceoperations/puppetproduction+4 -30
hieradata: move prometheus-openstack-exporter to cloudcontrol1007operations/puppetproduction+1 -1
hieradata: move Galera primary to cloudcontrol1007operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero created this task.Sep 20 2023, 12:13 PM
aborrero moved this task from Backlog to Radar on the User-aborrero board.
aborrero mentioned this in T341494: cloud @ eqiad: hardware re-racking plan.
aborrero updated the task description. (Show Details)Sep 20 2023, 12:17 PM
aborrero mentioned this in T341060: openstack eqiad1: introduce cloud-private and cloudlb.Sep 20 2023, 12:23 PM
• taavi removed a project: Goal.Sep 21 2023, 8:27 AM
aborrero updated the task description. (Show Details)Sep 21 2023, 10:36 AM
• taavi added a comment.Sep 21 2023, 10:36 AM

This is the active`maintain-dbusers` server at the moment. Moving that requires updating the database grants and firewall rules.

gerritbot added a comment.Sep 27 2023, 9:47 AM

Change 961348 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: move Galera primary to cloudcontrol1007

https://gerrit.wikimedia.org/r/961348

gerritbot added a project: Patch-For-Review.Sep 27 2023, 9:47 AM

Change 961349 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: move prometheus-openstack-exporter to cloudcontrol1007

https://gerrit.wikimedia.org/r/961349

• taavi moved this task from Backlog to In progress on the cloud-services-team (FY2023/2024-Q1-Q2) board.Sep 27 2023, 9:48 AM
gerritbot added a comment.Sep 27 2023, 9:51 AM

Change 961348 merged by Majavah:

[operations/puppet@production] hieradata: move Galera primary to cloudcontrol1007

https://gerrit.wikimedia.org/r/961348

gerritbot added a comment.Sep 27 2023, 9:51 AM

Change 961349 merged by Majavah:

[operations/puppet@production] hieradata: move prometheus-openstack-exporter to cloudcontrol1007

https://gerrit.wikimedia.org/r/961349

Maintenance_bot removed a project: Patch-For-Review.Sep 27 2023, 10:10 AM
gerritbot added a comment.Sep 27 2023, 6:31 PM

Change 961442 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Take cloudcontrol1006 out of service

https://gerrit.wikimedia.org/r/961442

gerritbot added a project: Patch-For-Review.Sep 27 2023, 6:31 PM
gerritbot added a comment.Sep 28 2023, 7:43 AM

Change 961442 merged by Majavah:

[operations/puppet@production] Take cloudcontrol1006 out of service

https://gerrit.wikimedia.org/r/961442

ops-monitoring-bot added a comment.Sep 28 2023, 7:55 AM

cookbooks.sre.hosts.decommission executed by taavi@cumin1001 for hosts: cloudcontrol1006.wikimedia.org

  • cloudcontrol1006.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
• taavi reassigned this task from • taavi to Jclark-ctr.Sep 28 2023, 7:56 AM
• taavi updated the task description. (Show Details)
• taavi subscribed.

Hi, this host is ready to be moved. Thanks!

Maintenance_bot removed a project: Patch-For-Review.Sep 28 2023, 8:10 AM
Jclark-ctr added a comment.Sep 29 2023, 2:28 PM

@taavi Moved Server. cable id# 5310 port 46

Jclark-ctr updated the task description. (Show Details)Sep 29 2023, 2:30 PM
• taavi claimed this task.Sep 29 2023, 2:57 PM
• taavi updated the task description. (Show Details)Oct 2 2023, 7:38 AM
gerritbot added a comment.Oct 2 2023, 7:39 AM

Change 962355 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] site: Put cloudcontrol1006 back into service

https://gerrit.wikimedia.org/r/962355

gerritbot added a project: Patch-For-Review.Oct 2 2023, 7:39 AM
ops-monitoring-bot added a comment.Oct 2 2023, 7:47 AM

Cookbook cookbooks.sre.hosts.reimage was started by taavi@cumin1001 for host cloudcontrol1006.eqiad.wmnet with OS bullseye

gerritbot added a comment.Oct 2 2023, 7:47 AM

Change 962355 merged by Majavah:

[operations/puppet@production] site: Put cloudcontrol1006 back into service

https://gerrit.wikimedia.org/r/962355

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2023, 8:10 AM
ops-monitoring-bot added a comment.Oct 2 2023, 9:06 AM

Cookbook cookbooks.sre.hosts.reimage started by taavi@cumin1001 for host cloudcontrol1006.eqiad.wmnet with OS bullseye completed:

  • cloudcontrol1006 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310020834_taavi_3922148_cloudcontrol1006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
cmooney mentioned this in T316544: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+.Oct 2 2023, 9:17 AM
• taavi closed this task as Resolved.Oct 2 2023, 10:00 AM
fnegri moved this task from In progress to Done on the cloud-services-team (FY2023/2024-Q1-Q2) board.Oct 3 2023, 5:59 PM
• taavi closed subtask T347381: Support maintain-dbusers on the new network layout as Resolved.Nov 20 2023, 12:36 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits