Page MenuHomePhabricator
Create Task
Maniphest T382356

replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev
Closed, ResolvedPublic
Actions

Description

T382220 suggests that cloudgw1002 might be having hardware issues, and @aborrero suggests that we replace it.

We have several servers already racked from a cancelled dev experiment that we could use to replace this workload without ordering more hardware.

Existing cloudgw servers:

cloudgw1001: C8; 8 cores, 32GB RAM, 2x 10GB ports, 2x240 sssd, purchased march 2021

cloudgw1002: D5; 8 cores, 32GB RAM, 2x 10GB ports, 2x240 sssd, purchased march 2021

Unused cloud-dev servers:

cloudnet1007-dev: E4; 2x12 cores, 64GB RAM, 2x 10GB ports, 4x960 ssd, purchased august 2023

cloudnet1008-dev: F4; 2x12 cores, 64GB RAM, 2x 10GB ports, 4x960 ssd, purchased august 2023

cloudcontrol1008-dev: D5; 2x12 cores, 64GB RAM, 2x 10GB ports, 4x960 ssd, purchased august 2023

cloudcontrol1009-dev: E4; 2x12 cores, 64GB RAM, 2x 10GB ports, 4x960 ssd, purchased august 2023

cloudcontrol1010-dev: F4; 2x12 cores, 64GB RAM, 2x 10GB ports, 4x960 ssd, purchased august 2023

I propose that we replace both cloudgw100[12] servers (not at the same time, of course) with renamed cloudnet100[78] boxes.

There are a couple of caveats:

  1. Does it matter that the replacement servers are in different racks? Pinging @aborrero and @cmooney for an answer
  1. Is renaming servers in place in a datacenter so awful that we should never ever do it? Pinging @RobH for an answer

Details

SubjectRepoBranchLines +/-
cloudgw1003: take over cloudgw1001operations/puppetproduction+13 -13
cloudgw1003: replace cloudgw1001operations/puppetproduction+2 -2
cloudgw100[34]: specify puppet 7operations/puppetproduction+21 -0
cloudgw1004: take over cloudgw1002operations/puppetproduction+13 -13
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.Dec 17 2024, 6:46 PM
Restricted Application edited projects, added cloud-services-team; removed cloud-services-team (FY2024/2025-Q1-Q2). · View Herald TranscriptDec 17 2024, 6:46 PM
Andrew added a parent task: T342455: Q1:rack/setup/install cloudcontrol100[8-10]-dev cloudnet100[7-8]-dev.Dec 17 2024, 6:47 PM
fnegri triaged this task as High priority.Dec 18 2024, 10:18 AM
fnegri edited projects, added Cloud-VPS, cloud-services-team (FY2024/2025-Q1-Q2); removed cloud-services-team.
cmooney added a comment.EditedDec 18 2024, 11:49 AM

Does it matter that the replacement servers are in different racks? Pinging @aborrero and @cmooney for an answer

Yeah we'll need to move them. The two cloudgw's should be in C8 and D5 in eqiad. If placed in E4/F4 the flow of traffic to and from those cabs to traverse the cloudgw would be massively sub-optimal and probably lead to congestion.

cloudnet1007-dev: E4; 2x12 cores, 64GB RAM, 2x 10GB ports, 4x960 ssd, purchased august 2023

If the CPUs are running at the same speed going from 8 to 12 does help. However the dual-socket system is a more complex beast, we will need to ensure that all the NIC RX/TX queues are assigned to CPU cores on the socket connected to the same PCIe as the NIC. Not sure if the kernel will do that by default or not. Also definitely worth disabling some of those offloads.

Andrew added a comment.Dec 18 2024, 1:06 PM

Created T382412 about relocating the cloudnet-dev servers.

RobH added a comment.Dec 18 2024, 1:18 PM

Servers can be renamed, but there are a few places that may cause issues if they are not updated.

  • Update the dns name for the IP and the mgmt IP dns entires (ipv4 and ipv6) and run dns update cookbook.
  • Update the name in netbox for the server hostname.
    • When the hostname is updated in netbox, a sub-task for the #ops-sitename queue should be created to apply new hostname labels on the front and back of the server. This is often forgotten, and can lead to confusion when troubleshooting hosts later.

There are even some directions on the lifecycle page and a rename cookbook that I have not personally run so not sure if it is accurate but I'd assume so since automation tends to keep their sections of the lifecycle page well updated. https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Rename_while_reimaging

The main thing folks overlook outside of DC ops is the mgmt IP dns and the physical hostname labels we have to update.

fnegri mentioned this in T382220: KernelError Server cloudgw1002 may have kernel errors.Dec 19 2024, 12:44 PM
Andrew mentioned this in T380805: Repurpose 5 config B servers.Jan 8 2025, 3:21 PM
fnegri edited projects, added cloud-services-team (FY2024/2025-Q3-Q4); removed cloud-services-team (FY2024/2025-Q1-Q2).Jan 14 2025, 5:35 PM
fnegri mentioned this in T382412: Relocate cloudnet1007-dev and cloudnet1008-dev to new racks and rename.Jan 21 2025, 5:20 PM
taavi moved this task from Unsorted to Network on the Cloud-VPS board.Jan 22 2025, 4:15 PM
RobH unsubscribed.Jan 22 2025, 5:15 PM
aborrero added a project: User-aborrero.Jan 28 2025, 1:22 PM
aborrero moved this task from Backlog to Next on the User-aborrero board.
aborrero added a comment.Jan 28 2025, 4:32 PM

reminder: verify VLAN trunk on the NIC of the cloudgw servers.

aborrero added a comment.Jan 29 2025, 12:33 PM
In T382356#10501284, @aborrero wrote:

reminder: verify VLAN trunk on the NIC of the cloudgw servers.

I have reviewed this, and they are correct:

gerritbot added a comment.Jan 29 2025, 12:41 PM

Change #1114997 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw1003: take over cloudgw1001

https://gerrit.wikimedia.org/r/1114997

gerritbot added a project: Patch-For-Review.Jan 29 2025, 12:41 PM
gerritbot added a comment.Jan 29 2025, 12:45 PM

Change #1114998 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw1004: take over cloudgw1002

https://gerrit.wikimedia.org/r/1114998

fnegri removed fnegri as the assignee of this task.Jan 29 2025, 3:40 PM
fnegri assigned this task to aborrero.
aborrero moved this task from Next to Doing on the User-aborrero board.Jan 29 2025, 3:43 PM
aborrero added a comment.Feb 4 2025, 11:51 AM

some interesting grafana dashboards to watch:

aborrero added a comment.EditedFeb 4 2025, 12:23 PM

DNS changes:

Then run:

aborrero@cumin1002:~ $ sudo cookbook sre.dns.netbox -t T382356 'cloudgw updates'

Stashbot added a comment.Feb 4 2025, 12:48 PM

Mentioned in SAL (#wikimedia-cloud) [2025-02-04T12:48:44Z] <arturo> replacing cloudgw1002 with cloudgw1004 - T382356

gerritbot added a comment.Feb 4 2025, 12:53 PM

Change #1114998 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw1004: take over cloudgw1002

https://gerrit.wikimedia.org/r/1114998

ops-monitoring-bot added a comment.Feb 4 2025, 12:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudgw1004.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 4 2025, 12:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudgw1002.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 4 2025, 1:04 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudgw1004.eqiad.wmnet with OS bookworm executed with errors:

  • cloudgw1004 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudgw1004.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 4 2025, 1:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudgw1004.eqiad.wmnet with OS bullseye

aborrero added a comment.Feb 4 2025, 1:10 PM

we decided to reimage the new hosts clodugw1004 and cloudgw1003 on bullseye rather than bookworm, and give a test to the puppet profile on bookworm first on codfw1dev.

gerritbot added a comment.Feb 4 2025, 1:12 PM

Change #1117189 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudgw100[34]: specify puppet 7

https://gerrit.wikimedia.org/r/1117189

gerritbot added a comment.Feb 4 2025, 1:21 PM

Change #1117189 abandoned by Andrew Bogott:

[operations/puppet@production] cloudgw100[34]: specify puppet 7

Reason:

It's set in the cloudgw role already.

https://gerrit.wikimedia.org/r/1117189

ops-monitoring-bot added a comment.Feb 4 2025, 1:35 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudgw1002.eqiad.wmnet with OS bookworm completed:

  • cloudgw1002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202502041317_andrew_1202782_cloudgw1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 4 2025, 1:44 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudgw1004.eqiad.wmnet with OS bullseye completed:

  • cloudgw1004 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202502041327_aborrero_1214295_cloudgw1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Andrew added a comment.Feb 4 2025, 2:02 PM

Now the two hosts are: cloudgw1004 (active), cloudgw1001 (standby). 1002 is ready for decom, and 1003 is ready to be put into service once we're confident about 1004.

aborrero added a comment.Feb 4 2025, 2:02 PM

cloudgw1004 took over cloudgw1002 successfully, and is now sustaining high traffic normally.

We will continue with cloudgw1001 in a couple of weeks.

aborrero mentioned this in T385600: Cloud DNS: investigate weird graph during cloudgw replacement operation.Feb 4 2025, 4:23 PM
aborrero closed subtask T385600: Cloud DNS: investigate weird graph during cloudgw replacement operation as Invalid.Feb 4 2025, 4:42 PM
aborrero moved this task from Doing to Next on the User-aborrero board.Feb 4 2025, 4:44 PM
Andrew closed subtask T382412: Relocate cloudnet1007-dev and cloudnet1008-dev to new racks and rename as Resolved.Feb 11 2025, 6:13 PM
Andrew mentioned this in T374830: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts such as gerrit.wikimedia.org.Feb 13 2025, 2:19 PM
gerritbot added a comment.Tue, Feb 18, 1:19 PM

Change #1120548 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudgw1003: replace cloudgw1001

https://gerrit.wikimedia.org/r/1120548

gerritbot added a comment.Tue, Feb 18, 1:21 PM

Change #1120548 abandoned by Andrew Bogott:

[operations/puppet@production] cloudgw1003: replace cloudgw1001

https://gerrit.wikimedia.org/r/1120548

gerritbot added a comment.Wed, Feb 19, 11:43 AM

Change #1114997 merged by Andrew Bogott:

[operations/puppet@production] cloudgw1003: take over cloudgw1001

https://gerrit.wikimedia.org/r/1114997

ops-monitoring-bot added a comment.Wed, Feb 19, 11:48 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudgw1003.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Wed, Feb 19, 11:49 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudgw1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, Feb 19, 12:27 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudgw1003.eqiad.wmnet with OS bullseye executed with errors:

  • cloudgw1003 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • New OS is bookworm but bullseye was requested
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudgw1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Wed, Feb 19, 12:28 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudgw1001.eqiad.wmnet with OS bookworm completed:

  • cloudgw1001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202502191210_andrew_1575642_cloudgw1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
fnegri mentioned this in T386111: PuppetDisabled Puppet disabled on cloudgw1002:9100.Wed, Feb 19, 12:28 PM
ops-monitoring-bot added a comment.Wed, Feb 19, 12:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudgw1003.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Wed, Feb 19, 12:30 PM
ops-monitoring-bot added a comment.Wed, Feb 19, 1:09 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudgw1003.eqiad.wmnet with OS bullseye completed:

  • cloudgw1003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202502191250_andrew_1589102_cloudgw1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Andrew added a subtask: T386810: decommission cloudgw100[12].Wed, Feb 19, 1:20 PM
aborrero closed this task as Resolved.Wed, Feb 19, 1:26 PM
Stashbot added a comment.Wed, Feb 19, 1:26 PM

Mentioned in SAL (#wikimedia-cloud) [2025-02-19T13:26:48Z] <arturo> manual failover of cloudgw1004 to cloudgw1003 T382356

aborrero mentioned this in T381078: cloudgw: suspected network problems.Wed, Feb 19, 1:28 PM
aborrero mentioned this in T386907: FQDN virt.cloudgw.eqiad1.wikimediacloud.org is missing.Thu, Feb 20, 9:45 AM
VRiley-WMF closed subtask T386810: decommission cloudgw100[12] as Resolved.Thu, Feb 27, 5:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits