Page MenuHomePhabricator
Create Task
Maniphest T345803

Connect two hosts in codfw row A/B for switch migration testing
Closed, ResolvedPublic
Actions

Description

Hi Papaul,

As discussed if it was possible to set up two hosts in codfw for migration testing that would be great.

Doesn't really matter where (except rack B1 obviously), probably one in each row but whatever is easiest for you guys. Also shouldn't make much difference if 1G or 10G.

Don't worry about running the provision for it in Netbox I can take care of that if you just post back what ports they're connected to.

Thanks!

Details

SubjectRepoBranchLines +/-
Specify partman receipe for sretest2003 & sretest2004operations/puppetproduction+1 -0
Include two new temp codfw sretest hosts in site.ppoperations/puppetproduction+1 -1
Customize query in gerrit

Event Timeline

cmooney triaged this task as Low priority.Sep 7 2023, 8:26 AM
cmooney created this task.
Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptSep 7 2023, 8:26 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
cmooney updated the task description. (Show Details)Sep 7 2023, 8:46 AM
Papaul moved this task from Backlog to Racking Tasks on the ops-codfw board.Sep 7 2023, 1:21 PM
Papaul added a comment.Sep 7 2023, 4:00 PM

We will be using to test the new codfw spine/leaf new design contint2001 and thumbor2004. contint2001 will be rename to sretest2003 and thumbor2004 to sretest2004. both server are plugged in port 41
Please leave this task assigned to me and don't close it once the testing please decommission the nodes and update this task

Thanks

cmooney added a comment.Sep 7 2023, 4:41 PM

Thanks @Papaul !

cmooney added a parent task: T327938: Codfw row A/B top-of-rack switch refresh.Sep 28 2023, 9:31 PM
gerritbot added a comment.Oct 18 2023, 8:46 PM

Change 966928 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Include two new temp codfw sretest hosts in site.pp

https://gerrit.wikimedia.org/r/966928

gerritbot added a project: Patch-For-Review.Oct 18 2023, 8:46 PM
gerritbot added a comment.Oct 18 2023, 8:52 PM

Change 966928 merged by Cathal Mooney:

[operations/puppet@production] Include two new temp codfw sretest hosts in site.pp

https://gerrit.wikimedia.org/r/966928

gerritbot added a comment.Oct 18 2023, 9:00 PM

Change 966932 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Specify partman receipe for sretest2003 & sretest2004

https://gerrit.wikimedia.org/r/966932

gerritbot added a comment.Oct 18 2023, 9:19 PM

Change 966932 merged by Cathal Mooney:

[operations/puppet@production] Specify partman receipe for sretest2003 & sretest2004

https://gerrit.wikimedia.org/r/966932

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 9:30 PM
Papaul added a subscriber: Jhancock.wm.Oct 20 2023, 4:25 AM

@cmooney @Jhancock.wm checked the server, no IP address set on it and she did reset it but it didn't resolve the issue. I asked to to upgrade the IDRAC firmware. I have seen that if the IDRAC version is too old the Redfish API will sometimes not be able to connect to the server so let see what she comes up with tomorrow after updating the IDRAC. Thanks

ops-monitoring-bot added a comment.Oct 26 2023, 2:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host sretest2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 27 2023, 6:12 AM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host sretest2003.codfw.wmnet with OS bullseye executed with errors:

  • sretest2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310261512_cmooney_904039_sretest2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 27 2023, 1:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host sretest2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 27 2023, 1:37 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host sretest2004.codfw.wmnet with OS bullseye executed with errors:

  • sretest2004 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 27 2023, 1:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host sretest2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 27 2023, 3:38 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host sretest2004.codfw.wmnet with OS bullseye completed:

  • sretest2004 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310271442_cmooney_1456159_sretest2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Nov 13 2023, 12:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host sretest2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 13 2023, 12:42 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host sretest2004.codfw.wmnet with OS bullseye executed with errors:

  • sretest2004 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 13 2023, 12:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host sretest2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 13 2023, 1:53 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host sretest2003.codfw.wmnet with OS bullseye executed with errors:

  • sretest2003 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 15 2023, 12:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host sretest2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 15 2023, 1:14 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host sretest2003.codfw.wmnet with OS bullseye completed:

  • sretest2003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311151252_cmooney_1178757_sretest2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Nov 15 2023, 2:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host sretest2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 15 2023, 3:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host sretest1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 15 2023, 6:56 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host sretest1001.eqiad.wmnet with OS bullseye completed:

  • sretest1001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202311151528_cmooney_1257572_sretest1001.out, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311151832_cmooney_1257572_sretest1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 16 2023, 6:07 AM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host sretest2004.codfw.wmnet with OS bullseye executed with errors:

  • sretest2004 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311151508_cmooney_1246017_sretest2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
    • The reimage failed, see the cookbook logs for the details
Papaul added a comment.Jan 22 2024, 11:46 PM

@cmooney can we get those 2 hosts back in decom? Thanks

cmooney added a comment.Feb 27 2024, 2:53 PM
In T345803#9479281, @Papaul wrote:

@cmooney can we get those 2 hosts back in decom? Thanks

I'm done with sretest2004 now, and I've run the decom cookbook for it.

sretest2003 let's leave for a few more weeks just in case there are any more niggles in the new racks, it's useful to have something to play about with there. I'll update this ticket when done. thanks!

ops-monitoring-bot added a comment.Feb 27 2024, 3:00 PM

cookbooks.sre.hosts.decommission executed by cmooney@cumin1002 for hosts: sretest2004.codfw.wmnet

  • sretest2004.codfw.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

cmooney added a comment.Mar 22 2024, 4:40 PM
In T345803#9479281, @Papaul wrote:

@cmooney can we get those 2 hosts back in decom? Thanks

@Papaul I'm done with these now. I'm running the decom cookbook for sretest2003, let me know if you need anything else.

Thanks!

cmooney removed a parent task: T327938: Codfw row A/B top-of-rack switch refresh.Mar 22 2024, 4:43 PM
ops-monitoring-bot added a comment.Mar 22 2024, 5:12 PM

cookbooks.sre.hosts.decommission executed by cmooney@cumin1002 for hosts: sretest2003.codfw.wmnet

  • sretest2003.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Jhancock.wm added a comment.Mar 25 2024, 5:02 PM

sretest2003 and 2004 have been renamed to their original server names and been offlined (including ssd removal).

Papaul closed this task as Resolved.Mar 26 2024, 6:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL