Page MenuHomePhabricator
Create Task
Maniphest T345602

Repurpose three decom servers as temporary ganeti-test1001/1002 and ganeti-test2004
Closed, ResolvedPublic
Actions

Description

Tracking task to not forget about them, from T344598#9114384

Details

SubjectRepoBranchLines +/-
Remove mentions of ganeti-test1001/2 and 2004operations/puppetproduction+0 -25
Change ganeti-test2004's role to ganeti_testoperations/puppetproduction+1 -1
ganeti-test2004: add to Puppetoperations/puppetproduction+12 -0
DNS add A for ganeti-test01.svc.eqiad.wmnetoperations/dnsmaster+2 -1
eqiad ganeti test setupoperations/puppetproduction+13 -6
ganeti-test100[12]: assign ganeti-test roleoperations/puppetproduction+1 -1
Add ganeti-test01.svc.eqiad.wmnet public certoperations/puppetproduction+23 -0
Add mock TLS key for ganeti-test01.svc.eqiad.wmnetlabs/privatemaster+3 -0
Add temp ganeti-test hostsoperations/puppetproduction+6 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

• ayounsi created this task.Sep 5 2023, 8:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 5 2023, 8:47 AM
gerritbot added a comment.Sep 5 2023, 11:04 AM

Change 954898 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Add temp ganeti-test hosts

https://gerrit.wikimedia.org/r/954898

gerritbot added a project: Patch-For-Review.Sep 5 2023, 11:04 AM
gerritbot added a comment.Sep 5 2023, 11:08 AM

Change 954898 merged by Ayounsi:

[operations/puppet@production] Add temp ganeti-test hosts

https://gerrit.wikimedia.org/r/954898

Maintenance_bot removed a project: Patch-For-Review.Sep 5 2023, 11:10 AM
ops-monitoring-bot added a comment.Sep 5 2023, 11:18 AM

Cookbook cookbooks.sre.hosts.reimage was started by ayounsi@cumin1001 for host ganeti-test1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 11:51 AM

Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1001 for host ganeti-test1001.eqiad.wmnet with OS bullseye executed with errors:

  • ganeti-test1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 5 2023, 11:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by ayounsi@cumin1001 for host ganeti-test1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 12:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by ayounsi@cumin1001 for host ganeti-test1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 1:03 PM

Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1001 for host ganeti-test1001.eqiad.wmnet with OS bullseye completed:

  • ganeti-test1001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051217_ayounsi_3956565_ganeti-test1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 5 2023, 1:16 PM

Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1001 for host ganeti-test1002.eqiad.wmnet with OS bullseye completed:

  • ganeti-test1002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051247_ayounsi_3988529_ganeti-test1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
gerritbot added a comment.Sep 5 2023, 2:04 PM

Change 954924 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[labs/private@master] Add mock TLS key for ganeti-test01.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/954924

gerritbot added a project: Patch-For-Review.Sep 5 2023, 2:04 PM

Change 954924 merged by Ayounsi:

[labs/private@master] Add mock TLS key for ganeti-test01.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/954924

Maintenance_bot removed a project: Patch-For-Review.Sep 5 2023, 2:10 PM
gerritbot added a comment.Sep 5 2023, 2:18 PM

Change 954946 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Add ganeti-test01.svc.eqiad.wmnet public cert

https://gerrit.wikimedia.org/r/954946

gerritbot added a project: Patch-For-Review.Sep 5 2023, 2:18 PM
gerritbot added a comment.Sep 5 2023, 2:21 PM

Change 954946 merged by Ayounsi:

[operations/puppet@production] Add ganeti-test01.svc.eqiad.wmnet public cert

https://gerrit.wikimedia.org/r/954946

Maintenance_bot removed a project: Patch-For-Review.Sep 5 2023, 2:29 PM
gerritbot added a comment.Sep 5 2023, 2:31 PM

Change 954950 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] ganeti-test100[12]: assign ganeti-test role

https://gerrit.wikimedia.org/r/954950

gerritbot added a project: Patch-For-Review.Sep 5 2023, 2:31 PM
• ayounsi mentioned this in rLPRIa3ecf8444cd0: Add mock TLS key for ganeti-test01.svc.eqiad.wmnet.Sep 5 2023, 2:34 PM
gerritbot added a comment.Sep 5 2023, 2:39 PM

Change 954950 merged by Ayounsi:

[operations/puppet@production] ganeti-test100[12]: assign ganeti-test role

https://gerrit.wikimedia.org/r/954950

Maintenance_bot removed a project: Patch-For-Review.Sep 5 2023, 3:10 PM
gerritbot added a comment.Sep 6 2023, 7:16 AM

Change 955284 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] eqiad ganeti test setup

https://gerrit.wikimedia.org/r/955284

gerritbot added a project: Patch-For-Review.Sep 6 2023, 7:16 AM
• ayounsi added a parent task: T300152: Investigate Ganeti in routed mode.Sep 6 2023, 7:20 AM
gerritbot added a comment.Sep 6 2023, 8:22 AM

Change 955284 merged by Ayounsi:

[operations/puppet@production] eqiad ganeti test setup

https://gerrit.wikimedia.org/r/955284

Maintenance_bot removed a project: Patch-For-Review.Sep 6 2023, 8:30 AM
gerritbot added a comment.Sep 6 2023, 9:44 AM

Change 955299 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/dns@master] DNS add A for ganeti-test01.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/955299

gerritbot added a project: Patch-For-Review.Sep 6 2023, 9:44 AM

Change 955299 merged by Ayounsi:

[operations/dns@master] DNS add A for ganeti-test01.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/955299

Maintenance_bot removed a project: Patch-For-Review.Sep 6 2023, 10:10 AM
• ayounsi renamed this task from Repurpose two decom servers as temporary ganeti-test1001/1002 to Repurpose three decom servers as temporary ganeti-test1001/1002 and ganeti-test2004.Sep 22 2023, 12:35 PM
• ayounsi updated the task description. (Show Details)
• ayounsi mentioned this in T345867: Decommission furud.
• ayounsi updated the task description. (Show Details)Sep 22 2023, 12:37 PM
ops-monitoring-bot added a comment.Oct 5 2023, 2:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ganeti-test2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 5 2023, 3:31 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ganeti-test2004.codfw.wmnet with OS bullseye executed with errors:

  • ganeti-test2004 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 5 2023, 6:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ganeti-test2004.codfw.wmnet with OS bullseye

Jhancock.wm added subscribers: Papaul, Jhancock.wm.Oct 5 2023, 7:16 PM

I'm still having trouble getting this ganeti-test2004 to behave. the mgmt and network addresses ping. the idrac is inaccessible through the web gui, but is through the root login. the idrac version is 2.83.83.83.

I tried to install the os anyway but it fails when it tries to set up a raid. there is no raid controller on this one. Are there any tips for fixing these problems? thanks ++ @Papaul

Papaul added a comment.Oct 5 2023, 7:35 PM

@Jhancock.wm first thing first you need to upgrade the idrac

ops-monitoring-bot added a comment.Oct 6 2023, 12:16 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ganeti-test2004.codfw.wmnet with OS bullseye executed with errors:

  • ganeti-test2004 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Oct 6 2023, 11:42 AM

Change 963998 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] ganeti-test2004: add to Puppet

https://gerrit.wikimedia.org/r/963998

gerritbot added a project: Patch-For-Review.Oct 6 2023, 11:42 AM
gerritbot added a comment.Oct 6 2023, 11:49 AM

Change 963998 merged by Ayounsi:

[operations/puppet@production] ganeti-test2004: add to Puppet

https://gerrit.wikimedia.org/r/963998

ops-monitoring-bot added a comment.Oct 6 2023, 11:55 AM

Cookbook cookbooks.sre.hosts.reimage was started by ayounsi@cumin1001 for host ganeti-test2004.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Oct 6 2023, 12:10 PM
ops-monitoring-bot added a comment.Oct 6 2023, 12:13 PM

Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1001 for host ganeti-test2004.codfw.wmnet with OS bullseye executed with errors:

  • ganeti-test2004 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 6 2023, 12:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by ayounsi@cumin1001 for host ganeti-test2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2023, 1:03 PM

Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1001 for host ganeti-test2004.codfw.wmnet with OS bullseye executed with errors:

  • ganeti-test2004 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Oct 6 2023, 4:22 PM

Change 964036 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Change ganeti-test2004's role to ganeti_test

https://gerrit.wikimedia.org/r/964036

gerritbot added a project: Patch-For-Review.Oct 6 2023, 4:22 PM
gerritbot added a comment.Oct 9 2023, 6:29 AM

Change 964036 merged by Ayounsi:

[operations/puppet@production] Change ganeti-test2004's role to ganeti_test

https://gerrit.wikimedia.org/r/964036

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2023, 6:30 AM
ops-monitoring-bot added a comment.Jan 9 2024, 5:05 PM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1002 for hosts: ganeti-test[1001-1002].eqiad.wmnet

  • ganeti-test1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ganeti-test1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Jan 9 2024, 5:17 PM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1002 for hosts: ganeti-test2004.codfw.wmnet

  • ganeti-test2004.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Jan 9 2024, 5:25 PM

Change 989212 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Remove mentions of ganeti-test1001/2 and 2004

https://gerrit.wikimedia.org/r/989212

gerritbot added a project: Patch-For-Review.Jan 9 2024, 5:25 PM
• ayounsi removed • ayounsi as the assignee of this task.Jan 9 2024, 5:41 PM
• ayounsi added projects: ops-eqiad, ops-codfw.

Those 3 servers have been decommissioned. Over to DCops to finish the process.

Thanks, they've been quite useful.

RobH added a subtask: T354680: decommission ganeti-test1001, ganeti-test1002.Jan 9 2024, 5:45 PM
RobH added a subtask: T354681: decommission ganeti-test2004.
RobH closed this task as Resolved.Jan 9 2024, 5:47 PM
RobH claimed this task.
RobH subscribed.

T354680 and T354681 filed for decom

Maintenance_bot added a project: SRE.Jan 9 2024, 6:29 PM
gerritbot added a comment.Jan 10 2024, 8:24 AM

Change 989212 merged by Ayounsi:

[operations/puppet@production] Remove mentions of ganeti-test1001/2 and 2004

https://gerrit.wikimedia.org/r/989212

Maintenance_bot removed a project: Patch-For-Review.Jan 10 2024, 8:30 AM
Jhancock.wm closed subtask T354681: decommission ganeti-test2004 as Resolved.Jan 10 2024, 3:03 PM
VRiley-WMF closed subtask T354680: decommission ganeti-test1001, ganeti-test1002 as Resolved.Jan 10 2024, 4:36 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits