Page MenuHomePhabricator
Create Task
Maniphest T296664

Our need for cloudvirt hypervisors with local disks
Closed, ResolvedPublic
Actions

Description

Until last week we were using cloudvirt1018 to host exactly two VMs:

toolsbeta-test-k8s-etcd-17
tools-k8s-etcd-13

cloudvirt1018 is having RAID problems (and due for refresh) so it's now drained and idle.

Each of the above VMs is part of a three-VM cluster that was spread over cloudvirt1018, 1019 and 1020. Now e have less redundancy as each cluster has two nodes on cloudvirt1019.

We will likely always need two 'fatvirt' hosts for the DB workloads on 1019 and 1020. Do we need a third fatvirt for those other two VMs? It seems like a lot of hardware to use on two VMs.

Details

SubjectRepoBranchLines +/-
Convert cloudvirt1028 into a local storage hypervisoroperations/puppetproduction+11 -6
Customize query in gerrit

Related Objects

Event Timeline

Andrew created this task.Nov 29 2021, 7:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 29 2021, 7:08 PM
Andrew mentioned this in T296592: Degraded RAID on cloudvirt1018.Nov 29 2021, 7:09 PM
taavi edited projects, added Toolforge; removed Tools.Nov 29 2021, 7:09 PM
gerritbot added a comment.Dec 2 2021, 12:59 AM

Change 743055 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Convert cloudvirt1028 into a local storage hypervisor

https://gerrit.wikimedia.org/r/743055

gerritbot added a project: Patch-For-Review.Dec 2 2021, 12:59 AM
gerritbot added a comment.Dec 2 2021, 1:45 AM

Change 743055 merged by Andrew Bogott:

[operations/puppet@production] Convert cloudvirt1028 into a local storage hypervisor

https://gerrit.wikimedia.org/r/743055

ops-monitoring-bot added a comment.Dec 2 2021, 1:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

Maintenance_bot removed a project: Patch-For-Review.Dec 2 2021, 2:10 AM
ops-monitoring-bot added a comment.Dec 2 2021, 2:14 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 2 2021, 2:15 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 2 2021, 2:40 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 2 2021, 2:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 2 2021, 2:50 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 2 2021, 6:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 3 2021, 4:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 3 2021, 4:39 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 3 2021, 4:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 3 2021, 4:42 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 3 2021, 4:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 3 2021, 4:56 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 3 2021, 4:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 3 2021, 5:35 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 3 2021, 5:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 3 2021, 5:47 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 3 2021, 5:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 3 2021, 5:56 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 6 2021, 6:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 6 2021, 6:43 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 6 2021, 6:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 6 2021, 6:52 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 6 2021, 7:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 6 2021, 7:01 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 6 2021, 7:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 6 2021, 7:03 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 7 2021, 7:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 7 2021, 7:56 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 7 2021, 9:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 7 2021, 10:06 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 7 2021, 10:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 7 2021, 11:21 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Downtimed on Icinga
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 8 2021, 3:37 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Dec 8 2021, 4:50 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster completed:

  • cloudvirt1028 (WARN)
    • Downtimed on Icinga
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202112080337_andrew_654_cloudvirt1028.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 11 2021, 7:03 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS buster executed with errors:

  • cloudvirt1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
Andrew closed this task as Resolved.Jan 4 2022, 10:07 PM
Andrew claimed this task.

cloudvirt1028 is now the third localdisk host. No actions left here until the time comes to refresh 1019, 1020 or 1028.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL