Page MenuHomePhabricator
Create Task
Maniphest T231068

Spicerack: improve support for Ganeti VMs
Closed, ResolvedPublic
Actions

Description

In order to be able to perform some actions on VMs like on physical hosts (for example when decommissioning), some improvement is needed in the Netbox and Ganeti support in Spicerack.

Namely:

  • given a FQDN an easy way to get if the host is physical or virtual (ganeti). We could do that with a cumin query, but seems more correct to get the info from Netbox probably.
  • given a FQDN of a Ganeti VM get the Ganeti cluster it belongs (that in turn would allow to perform Ganeti CLI actions). This should probably come from Netbox.
  • evaluate if we should support also FQDNs in the Netbox module transparently in addition to hostnames
  • [optional] evaluate the possibility to expose some GanetiCLI object from the ganeti spicerack interface to allow to perform gnt commands directly on the right master host.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedVolansT231066 Host decommission improvements
 Resolved• crusnovT231068 Spicerack: improve support for Ganeti VMs

Event Timeline

Volans created this task.Aug 23 2019, 10:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 23 2019, 10:39 AM
Volans triaged this task as Medium priority.Aug 23 2019, 10:39 AM
Volans updated the task description. (Show Details)Aug 23 2019, 10:46 AM
• crusnov added a comment.Aug 29 2019, 4:07 AM

Finding which cluster, or if the instance by fqdn is a Ganeti instance, could be done as easily as trrying to look it up in every configured cluster, and checking if there's information. We could provide utility functions to perform those actions trivially.

Supporting FQDN in netbox should be easy enough.

There is already a utility function to get the master node for the ganeti cluster, so making something that uses that to execute gnt commands would be very straight forward if that seems reasonable.

gerritbot added a comment.Sep 3 2019, 3:29 AM

Change 533984 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/software/spicerack@master] ganeti: Add ability to get ganeti cluster for given instance

https://gerrit.wikimedia.org/r/533984

gerritbot added a project: Patch-For-Review.Sep 3 2019, 3:29 AM
gerritbot added a comment.Sep 3 2019, 4:44 AM

Change 533987 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/software/spicerack@master] netbox: Transparently support read-only operations for virtual machines

https://gerrit.wikimedia.org/r/533987

• crusnov added a comment.Sep 3 2019, 4:47 AM

Okay I've implemented changes to the netbox and ganeti modules as linked above which should allow all of the operations requested. I have not implemented writing status to Ganeti VMs since this information is updated automatically but it should be relatively straight forward to implement if desired.

Basically the change to the ganeti module allows you to discover the cluster that a particular machine belongs to (and indeed, if it belongs to any). The change to netbox treats VMs transparently as deices for read purposes, and marks the information returned by fetch_host_detail with is_virtual and cluster_name.

In terms of allowing running commands on the Ganeti host (the gnt* commands), that will be a separate patch if we stil lwant it.

gerritbot added a comment.Sep 19 2019, 2:47 PM

Change 533984 merged by CRusnov:
[operations/software/spicerack@master] ganeti: Add ability to get ganeti cluster for given instance

https://gerrit.wikimedia.org/r/533984

gerritbot added a comment.Sep 24 2019, 4:55 PM

Change 533987 merged by CRusnov:
[operations/software/spicerack@master] netbox: Transparently support read-only operations for virtual machines

https://gerrit.wikimedia.org/r/533987

Maintenance_bot removed a project: Patch-For-Review.Sep 24 2019, 5:10 PM
• crusnov added a comment.Nov 25 2019, 4:42 PM

The non-optional ask on this is complete. I will leave this open to track the gnt* command proxying.

• crusnov lowered the priority of this task from Medium to Low.Dec 9 2019, 6:28 AM
Volans added a comment.Jan 20 2020, 10:56 AM

Currently we have a bunch of race conditions in the decommission path of VMs. The current actions can be summarized in:

  • an operator runs the decommission cookbook on a VM, removing it from PuppetDB
  • an operator removes the Ganeti VM manually from the cluster
  • at any point the auto-sync of ganeti will update Netbox removing the VM from it
  • at any point the PuppetDB report will run

Based on the timeline of those 4 events in some cases the PuppetDB Netbox report will alert for a bit before auto-resolving itself once all 4 steps have run. This generate unnecessary noise and should be avoided.

I think that the shortest path to solve this is:

  • add a simple support for Ganeti gnt-* commands on spicerack, at least supporting remove with --force and --shutdown-timeout=0 for now.
  • add to the decommission cookbook the removal of the Ganeti VM directly, probably right before the puppetdb removal
  • add to the decommission cookbook the force run of Ganety VM sync for the affected cluster to ensure Netbox is up to date

With this in place the race condition will be limited to few seconds between PuppetDB removal and Netbox sync update.
If deemed necessary we could improve this even more pausing the PuppetDB report during the execution of the cookbook.

Volans raised the priority of this task from Low to Medium.Jan 20 2020, 11:14 AM
gerritbot added a comment.Jan 20 2020, 3:27 PM

Change 566054 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] ganeti: add initial support for gnt-instance

https://gerrit.wikimedia.org/r/566054

gerritbot added a project: Patch-For-Review.Jan 20 2020, 3:27 PM
gerritbot added a comment.Jan 23 2020, 6:09 PM

Change 566054 merged by jenkins-bot:
[operations/software/spicerack@master] ganeti: add initial support for gnt-instance

https://gerrit.wikimedia.org/r/566054

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2020, 6:10 PM
gerritbot added a comment.Jan 25 2020, 12:41 AM

Change 567164 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] spicerack: add getter for the Netbox master host

https://gerrit.wikimedia.org/r/567164

gerritbot added a project: Patch-For-Review.Jan 25 2020, 12:41 AM

Change 567168 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] ganeti: add cluster to GntInstance

https://gerrit.wikimedia.org/r/567168

gerritbot added a comment.Jan 25 2020, 12:48 AM

Change 567169 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.hosts.decommission: improve Ganeti VM support

https://gerrit.wikimedia.org/r/567169

Volans added a comment.Jan 25 2020, 12:52 AM
In T231068#5816225, @Volans wrote:

I think that the shortest path to solve this is:

  • add a simple support for Ganeti gnt-* commands on spicerack, at least supporting remove with --force and --shutdown-timeout=0 for now.
  • add to the decommission cookbook the removal of the Ganeti VM directly, probably right before the puppetdb removal
  • add to the decommission cookbook the force run of Ganety VM sync for the affected cluster to ensure Netbox is up to date

Plan implemented in the above CRs, pending review. The cookbook issues first the gnt shutdown that is quick and then forces the Geneti-Netbox sync and performs the gnt remove only at the end to reduce the race condition window with the Netbox report.

gerritbot added a comment.Jan 25 2020, 1:23 AM

Change 567175 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] netbox: rename injected property in host details

https://gerrit.wikimedia.org/r/567175

gerritbot added a comment.Jan 30 2020, 10:07 PM

Change 567164 merged by jenkins-bot:
[operations/software/spicerack@master] spicerack: add getter for the Netbox master host

https://gerrit.wikimedia.org/r/567164

gerritbot added a comment.Jan 30 2020, 10:07 PM

Change 567168 merged by jenkins-bot:
[operations/software/spicerack@master] ganeti: add cluster to instance()

https://gerrit.wikimedia.org/r/567168

gerritbot added a comment.Jan 30 2020, 10:11 PM

Change 567175 merged by jenkins-bot:
[operations/software/spicerack@master] netbox: rename injected property in host details

https://gerrit.wikimedia.org/r/567175

gerritbot added a comment.Feb 12 2020, 6:46 PM

Change 571780 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] ganeti: use canonical cluster names

https://gerrit.wikimedia.org/r/571780

gerritbot added a comment.Feb 13 2020, 3:08 PM

Change 571997 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] ganeti: add logging for GntInstance actions

https://gerrit.wikimedia.org/r/571997

gerritbot added a comment.Feb 13 2020, 3:08 PM

Change 571998 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/spicerack@master] ganeti: add VM creation capability

https://gerrit.wikimedia.org/r/571998

gerritbot added a comment.Feb 13 2020, 3:13 PM

Change 571999 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.ganeti.makevm: refactor for new spicerack

https://gerrit.wikimedia.org/r/571999

gerritbot added a comment.Feb 25 2020, 5:42 PM

Change 571780 merged by jenkins-bot:
[operations/software/spicerack@master] ganeti: use canonical cluster names

https://gerrit.wikimedia.org/r/571780

gerritbot added a comment.Feb 25 2020, 5:42 PM

Change 571997 merged by jenkins-bot:
[operations/software/spicerack@master] ganeti: add logging for GntInstance actions

https://gerrit.wikimedia.org/r/571997

gerritbot added a comment.Feb 25 2020, 5:44 PM

Change 571998 merged by jenkins-bot:
[operations/software/spicerack@master] ganeti: add VM creation capability

https://gerrit.wikimedia.org/r/571998

gerritbot added a comment.Feb 26 2020, 2:01 PM

Change 567169 merged by jenkins-bot:
[operations/cookbooks@master] sre.hosts.decommission: improve Ganeti VM support

https://gerrit.wikimedia.org/r/567169

gerritbot added a comment.Feb 26 2020, 2:28 PM

Change 575017 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.hosts.decommission: fix Ganeti VM decom path

https://gerrit.wikimedia.org/r/575017

gerritbot added a comment.Feb 26 2020, 2:32 PM

Change 575017 merged by jenkins-bot:
[operations/cookbooks@master] sre.hosts.decommission: fix Ganeti VM decom path

https://gerrit.wikimedia.org/r/575017

gerritbot added a comment.Feb 26 2020, 2:59 PM

Change 571999 merged by jenkins-bot:
[operations/cookbooks@master] sre.ganeti.makevm: refactor for new spicerack

https://gerrit.wikimedia.org/r/571999

Maintenance_bot removed a project: Patch-For-Review.Feb 26 2020, 3:10 PM
Volans closed this task as Resolved.May 17 2021, 9:22 AM

This work has been superseded by more recent work on the Ganeti spicerack module, that now offers an easy way to get all the informations listed in the task description. See https://doc.wikimedia.org/spicerack/master/api/index.html#spicerack.Spicerack.netbox_server for more information.

Volans mentioned this in rCCKB43d739ef8498: sre.hosts.decommission: improve Ganeti VM support.Dec 14 2022, 3:25 PM
Volans mentioned this in rCCKB9508fa5a94bd: sre.hosts.decommission: fix Ganeti VM decom path.
Volans mentioned this in rCCKB1aec6ce9e627: sre.ganeti.makevm: refactor for new spicerack.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits