Page MenuHomePhabricator

IcingaHosts.wait_for_downtimed() does not honor dry_run
Closed, ResolvedPublic

Description

Each IcingaHosts.wait_for_downtimed() call in dry_run currently takes 120 seconds which can make dry_run's pretty slow with larger number of host.

Event Timeline

JMeybohm triaged this task as Medium priority.Aug 18 2022, 9:34 AM
JMeybohm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Volans subscribed.

Indeed, I can confirm the issue. The problem comes from the a bit automagic dry-run handling in the @retry decorator (code), that reduces the tries to 1 in dry-run mode but has a bit of heuristic to determine if the current run is dry-run or not.
In this case the problem comes from the fact that the IcingaHosts class doesn't have a self._dry_run property to be checked fo dry-run.

So the course of action here should be:

  • modify the `__init__() signature of IcingaHosts` to add the dry_run parameter similarly to many other spicerack classes (example)
  • it's possible that the dry_run property will be reported as not used and you might need to add it to the vulture_whitelist file.
  • modify the icinga_hosts() method in the Spicerack class to pass the dry_run parameter when instantiating the class (example)
  • check if there are other uses of @retry across the codebase in classes that don't have the dry_run parameter to add it to them too if missing
  • [stretch/bonus points] look if we can have an easy was to check that in CI so that we can report usages of @retry in places that don't have dry_run awareness.

Change 828800 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/software/spicerack@master] Icinga: Add dry-run option.

https://gerrit.wikimedia.org/r/828800

Change 828800 merged by Slyngshede:

[operations/software/spicerack@master] Icinga: Add dry-run option.

https://gerrit.wikimedia.org/r/828800

SLyngshede-WMF changed the task status from Open to In Progress.Sep 1 2022, 11:08 AM

Missing the two last bulletpoints:

  • check if there are other uses of @retry across the codebase in classes that don't have the dry_run parameter to add it to them too if missing
  • [stretch/bonus points] look if we can have an easy was to check that in CI so that we can report usages of @retry in places that don't have dry_run awareness.

We're missing a "dry_run" for services and puppet, but Puppet doesn't need is as the decorator also checks for _remote_hosts.

PuppetMaster Class needs dry_run, this can be done by letting the class inherit from RemoteHostsAdapter.
Service class should have a dry_run as well, but is currently a dataclass.

Change 903235 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/software/spicerack@master] Puppet: PuppetMaster class should inherit RemoteHostsAdapter

https://gerrit.wikimedia.org/r/903235

Change 903237 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/software/spicerack@master] Service: Ensure that dry_run is parsed to dataclass.

https://gerrit.wikimedia.org/r/903237

Change 903235 merged by jenkins-bot:

[operations/software/spicerack@master] Puppet: PuppetMaster class should inherit RemoteHostsAdapter

https://gerrit.wikimedia.org/r/903235

Change 903237 merged by jenkins-bot:

[operations/software/spicerack@master] Service: Ensure that dry_run is passed to dataclass.

https://gerrit.wikimedia.org/r/903237