Each IcingaHosts.wait_for_downtimed() call in dry_run currently takes 120 seconds which can make dry_run's pretty slow with larger number of host.
Description
Details
Event Timeline
Indeed, I can confirm the issue. The problem comes from the a bit automagic dry-run handling in the @retry decorator (code), that reduces the tries to 1 in dry-run mode but has a bit of heuristic to determine if the current run is dry-run or not.
In this case the problem comes from the fact that the IcingaHosts class doesn't have a self._dry_run property to be checked fo dry-run.
So the course of action here should be:
- modify the `__init__() signature of IcingaHosts` to add the dry_run parameter similarly to many other spicerack classes (example)
- it's possible that the dry_run property will be reported as not used and you might need to add it to the vulture_whitelist file.
- modify the icinga_hosts() method in the Spicerack class to pass the dry_run parameter when instantiating the class (example)
- check if there are other uses of @retry across the codebase in classes that don't have the dry_run parameter to add it to them too if missing
- [stretch/bonus points] look if we can have an easy was to check that in CI so that we can report usages of @retry in places that don't have dry_run awareness.
Change 828800 had a related patch set uploaded (by Slyngshede; author: Slyngshede):
[operations/software/spicerack@master] Icinga: Add dry-run option.
Change 828800 merged by Slyngshede:
[operations/software/spicerack@master] Icinga: Add dry-run option.
Missing the two last bulletpoints:
- check if there are other uses of @retry across the codebase in classes that don't have the dry_run parameter to add it to them too if missing
- [stretch/bonus points] look if we can have an easy was to check that in CI so that we can report usages of @retry in places that don't have dry_run awareness.
We're missing a "dry_run" for services and puppet, but Puppet doesn't need is as the decorator also checks for _remote_hosts.
PuppetMaster Class needs dry_run, this can be done by letting the class inherit from RemoteHostsAdapter.
Service class should have a dry_run as well, but is currently a dataclass.
Change 903235 had a related patch set uploaded (by Slyngshede; author: Slyngshede):
[operations/software/spicerack@master] Puppet: PuppetMaster class should inherit RemoteHostsAdapter
Change 903237 had a related patch set uploaded (by Slyngshede; author: Slyngshede):
[operations/software/spicerack@master] Service: Ensure that dry_run is parsed to dataclass.
Change 903235 merged by jenkins-bot:
[operations/software/spicerack@master] Puppet: PuppetMaster class should inherit RemoteHostsAdapter
Change 903237 merged by jenkins-bot:
[operations/software/spicerack@master] Service: Ensure that dry_run is passed to dataclass.