The spicerack.service.Service.check_dns_state is decorated wit the retry method as follows:
@retry(backoff_mode="constant", exceptions=(DiscoveryStateError,), dynamic_params_callbacks=(set_tries,)) def check_dns_state(self, ip_per_dc_map: dict[str, Union[IPv4Address, IPv6Address]], record_name: str = "", tries: int = 15) -> None: """ Arguments: tries: the number of retries to attempt before failing. """
This suggests to me that the function should, in the default instance, retry three times . however when using this function in the SREDiscoveryNoLVSBatchRunnerBase cookbooks where we call the function via self.service.check_dns_state(self.ip_per_dc_map) we see that it only retries three times
Setting pooled=False for tags: {'dnsdisc': '(pki)', 'name': 'codfw'}
Sleeping for 15 seconds
[1/3, retrying in 3.00s] Attempt to run 'spicerack.service.Service.check_dns_state' raised: Error checking auth dns for pki.discovery.wmnet in codfw: resolved to 10.192.16.172, a different IP.
[2/3, retrying in 3.00s] Attempt to run 'spicerack.service.Service.check_dns_state' raised: Error checking auth dns for pki.discovery.wmnet in codfw: resolved to 10.192.16.172, a different IP.
Exception raised while executing cookbook sre.pki.restart-reboot:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 212, in run
raw_ret = runner.run()
File "/srv/deployment/spicerack/cookbooks/sre/__init__.py", line 666, in run
report = super().run()
File "/srv/deployment/spicerack/cookbooks/sre/__init__.py", line 451, in run
return self.batch_action()
File "/srv/deployment/spicerack/cookbooks/sre/__init__.py", line 424, in batch_action
self.group_action(host_group_idx, number_of_batches)
File "/srv/deployment/spicerack/cookbooks/sre/__init__.py", line 657, in group_action
self.service.check_dns_state(self.ip_per_dc_map)
File "/usr/lib/python3/dist-packages/wmflib/decorators.py", line 210, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/spicerack/service.py", line 512, in check_dns_state
self.discovery.get(record_name).check_service_ips(self.ip, ip_per_dc_map)
File "/usr/lib/python3/dist-packages/spicerack/service.py", line 115, in check_service_ips
raise DiscoveryStateError(
spicerack.service.DiscoveryStateError: Error checking auth dns for pki.discovery.wmnet in codfw: resolved to 10.192.16.172, a different IP was expected.