Follow up from T314070: Investigate navtiming.py outage 2022-07-28

In T314070#8112710, @Krinkle wrote:

Summary:

Two weeks ago, a new Etcd host was created and added to the advertised candidates (a DNS-SRV record).
One week ago, the old host was removed from DNS-SRV advertising.
Today, the old host was shutdown, after a week of no longer being advertised.

The python client library that we use in navtiming.py, appears to cache these DNS records indefinitely in the current process, no matter whether you re-create or re-connect, it will only pick from the candidates it saw the first time, so only old ones.

We have a high-level loop with try-catch in navtiming.py that will restart "everything" if we encounter a fatal like this, but "everything" while covering the entirety of our Python source code, does not cover the (hidden) DNS cache from the Python library - ergo an infinite loop of crashes.

Action items:

[…]

• Change navtiming.py such that the process will exit completely if the connection fails. This way, if the above scenario happens again, the process is automatically restarted by systemd and thus recovers from the problem. Whereas today, it keeps re-trying the same (bad) configuration infinitely.
• Confirm that these kinds of exists indeed result in automatic restart by systemd.

Bonus:

• Learn whether there is a limit to systemd restarts, does it ever give up?
• If it does, how will we know that the service is down? Is there an alert for this? Does the service owner get notified?