Today @Fabfur and I were running the decommission and makevm cookbooks at the same time. The output of the decommission cookbook was sent to the makevm operator (me) and vice versa, leading to some confusion. Luckily we were able to coordinate with each other and figure it out.
It looks like there is work in progess to add locking to cookbooks , which would be an acceptable workaround.
Thanks for looking and please let us know if you need more info.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Volans | T341973 Spicerack: add distributed locking support | |||
| Resolved | Volans | T342130 Unrelated DNS diffs shown if decommission and makevm cookbooks run at the same time |
Was thinking a bit more about this...would it work to do some minimal sanity-checking on the DNS changes (such as the DNS record change matches the hostname invoked by the cookbook) , then simply print the DNS changes at the end? Maybe a flag to enable manual review if needed?
It looks like there is work in progess to add locking to cookbooks , which would be an acceptable workaround.
indeed this is the way we plan to support this use case
for clarification are you proposing this in addition to or instead of locking. and if the later how would you handle unexpected data?
ill let Ricardo give an authoritative answer to this when he is back, however i dont think this will be as simple as it sounds. dns changes are pushed via the sre.dns.netbox cookbook, all other cookbooks simply call this one. this cookbook simply takes the data from netbox and pushes it into the dns repo. It has no concept of what is suppose to be in the change, this is by design as the cookbook is used for deploying emergency changes and more complex changes such as adding new zone files which can require some manual steps. its also worth noting that most of the heavy lifting of the dns cookbook is actually done via a netbox script so the refactor would need to happen in both places
Indeed what John said.
For context the integrity of the data is currently ensured by git fast-forward only. If two people runs it at the same time the first one that merges wins and the other will get an explicit failure because the underlying git repository would have been modified before the second merge. Re-running it will show the remaining diff (if any) left out to be merged.
About the locking, it will surely help to avoid running more than one sync at a time, but will not prevent spurious diffs being presented to the user as they depend on the data on Netbox. So if someone manually changes something in Netbox that needs to get reflected in the DNS the next person running the cookbook will get the diff.
With exclusive locking now in place for the sre.dns.netbox cookbook I think we can consider this resolved.