Page MenuHomePhabricator

sre.dns.netbox cookbook emergency improvements
Closed, ResolvedPublic

Description

As discussed on IRC we need to improve the current sre.dns.netbox cookbook to cope with emergency scenarios with the following:

  • allow to run it and merge and push the changes without deploying them to the actual gdnsd (basically skiiping the authdns-update). This will allow to stage some automated changes that require also a commit in the manual repository. The workflow would be as follows:
    • perform changes in Netbox
    • run the sre.dns.netbox cookbook with the --skip-authdns-update flag
    • merge the related manual patch needed in the operations/dns repository
    • manually run the authdns-update script that will deploy both changes at the same time
  • add an emergency edit mode to the cookbook that will generate all the records as usual but then stop to allow the user to ssh into netbox1001 in a temporary path, manually edit any file, commit --amend them and give back to the cookbook the SHA1 of the new commit to push. At this point the cookbook will continue it's usual deploy with the modified files.

Event Timeline

Volans triaged this task as High priority.Oct 7 2020, 8:37 AM
Volans created this task.

Change 632697 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: add --skip-authdns-update option

https://gerrit.wikimedia.org/r/632697

Change 632745 had a related patch set uploaded (by Volans; owner: Volans):
[operations/software/netbox-extras@master] dns: add --keep-files option

https://gerrit.wikimedia.org/r/632745

Change 632746 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: add --emergency-manual-edit option

https://gerrit.wikimedia.org/r/632746

Change 632745 merged by Volans:
[operations/software/netbox-extras@master] dns: add --keep-files option

https://gerrit.wikimedia.org/r/632745

Change 632697 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: add --skip-authdns-update option

https://gerrit.wikimedia.org/r/632697

Change 632746 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: add --emergency-manual-edit option

https://gerrit.wikimedia.org/r/632746

Change 632950 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] added prefix 91.198.174.224/27, adapt INCLUDE

https://gerrit.wikimedia.org/r/632950

Change 632950 merged by Volans:
[operations/dns@master] added prefix 91.198.174.224/27, adapt INCLUDE

https://gerrit.wikimedia.org/r/632950

Change 632971 had a related patch set uploaded (by Volans; owner: Volans):
[operations/cookbooks@master] sre.dns.netbox: improve user message

https://gerrit.wikimedia.org/r/632971

Change 632971 merged by jenkins-bot:
[operations/cookbooks@master] sre.dns.netbox: improve user message

https://gerrit.wikimedia.org/r/632971