Keep this open until the situation is resolved as a reminder!
Software and hardware plans didn't align - it's time to decom multatuli (ns2) and we don't have a replacement box, and software plans to merge authdns functionality into the recdns clusters isn't complete yet.
The new esams ganeti cluster (ganeti300[123]) won't be in use for a while yet (also because we're behind on software plans) - so we're going to loan out ganeti3003 as the current ns2 authdns server, temporarily.
Change 546171 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] ganeti3003: provision as authdns::server
Change 546172 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] authdns_servers: add ganeti3003
Change 546173 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] authdns_servers: remove multatuli
Change 546171 merged by BBlack:
[operations/puppet@production] ganeti3003: provision as authdns::server
Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts:
['ganeti3003.esams.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201910251335_bblack_10278.log.
Completed auto-reimage of hosts:
['ganeti3003.esams.wmnet']
Of which those FAILED:
['ganeti3003.esams.wmnet']
Change 546172 merged by BBlack:
[operations/puppet@production] authdns_servers: add ganeti3003
Change 546173 merged by BBlack:
[operations/puppet@production] authdns_servers: remove multatuli
Mentioned in SAL (#wikimedia-operations) [2019-12-04T14:24:11Z] <bblack> ns2 authdns: re-route from ganeti3003 to dns3001 - T236479
Change 554518 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Revert "authdns_servers: add ganeti3003"
Change 554519 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Revert "ganeti3003: provision as authdns::server"
Change 554518 merged by BBlack:
[operations/puppet@production] Revert "authdns_servers: add ganeti3003"
Change 554519 merged by BBlack:
[operations/puppet@production] Revert "ganeti3003: provision as authdns::server"
Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts:
['ganeti3003.esams.wmnet']
The log can be found in /var/log/wmf-auto-reimage/201912041505_bblack_122302.log.
Completed auto-reimage of hosts:
['ganeti3003.esams.wmnet']
Of which those FAILED:
['ganeti3003.esams.wmnet']
Our ns2 service address is now re-routed to dns3001, and ganeti3003 is reimaged back to spare::system.
Change 554533 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] ganeti3003: buster installer
Change 554533 merged by BBlack:
[operations/puppet@production] ganeti3003: buster installer
Change 554946 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] clean up leftover ganeti3003 config
Change 554946 merged by BBlack:
[operations/puppet@production] clean up leftover ganeti3003 config