Page MenuHomePhabricator
Paste P45906

makevm
ActivePublic
Actions

Authored by jhathaway on Mar 21 2023, 4:38 PM.
Tags
None
Referenced Files
F36921901: makevm
Mar 21 2023, 4:38 PM
Subscribers
None
sudo cookbook sre.ganeti.makevm \
--vcpus 2 --memory 4 --disk 50 --network public \
--cluster eqiad --group C dborch1002
Ready to create Ganeti VM dborch1002.wikimedia.org in the eqiad cluster on group C with 2 vCPUs, 4GB of RAM, 50GB of disk in the public network.
==> Is this correct?
Type "go" to proceed or "abort" to interrupt the execution
> go
User input is: "go"
START - Cookbook sre.ganeti.makevm for new host dborch1002.wikimedia.org
Allocated IPv4 208.80.154.77/26
Set DNS name of IP 208.80.154.77/26 to dborch1002.wikimedia.org
Allocated IPv6 2620:0:861:3:208:80:154:77/64 with DNS name dborch1002.wikimedia.org
START - Cookbook sre.dns.netbox
Generating the DNS records from Netbox data. It will take a couple of minutes.
----- OUTPUT of 'cd /tmp && runus...2.wikimedia.org"' -----
2023-03-21 15:51:28,598 [INFO] Gathering devices, interfaces, addresses and prefixes from Netbox
2023-03-21 15:52:30,210 [ERROR] Failed to run
Traceback (most recent call last):
File "/srv/deployment/netbox-extras/dns/generate_dns_snippets.py", line 695, in main
batch_status, ret_code = run_commit(args, config, tmpdir)
File "/srv/deployment/netbox-extras/dns/generate_dns_snippets.py", line 600, in run_commit
netbox.collect()
File "/srv/deployment/netbox-extras/dns/generate_dns_snippets.py", line 169, in collect
address.assigned_object = self.physical_interfaces[address.assigned_object_id]
KeyError: 20463
================
PASS | | 0% (0/1) [01:02<?, ?hosts/s]
FAIL |██████████████████████████████████████████████████████████████████| 100% (1/1) [01:02<00:00, 62.69s/hosts]
100.0% (1/1) of nodes failed to execute command 'cd /tmp && runus...2.wikimedia.org"': netbox1002.eqiad.wmnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: 'cd /tmp && runus...2.wikimedia.org"'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
Exception raised while executing cookbook sre.dns.netbox:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run
raw_ret = runner.run()
File "/usr/lib/python3/dist-packages/spicerack/_module_api.py", line 18, in run
return self._run(self.args, self.spicerack)
File "/srv/deployment/spicerack/cookbooks/sre/dns/netbox.py", line 72, in run
results = netbox_host.run_sync(command, is_safe=True)
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 520, in run_sync
return self._execute(
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 720, in _execute
raise RemoteExecutionError(ret, "Cumin execution failed")
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
START - Cookbook sre.dns.wipe-cache dborch1002.wikimedia.org on all recursors
===== NODE GROUP =====
(14) dns[1001-1003,2001-2003,3001-3002,4003-4004,5003-5004,6001-6002].wikimedia.org
----- OUTPUT of 'sudo rec_control...02.wikimedia.org' -----
wiped 0 records, 0 negative records, 0 packets
================
PASS |████████████████████████████████████████████████████████████████| 100% (14/14) [00:03<00:00, 4.46hosts/s]
FAIL | | 0% (0/14) [00:03<?, ?hosts/s]
100.0% (14/14) success ratio (>= 100.0% threshold) for command: 'sudo rec_control...02.wikimedia.org'.
100.0% (14/14) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dborch1002.wikimedia.org on all recursors
The Ganeti's command output will be printed at the end.
Creating VM dborch1002.wikimedia.org in cluster eqiad with group=C vcpus=2 memory=4GB disk=50GB link=public. This may take a few minutes.
----- OUTPUT of 'gnt-instance add...02.wikimedia.org' -----
Tue Mar 21 15:52:35 2023 - INFO: No-installation mode selected, disabling startup
Failure: prerequisites not met for this operation:
error type: resolver_error, error details:
The given name (dborch1002.wikimedia.org) does not resolve: Name or service not known
================
PASS | | 0% (0/1) [00:01<?, ?hosts/s]
FAIL |██████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00, 1.34s/hosts]
100.0% (1/1) of nodes failed to execute command 'gnt-instance add...02.wikimedia.org': ganeti1028.eqiad.wmnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: 'gnt-instance add...02.wikimedia.org'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
Exception raised while executing cookbook sre.ganeti.makevm:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run
raw_ret = runner.run()
File "/srv/deployment/spicerack/cookbooks/sre/ganeti/makevm.py", line 186, in run
instance.add(group=self.group.name, vcpus=self.vcpus, memory=self.memory, disk=self.disk, link=self.network)
File "/usr/lib/python3/dist-packages/spicerack/ganeti.py", line 290, in add
results = self._master.run_sync(command, print_output=True)
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 520, in run_sync
return self._execute(
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 720, in _execute
raise RemoteExecutionError(ret, "Cumin execution failed")
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
Deleting assigned IP 208.80.154.77/26
Deleting assigned IP 2620:0:861:3:208:80:154:77/64
START - Cookbook sre.dns.netbox
Generating the DNS records from Netbox data. It will take a couple of minutes.
----- OUTPUT of 'cd /tmp && runus...2.wikimedia.org"' -----
2023-03-21 15:52:37,054 [INFO] Gathering devices, interfaces, addresses and prefixes from Netbox
2023-03-21 15:53:38,062 [INFO] Gathered 2735 devices from Netbox
2023-03-21 15:53:38,062 [INFO] Generating DNS records
2023-03-21 15:53:46,774 [INFO] Generated 16836 direct and reverse records (8418 each) in 30 direct zones and 249 reverse zones
2023-03-21 15:53:46,775 [INFO] Cloning /srv/netbox-exports/dns.git/ to /tmp/dns-c25pcHBldHM-0yxbibtu ...
2023-03-21 15:53:46,967 [INFO] Generating zonefile snippets to directory /tmp/dns-c25pcHBldHM-0yxbibtu
2023-03-21 15:53:47,920 [INFO] Nothing to commit!
2023-03-21 15:53:48,272 [INFO] Temporary directory /tmp/dns-c25pcHBldHM-0yxbibtu removed.
METADATA: {"no_changes": true}
================
PASS |██████████████████████████████████████████████████████████████████| 100% (1/1) [01:12<00:00, 72.00s/hosts]
FAIL | | 0% (0/1) [01:12<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'cd /tmp && runus...2.wikimedia.org"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
No changes to deploy.
END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
START - Cookbook sre.dns.wipe-cache dborch1002.wikimedia.org on all recursors
===== NODE GROUP =====
(1) dns1002.wikimedia.org
----- OUTPUT of 'sudo rec_control...02.wikimedia.org' -----
wiped 0 records, 1 negative records, 4 packets
===== NODE GROUP =====
(13) dns[1001,1003,2001-2003,3001-3002,4003-4004,5003-5004,6001-6002].wikimedia.org
----- OUTPUT of 'sudo rec_control...02.wikimedia.org' -----
wiped 0 records, 0 negative records, 0 packets
================
PASS |████████████████████████████████████████████████████████████████| 100% (14/14) [00:03<00:00, 4.12hosts/s]
FAIL | | 0% (0/14) [00:03<?, ?hosts/s]
100.0% (14/14) success ratio (>= 100.0% threshold) for command: 'sudo rec_control...02.wikimedia.org'.
100.0% (14/14) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dborch1002.wikimedia.org on all recursors
Syncing VMs in DC eqiad to Netbox
----- OUTPUT of 'systemctl start ...iad_sync.service' -----
================
PASS |██████████████████████████████████████████████████████████████████| 100% (1/1) [00:05<00:00, 5.21s/hosts]
FAIL | | 0% (0/1) [00:05<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'systemctl start ...iad_sync.service'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dborch1002.wikimedia.org
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL