Page MenuHomePhabricator
Create Task
Maniphest T159721

labvirt1001 and 1002 cannot launch new VMs
Closed, ResolvedPublic
Actions

Description

IMPORTANT: both 1001 and 1002 are currently depooled from scheduling

This may be happening on 1002 as well, although I haven't tested there yet.

2017-03-06 15:49:20.745 16939 INFO nova.compute.claims [req-69ee5935-97bd-40c5-b490-d48496a1bf19 novaadmin admin - - -] [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] Claim successful
2017-03-06 15:49:21.536 16939 INFO nova.virt.libvirt.driver [req-69ee5935-97bd-40c5-b490-d48496a1bf19 novaadmin admin - - -] [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] Creating image
2017-03-06 15:49:23.339 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Auditing locally available compute resources for node labvirt1001.eqiad.wmnet
2017-03-06 15:49:33.100 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Total usable vcpus: 48, total allocated vcpus: 87
2017-03-06 15:49:33.101 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Final resource view: name=labvirt1001.eqiad.wmnet phys_ram=386948MB used_ram=190976MB phys_disk=2233GB used_disk=1860GB total_vcpus=48 used_vcpus=87 pci_stats=None
2017-03-06 15:49:33.184 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Compute_service record updated for labvirt1001:labvirt1001.eqiad.wmnet
2017-03-06 15:49:52.101 16939 INFO nova.virt.libvirt.firewall [req-69ee5935-97bd-40c5-b490-d48496a1bf19 novaadmin admin - - -] [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] Called setup_basic_filtering in nwfilter
2017-03-06 15:49:52.103 16939 INFO nova.virt.libvirt.firewall [req-69ee5935-97bd-40c5-b490-d48496a1bf19 novaadmin admin - - -] [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] Ensuring static filters
2017-03-06 15:50:24.117 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Auditing locally available compute resources for node labvirt1001.eqiad.wmnet
2017-03-06 15:54:37.086 16939 ERROR nova.virt.libvirt.guest [req-69ee5935-97bd-40c5-b490-d48496a1bf19 novaadmin admin - - -] Error launching a defined domain with XML: <domain type='kvm'>
  <name>i-0006512e</name>
  <uuid>14e0d779-ee11-4aa2-9cc1-d89160310800</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="12.0.4"/>
      <nova:name>labvirt1001test</nova:name>
      <nova:creationTime>2017-03-06 15:49:51</nova:creationTime>
      <nova:flavor name="m1.small">
        <nova:memory>2048</nova:memory>
        <nova:disk>20</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="novaadmin">novaadmin</nova:user>
        <nova:project uuid="admin">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="ea1ee8b4-c4cf-4d68-9ea6-cfaeb710b9b0"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>12.0.4</entry>
      <entry name='serial'>35353636-3435-4d32-3235-313330334250</entry>
      <entry name='uuid'>14e0d779-ee11-4aa2-9cc1-d89160310800</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-vivid'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/14e0d779-ee11-4aa2-9cc1-d89160310800/disk'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:3e:91:ca'/>
      <source bridge='br1102'/>
      <model type='virtio'/>
      <filterref filter='nova-instance-i-0006512e-fa163e3e91ca'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/14e0d779-ee11-4aa2-9cc1-d89160310800/console.log'/>
      <target port='0'/>
    </serial>
    <serial type='pty'>
      <target port='1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/14e0d779-ee11-4aa2-9cc1-d89160310800/console.log'/>
      <target type='serial' port='0'/>
    </console>
    <channel type='pty'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [req-69ee5935-97bd-40c5-b490-d48496a1bf19 novaadmin admin - - -] [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] Instance failed to spawn
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] Traceback (most recent call last):
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2156, in _build_resources
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     yield resources
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     block_device_info=block_device_info)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2531, in spawn
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     block_device_info=block_device_info)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4613, in _create_domain_and_network
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     xml, pause=pause, power_on=power_on)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4543, in _create_domain
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     guest.launch(pause=pause)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 142, in launch
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     self._encoded_xml, errors='ignore')
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     six.reraise(self.type_, self.value, self.tb)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 137, in launch
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     return self._domain.createWithFlags(flags)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     rv = execute(f, *args, **kwargs)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     six.reraise(c, e, tb)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     rv = meth(*args, **kwargs)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1059, in createWithFlags
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] libvirtError: error from service: ListActivatableNames: Connection is closed
2017-03-06 15:54:37.088 16939 ERROR nova.compute.manager [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] 
2017-03-06 15:54:37.099 16939 INFO nova.compute.manager [req-69ee5935-97bd-40c5-b490-d48496a1bf19 novaadmin admin - - -] [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] Terminating instance
2017-03-06 15:54:39.656 16939 INFO nova.virt.libvirt.driver [-] [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] Instance destroyed successfully.
2017-03-06 15:54:40.482 16939 INFO nova.virt.libvirt.driver [req-69ee5935-97bd-40c5-b490-d48496a1bf19 novaadmin admin - - -] [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] Deleting instance files /var/lib/nova/instances/14e0d779-ee11-4aa2-9cc1-d89160310800_del
2017-03-06 15:54:40.484 16939 INFO nova.virt.libvirt.driver [req-69ee5935-97bd-40c5-b490-d48496a1bf19 novaadmin admin - - -] [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] Deletion of /var/lib/nova/instances/14e0d779-ee11-4aa2-9cc1-d89160310800_del complete
2017-03-06 15:54:45.171 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Total usable vcpus: 48, total allocated vcpus: 87
2017-03-06 15:54:45.173 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Final resource view: name=labvirt1001.eqiad.wmnet phys_ram=386948MB used_ram=190976MB phys_disk=2233GB used_disk=1860GB total_vcpus=48 used_vcpus=87 pci_stats=None
2017-03-06 15:54:45.243 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Compute_service record updated for labvirt1001:labvirt1001.eqiad.wmnet
2017-03-06 15:54:45.438 16939 WARNING nova.compute.manager [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] While synchronizing instance power states, found 30 instances in the database and 32 instances on the hypervisor.
2017-03-06 15:54:47.877 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Auditing locally available compute resources for node labvirt1001.eqiad.wmnet
2017-03-06 15:54:55.172 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Total usable vcpus: 48, total allocated vcpus: 87
2017-03-06 15:54:55.174 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Final resource view: name=labvirt1001.eqiad.wmnet phys_ram=386948MB used_ram=190976MB phys_disk=2233GB used_disk=1860GB total_vcpus=48 used_vcpus=87 pci_stats=None
2017-03-06 15:54:55.239 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Compute_service record updated for labvirt1001:labvirt1001.eqiad.wmnet
2017-03-06 15:54:56.175 16939 INFO nova.compute.manager [-] [instance: 14e0d779-ee11-4aa2-9cc1-d89160310800] During sync_power_state the instance has a pending task (spawning). Skip.
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher [req-0894b6b0-71c1-436d-a3f4-81bd30a4e216 - - - - -] Exception during message handling: 413998
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 470, in decorated_function
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     payload)
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 72, in wrapped
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1387, in refresh_instance_security_rules
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     return _sync_refresh()
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1382, in _sync_refresh
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     return self.driver.refresh_instance_security_rules(instance)
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5074, in refresh_instance_security_rules
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     self.firewall_driver.refresh_instance_security_rules(instance)
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 434, in refresh_instance_security_rules
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     self.do_refresh_instance_rules(instance)
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 466, in do_refresh_instance_rules
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher     _instance, network_info = self.instance_info[instance.id]
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher KeyError: 413998
2017-03-06 15:54:56.183 16939 ERROR oslo_messaging.rpc.dispatcher 
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher [req-bdf82c31-8f78-4efa-a198-2409fd0beebf - - - - -] Exception during message handling: 413998
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 470, in decorated_function
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     payload)
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 72, in wrapped
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1387, in refresh_instance_security_rules
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     return _sync_refresh()
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1382, in _sync_refresh
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     return self.driver.refresh_instance_security_rules(instance)
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5074, in refresh_instance_security_rules
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     self.firewall_driver.refresh_instance_security_rules(instance)
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 434, in refresh_instance_security_rules
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     self.do_refresh_instance_rules(instance)
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/firewall.py", line 466, in do_refresh_instance_rules
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher     _instance, network_info = self.instance_info[instance.id]
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher KeyError: 413998
2017-03-06 15:54:56.195 16939 ERROR oslo_messaging.rpc.dispatcher 
2017-03-06 15:55:26.206 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Auditing locally available compute resources for node labvirt1001.eqiad.wmnet
2017-03-06 15:55:33.759 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Total usable vcpus: 48, total allocated vcpus: 87
2017-03-06 15:55:33.760 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Final resource view: name=labvirt1001.eqiad.wmnet phys_ram=386948MB used_ram=188928MB phys_disk=2233GB used_disk=1840GB total_vcpus=48 used_vcpus=87 pci_stats=None
2017-03-06 15:55:33.838 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Compute_service record updated for labvirt1001:labvirt1001.eqiad.wmnet
2017-03-06 15:56:28.175 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Auditing locally available compute resources for node labvirt1001.eqiad.wmnet
2017-03-06 15:56:35.416 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Total usable vcpus: 48, total allocated vcpus: 87
2017-03-06 15:56:35.417 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Final resource view: name=labvirt1001.eqiad.wmnet phys_ram=386948MB used_ram=188928MB phys_disk=2233GB used_disk=1840GB total_vcpus=48 used_vcpus=87 pci_stats=None
2017-03-06 15:56:35.468 16939 INFO nova.compute.resource_tracker [req-990edea0-9f17-4c12-8c80-42045964c4e7 - - - - -] Compute_service record updated for labvirt1001:labvirt1001.eqiad.wmnet

Details

SubjectRepoBranchLines +/-
Nova: Add labvirt1002 back to the scheduler pooloperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedAndrewT159721 labvirt1001 and 1002 cannot launch new VMs
 ResolvedAndrewT159835 Labvirt1001 has insanely slow IO

Event Timeline

Andrew created this task.Mar 6 2017, 3:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2017, 3:58 PM
Paladox subscribed.Mar 6 2017, 4:02 PM
Andrew created subtask T159835: Labvirt1001 has insanely slow IO.Mar 7 2017, 3:23 PM
• chasemp triaged this task as High priority.Mar 17 2017, 12:35 PM
• chasemp renamed this task from labvirt1001 can't launch new VMs to labvirt1001 and 1002 cannot launch new VMs.Mar 20 2017, 12:54 PM
• chasemp mentioned this in T159459: openstack instance creation sometimes takes >480s.
• chasemp updated the task description. (Show Details)
bd808 moved this task from Triage to OpenStack on the Cloud-Services board.Mar 26 2017, 9:02 PM
gerritbot subscribed.Mar 28 2017, 2:02 PM

Change 345145 had a related patch set uploaded (by Andrew Bogott):
[operations/puppet@production] Nova: Add labvirt1002 back to the scheduler pool

https://gerrit.wikimedia.org/r/345145

gerritbot added a project: Patch-For-Review.Mar 28 2017, 2:02 PM
gerritbot added a comment.Mar 28 2017, 2:07 PM

Change 345145 merged by Andrew Bogott:
[operations/puppet@production] Nova: Add labvirt1002 back to the scheduler pool

https://gerrit.wikimedia.org/r/345145

Andrew closed this task as Resolved.Mar 28 2017, 2:36 PM

I wasn't able to reproduce any problems with 1002, so I've put it back in the scheduling pool. 1001 has it's own bug, T159835, where progress is tracked.

Andrew closed subtask T159835: Labvirt1001 has insanely slow IO as Resolved.Apr 13 2017, 6:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits