Page MenuHomePhabricator

deployment-upload won't start, upload.beta.wmflabs.org down
Closed, ResolvedPublic

Event Timeline

greg triaged this task as Unbreak Now! priority.Apr 1 2016, 4:40 PM
greg added a subscriber: greg.

@Krenair: did you see anything note worthy when looking at this?

When you try to start the instance, Horizon shows a success message in the UI. But when you go to the action log, you find the action listed with message 'Error'

nova-compute says:

2016-04-01 17:01:24.328 56558 ERROR oslo_messaging.rpc.dispatcher [req-659dfa94-89f9-41cf-80a4-d81f1a42524a novaadmin deployment-prep - - -] Exception during message handling: no such option: libvirt_images_type

I don't know what that's about... step one will be to upgrade nova-compute on that node and see if that makes a difference.

Quoted Text

2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, new_args)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6936, in start_instance
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return self.manager.start_instance(ctxt, instance)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher payload)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in exit
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args,
kw)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 333, in decorated_function
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in exit
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 304, in decorated_function
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 383, in decorated_function
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args,
kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 361, in decorated_function
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in exit
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 349, in decorated_function
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2852, in start_instance
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher self._power_on(context, instance)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2825, in _power_on
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2241, in power_on
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher self._hard_reboot(context, instance, network_info, block_device_info)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2122, in _hard_reboot
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher disk_info_json)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5958, in _create_images_and_backing
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher ephemeral_size=instance.ephemeral_gb)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 231, in cache
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher *args,
kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 480, in create_image
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher prepare_template(target=base, max_size=size, *args, kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return f(*args,
kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 221, in fetch_func_sync
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher fetch_func(target=target, *args, **kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2597, in _create_ephemeral
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher self._create_local(target, ephemeral_size)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2586, in _create_local
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher if not CONF.libvirt_images_type == "lvm":
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1870, in getattr
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher raise NoSuchOptError(name)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher NoSuchOptError: no such option: libvirt_images_type
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher

Quoted Text

hmm...would be nice to recreate that instance; however, the only role applied seems to be role::beta::uploadservice which just adds some text to the MotD and opens ferm rules for port 80 :\

deployment-upload is barely puppetized. It has been hacked in a few years ago as a way to emulate the media servers we have used before migrating to Swift. So we definitely want to keep a copy of it :-/

deployment-upload is back up now -- reviving it required a truly horrific hack resulting from a different, legacy hack that we've been making to libvirt back since the Ryan Lane days. I've no idea why this particular instance was bit by the problem, but it probably has to do with the VM dating back to 2013.

So, I've opened a bug to remind me to sort out the hack that this ran up against... YOUR job is to puppetize the damn thing :)

-A

Krenair assigned this task to Andrew.

@Andrew Nice! Thank you very much to have ported the hack :-}

Change 281683 had a related patch set uploaded (by Andrew Bogott):
In our libvirt hack, rename libvirt_images_type to images_type

https://gerrit.wikimedia.org/r/281683

Change 281683 merged by Andrew Bogott:
In our libvirt hack, rename libvirt_images_type to images_type

https://gerrit.wikimedia.org/r/281683