Page MenuHomePhabricator
Create Task
Maniphest T131322

deployment-upload won't start, upload.beta.wmflabs.org down
Closed, ResolvedPublic
Actions

Details

SubjectRepoBranchLines +/-
In our libvirt hack, rename libvirt_images_type to images_typeoperations/puppetproduction+3 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

Krenair created this task.Mar 30 2016, 10:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 30 2016, 10:17 PM
Krenair merged a task: T131472: Thumbnails and images seem to be 503 erroring on Beta Commons.Apr 1 2016, 2:21 PM
Krenair added subscribers: Chippyy, Steinsplitter.
greg triaged this task as Unbreak Now! priority.Apr 1 2016, 4:40 PM
greg subscribed.

@Krenair: did you see anything note worthy when looking at this?

Krenair added a comment.Apr 1 2016, 4:41 PM

When you try to start the instance, Horizon shows a success message in the UI. But when you go to the action log, you find the action listed with message 'Error'

thcipriani added a subscriber: Andrew.Apr 1 2016, 4:57 PM
Andrew added a comment.EditedApr 1 2016, 5:02 PM

nova-compute says:

2016-04-01 17:01:24.328 56558 ERROR oslo_messaging.rpc.dispatcher [req-659dfa94-89f9-41cf-80a4-d81f1a42524a novaadmin deployment-prep - - -] Exception during message handling: no such option: libvirt_images_type

I don't know what that's about... step one will be to upgrade nova-compute on that node and see if that makes a difference.

Quoted Text

2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, new_args)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6936, in start_instance
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return self.manager.start_instance(ctxt, instance)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher payload)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in exit
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, kw)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 333, in decorated_function
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in exit
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 304, in decorated_function
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 383, in decorated_function
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 361, in decorated_function
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in exit
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 349, in decorated_function
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2852, in start_instance
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher self._power_on(context, instance)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2825, in _power_on
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2241, in power_on
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher self._hard_reboot(context, instance, network_info, block_device_info)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2122, in _hard_reboot
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher disk_info_json)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5958, in _create_images_and_backing
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher ephemeral_size=instance.ephemeral_gb)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 231, in cache
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher *args, kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 480, in create_image
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher prepare_template(target=base, max_size=size, *args, kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher return f(*args, kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 221, in fetch_func_sync
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher fetch_func(target=target, *args, **kwargs)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2597, in _create_ephemeral
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher self._create_local(target, ephemeral_size)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2586, in _create_local
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher if not CONF.libvirt_images_type == "lvm":
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1870, in getattr
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher raise NoSuchOptError(name)
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher NoSuchOptError: no such option: libvirt_images_type
2016-04-01 17:01:24.328 56558 TRACE oslo_messaging.rpc.dispatcher

Quoted Text

thcipriani subscribed.Apr 1 2016, 5:54 PM

hmm...would be nice to recreate that instance; however, the only role applied seems to be role::beta::uploadservice which just adds some text to the MotD and opens ferm rules for port 80 :\

hashar subscribed.Apr 1 2016, 7:07 PM

deployment-upload is barely puppetized. It has been hacked in a few years ago as a way to emulate the media servers we have used before migrating to Swift. So we definitely want to keep a copy of it :-/

Andrew added a subtask: T131548: Sort out our libvirt qcow2 hack with the upstream.Apr 1 2016, 9:10 PM

deployment-upload is back up now -- reviving it required a truly horrific hack resulting from a different, legacy hack that we've been making to libvirt back since the Ryan Lane days. I've no idea why this particular instance was bit by the problem, but it probably has to do with the VM dating back to 2013.

So, I've opened a bug to remind me to sort out the hack that this ran up against... YOUR job is to puppetize the damn thing :)

-A

Krenair closed this task as Resolved.Apr 1 2016, 9:14 PM
Krenair assigned this task to Andrew.
greg mentioned this in T84950: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking).Apr 1 2016, 9:22 PM
In T131322#2170491, @Andrew wrote:

YOUR job is to puppetize the damn thing :)

Just for the record, that's mostly blocked on (afaiui):

Filippo is pretty far along: T64835#1947921

hashar added a comment.Apr 4 2016, 10:14 AM

@Andrew Nice! Thank you very much to have ported the hack :-}

gerritbot subscribed.Apr 5 2016, 4:39 PM

Change 281683 had a related patch set uploaded (by Andrew Bogott):
In our libvirt hack, rename libvirt_images_type to images_type

https://gerrit.wikimedia.org/r/281683

gerritbot added a project: Patch-For-Review.Apr 5 2016, 4:39 PM
gerritbot added a comment.Apr 11 2016, 1:50 PM

Change 281683 merged by Andrew Bogott:
In our libvirt hack, rename libvirt_images_type to images_type

https://gerrit.wikimedia.org/r/281683

Andrew closed subtask T131548: Sort out our libvirt qcow2 hack with the upstream as Resolved.Mar 10 2017, 8:55 PM
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptMar 10 2017, 8:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits