The WMCS integration project holds instances running Jenkins agents. We need more disk space and via T340070 I have requested a larger flavor raising the Ephemeral disk space from 60GB to 90GB (g3.cores8.ram24.disk20.ephemeral60.4xiops and g3.cores8.ram24.disk20.ephemeral90.4xiops).
I have changed the flavor for integration-agent-docker-1039 via Horizon. It instantly reboots the instance and once rebooted the Linux kernel still sees sdb as a 60G disk:
[Thu Jun 29 17:27:03 2023] sd 2:0:0:1: [sdb] 125829120 512-byte logical blocks: (64.4 GB/60.0 GiB)
I tried to hard reboot it and that does not change anything: the Ephemeral disk is not resized.
Various links:
https://access.redhat.com/solutions/3081971#fn:1 is titled Instance cannot resize ephemeral disk in Red Hat OpenStack and marked as Solution in progress. Then it is behind a paywall so there are not much details. I am guessing OpenStack is unable to resize ephemeral disk space.
I also found an old nova bug which mentions issue resizing rbd disks: SRU: nova resize doesn't resize(extend) rbd disk files when using rbd disk backend but that got marked fixed a while ago.
Maybe there are some traces on our backend which can help diagnose the root cause?
Is that doable using openstack resize or maybe cinder <volume ID> <new GB size>? If we could manually resize the Ephemeral disks that would save us from having to rebuild and reprovision the whole fleet of instances which is rather costly ;]
Errors found by @dcaro:
On the cloudvirt that runs the VM:
Jun 29 17:28:30 cloudvirt1047 nova-compute[1277452]: 2023-06-29 17:28:30.252 1277452 WARNING nova.virt.libvirt.imagecache [None req-512c7717-f075-46da-802e-4405adfbccac - - - - - -] ephemeral_60_40d1d2c ephemeral image was used by instance but no back files existing!
And the full stacktrace from Logstash:
[None req-4fbac892-c082-478c-a411-2195daf6b45c hashar integration - - default default]
Exception during message handling:
nova.exception.ResizeError:
Resize error: Unable to resize disk down.
#0122023-06-29 17:59:55.733 1277452 ERROR oslo_messaging.rpc.server
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 10596, in _error_out_instance_on_exception
yield
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 5874, in _resize_instance
disk_info = self.driver.migrate_disk_and_power_off(
File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 11377, in migrate_disk_and_power_off
raise exception.InstanceFaultRollback(
nova.exception.InstanceFaultRollback: Instance rollback performed due to: Resize error: Unable to resize disk down.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
res = self.dispatcher.dispatch(message)
File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
result = func(ctxt, **new_args)
File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 71, in wrapped
_emit_versioned_exception_notification(
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
return f(self, context, *args, **kw)
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 184, in decorated_function
LOG.warning("Failed to revert task state for instance. "
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 155, in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 212, in decorated_function
compute_utils.add_instance_fault_from_exc(context,
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 201, in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 5841, in resize_instance
self._revert_allocation(context, instance, migration)
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 5837, in resize_instance
self._resize_instance(context, instance, image, migration,
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 5903, in _resize_instance
self.compute_rpcapi.finish_resize(context, instance,
File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 10609, in _error_out_instance_on_exception
raise error.inner_exception
nova.exception.ResizeError: Resize error: Unable to resize disk down.