This task is to track work related to validating the architecture described here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/notes/NAT_loophole/NFS#idea_3:_manila_with_generic_driver_using_DHSS=true
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T272395 Cloud: reduce NAT exceptions from cloud to production | |||
| Open | Andrew | T291405 [NFS] Reduce or eliminate bare-metal NFS servers | |||
| Resolved | aborrero | T291257 Cloud: NFS: PoC: manila with generic driver using DHSS=true |
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older Changes
Comment Actions
Change 721805 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: bootstrap manila component
Comment Actions
Change 721805 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: bootstrap manila component
Comment Actions
Change 722355 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: don't show config file diff
Comment Actions
Change 722355 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: don't show config file diff
Comment Actions
Change 722357 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: correct variable expansion in template
Comment Actions
Change 722357 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: correct variable expansion in template
Comment Actions
Change 722567 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: introduce rabbit configuration
Comment Actions
Change 722568 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: fix datatype for db_host config value
Comment Actions
Change 722569 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[labs/private@master] hieradata: openstack: manila: introduce placeholder for rabbit password
Comment Actions
Change 722567 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: introduce rabbit configuration
Comment Actions
Change 722568 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: fix datatype for db_host config value
Comment Actions
Change 722569 merged by Arturo Borrero Gonzalez:
[labs/private@master] hieradata: openstack: manila: introduce placeholder for rabbit password
Comment Actions
Change 722581 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] hieradata: openstack: codfw1dev: add missing manila key
Comment Actions
Change 722581 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] hieradata: openstack: codfw1dev: add missing manila key
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-09-21T10:49:00Z] <arturo> [codfw1dev] create manila database on cloudcontrol-dev nodes (galera) T291257
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-09-21T10:57:05Z] <arturo> [codfw1dev] created manila user @ labtestwikitech (T291257)
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-09-21T11:06:10Z] <arturo> [codfw1dev] created manila project (T291257)
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-09-21T11:06:28Z] <arturo> [codfw1dev] give manila user admin role @ manila project (T291257)
Comment Actions
Change 722588 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[labs/private@master] hieradata: openstack: manila: add user password placeholder
Comment Actions
Change 722590 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: fix typo in hiera key for rabbit pass
Comment Actions
Change 722591 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: introduce support for manila user
Comment Actions
Change 722590 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: fix typo in hiera key for rabbit pass
Comment Actions
Change 722588 merged by Arturo Borrero Gonzalez:
[labs/private@master] hieradata: openstack: manila: add user password placeholder
Comment Actions
Change 722591 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: introduce support for manila user
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-09-21T11:32:01Z] <arturo> [codfw1dev] populated manila DB & created service endpoints (T291257)
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-09-21T11:45:04Z] <arturo> [codfw1dev] created rabbitmq user (T291257)
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-09-21T12:13:11Z] <arturo> [codfw1dev] trying to create a manila service image (T291257)
Comment Actions
Change 722602 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: configure additional bits for the service image
Comment Actions
Change 722603 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[labs/private@master] hieradata: openstack: manila: add service instance password
Comment Actions
Change 722602 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: configure additional bits for the service image
Comment Actions
Change 722603 merged by Arturo Borrero Gonzalez:
[labs/private@master] hieradata: openstack: manila: add service instance password
Comment Actions
Change 722607 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: codfw1dev: manila: enable services
Comment Actions
In the first iteration I'll be trying to use base images from https://opendev.org/openstack/manila-image-elements for the service instance as recommended in the upstream docs here https://docs.openstack.org/manila/victoria/install/post-install.html#post-install
This image is configured to use user/pass auth by default, but we should revisit that.
Comment Actions
Change 722607 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: codfw1dev: manila: enable services
Comment Actions
Change 722639 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: install python3-manilaclient
Comment Actions
Change 722639 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: install python3-manilaclient
Comment Actions
Change 722645 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: install manila-data package
Comment Actions
Note:
manila-share is refusing to start con cloudcontrol2001-dev, with stack trace:
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service Traceback (most recent call last): 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 807, in run_service 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service service.start() 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/manila/service.py", line 129, in start 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service self.manager.init_host() 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 193, in wrapped 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service return f(self, *args, **kwargs) 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 330, in init_host 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service (self.driver.service_instance_manager.network_helper. 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service return f(*args, **kwargs) 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 247, in network_helper 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service self._network_helper.setup_connectivity_with_service_instances() 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 970, in setup_connectivity_with_service_instances 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service self.service_network_id) 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service return f(*args, **kwargs) 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 804, in service_network_id 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service self._service_network_id = self._get_service_network_id() 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service return f(*args, **kwargs) 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 825, in _get_service_network_id 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service self.admin_project_id, service_network_name)['id'] 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/manila/network/neutron/api.py", line 252, in network_create 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service message=e.message) 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service manila.exception.NetworkException: Unable to create the network. No tenant network is available for allocation. 2021-09-21 16:11:05.544 10696 ERROR oslo_service.service Neutron server returns request_ids: ['req-f8729973-d668-49cc-be36-e6b05865e245']
Reading the reference source code led me to:
/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py
def _get_service_network_id(self): """Finds existing or creates new service network.""" service_network_name = self.get_config_option("service_network_name") networks = [] for network in self.neutron_api.get_all_admin_project_networks(): if network['name'] == service_network_name: networks.append(network) if len(networks) > 1: raise exception.ServiceInstanceException( _('Ambiguous service networks.')) elif not networks: return self.neutron_api.network_create( self.admin_project_id, service_network_name)['id'] else: return networks[0]['id']
it doesn't find a network, because:
/usr/lib/python3/dist-packages/manila/network/neutron/api.py
def get_all_admin_project_networks(self): search_opts = {'tenant_id': self.admin_project_id, 'shared': False} nets = self.client.list_networks(**search_opts).get('networks', []) return nets
It turns out that we created our network with shared=True:
root@cloudcontrol2001-dev:~# openstack network show 05a5494a-184f-4d5c-9e98-77ae61c56daa +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2018-03-16T21:38:53Z | | description | | | dns_domain | None | | id | 05a5494a-184f-4d5c-9e98-77ae61c56daa | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 1500 | | name | lan-flat-cloudinstances2b | | port_security_enabled | True | | project_id | admin | | provider:network_type | flat | | provider:physical_network | cloudinstances2b | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 3 | | router:external | Internal | | segments | None | | shared | True | | status | ACTIVE | | subnets | 7adfcebe-b3d0-4315-92fe-e8365cc80668 | | tags | | | updated_at | 2020-01-02T14:27:46Z | +---------------------------+--------------------------------------+
Apparently manila requires the base flat network to be shared=false? I need to investigate this a bit more. Why do we have our network created with shared=true?
If this approach doesn't work, I think I can try with other network plugins, or even with DHSS=false so we would need to create NFS VMs by hand and manila would only manage the shares/exports.
Comment Actions
Change 722645 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: install manila-data package
Comment Actions
You must always specify a share network when creating a share with a share type that requests hard multi-tenancy, i.e., has extra-spec ‘driver_handles_share_servers=True’.
Comment Actions
I filled upstream bug requesting the documentation be extended: https://bugs.launchpad.net/manila/+bug/1944696
Comment Actions
From reading the manila source code, I draw the following conclusions:
Using the generic driver there are 2 options:
- use DHSS=true --> manila creates a nova VM and then injects shares configurations on it
It apparently will read the following specific config options in manila.conf:
service_image_name = xxxx service_instance_name_template = xxxx manila_service_keypair_name = xxxx path_to_public_key = xxxx service_instance_security_group = xxxx service_instance_flavor_id = xxxx service_network_name = xxxx service_network_cidr = xxxx service_network_division_mask = xxxx interface_driver = xxxx connect_share_server_to_tenant_network = xxxx admin_network_id = xxxx admin_subnet_id = xxxx
NOTE: within this mode, there are apparently 2 options, using tenant-networks or an admin-managed network (which is what would be of our interest). This logic however, can only be seen on the source code and is not explained in the docs (see https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/service_instance.py)
To use the admin-controlled neutron network, the config should have (see https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/service_instance.py#L779):
connect_share_server_to_tenant_network = true admin_network_id = xxxx admin_subnet_id = xxxx interface_driver = manila.network.linux.interface.NoopInterfaceDriver
- use DHSS=false --> the service instance should be created by hand, and then manila will inject share configuration on it. It will still request IP addresses per share.
It apparently will read the following config options in manila.conf:
service_instance_name_or_id = xxxx service_net_name_or_ip = xxxx tenant_net_name_or_ip = xxxx
- Both DHSS=true and DHSS=false have in common the following config options for manila.conf (see https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/service_instance.py):
service_instance_user = xxxx service_instance_password = xxxx path_to_private_key = xxxx max_time_to_build_instance = xxxx limit_ssh_access = xxxx
Other global options for this driver can be seen at https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/generic.py
Comment Actions
I was able to start the manila-share agent with this configuration file:
[DEFAULT] debug = true osapi_share_base_URL = openstack.codfw1dev.wikimediacloud.org auth_strategy = keystone state_path = /var/lib/manila my_ip = 208.80.153.59 osapi_share_listen = 208.80.153.59 osapi_share_listen_port = 18786 osapi_share_workers = 1 osapi_share_use_ssl = false transport_url = rabbit://manila:REDACTED@cloudcontrol2001-dev.wikimedia.org:5672,manila:REDACTED@cloudcontrol2003-dev.wikimedia.org:5672,manila:REDACTED@cloudcontrol2004-dev.wikimedia.org:5672 network_plugin_ipv4_enabled = true network_plugin_ipv6_enabled = false neutron_net_id =05a5494a-184f-4d5c-9e98-77ae61c56daa neutron_subnet_id = 7adfcebe-b3d0-4315-92fe-e8365cc80668 enabled_share_backends = wmcscinderbackend [wmcscinderbackend] share_backend_name = wmcscinderbackend share_driver = manila.share.drivers.generic.GenericShareDriver driver_handles_share_servers = True enabled_share_protocols = NFS interface_driver = manila.network.linux.interface.NoopInterfaceDriver share_name_template = share-%s share_snapshot_name_template = share-snapshot-%s quota_shares = 50 quota_snapshots = 50 quota_gigabytes = 1000 quota_snapshot_gigabytes = 1000 quota_share_networks = 0 quota_share_replicas = 0 quota_replica_gigabytes = 0 max_gigabytes = 10000 share_mount_template = mount -vt %(proto)s %(options)s %(export)s %(path)s share_unmount_template = umount -v %(path)s volume_name_template = manila-share-%s volume_snapshot_name_template = manila-snapshot-%s share_mount_path = /shares share_helpers = NFS=manila.share.drivers.helpers.NFSHelper share_volume_fstype = ext4 volume_api_class = manila.volume.cinder.API cinder_volume_type = 31fc5bfc-bc32-40e8-8865-90cfdf67a977 service_instance_user = manila service_instance_password = REDACTED service_net_name_or_ip = lan-flat-cloudinstances2b service_image_name = 670e1578-5849-40c0-ad16-a0accb42363f service_instance_flavor_id = cb15aca6-3ff5-4623-8d21-0d37d3092835 connect_share_server_to_tenant_network = True admin_network_id = 05a5494a-184f-4d5c-9e98-77ae61c56daa admin_subnet_id = 7adfcebe-b3d0-4315-92fe-e8365cc80668 automatic_share_server_cleanup = false [cinder] region_name = codfw1dev-r auth_url = http://openstack.codfw1dev.wikimediacloud.org:8776/v3/ auth_type = v3password password = REDACTED project_domain_name = default project_name = admin auth_type = v3password default_domain_name = admin domain_name = default password = REDACTED project_domain_name = default project_name = admin user_domain_id = default user_domain_name = default username = novaadmin [database] connection = mysql+pymysql://manila:REDACTED@openstack.codfw1dev.wikimediacloud.org/manila [glance] api_microversion = 2 region_name = codfw1dev-r auth_url = http://openstack.codfw1dev.wikimediacloud.org:9292 auth_type = v3password default_domain_name = admin domain_name = default password = REDACTED project_domain_name = default project_name = admin user_domain_id = default user_domain_name = default username = novaadmin [keystone_authtoken] auth_host = openstack.codfw1dev.wikimediacloud.org auth_protocol = http www_authenticate_uri = http://openstack.codfw1dev.wikimediacloud.org:5000 auth_url = http://openstack.codfw1dev.wikimediacloud.org:35357 auth_type = password project_domain_name = default user_domain_name = default project_name = manila username = manila password = REDACTED memcached_servers = cloudcontrol2001-dev.wikimedia.org:11211,cloudcontrol2003-dev.wikimedia.org:11211,cloudcontrol2004-dev.wikimedia.org:11211 region_name = codfw1dev-r [neutron] service_metadata_proxy = true metadata_proxy_shared_secret = REDACTED auth_url = http://openstack.codfw1dev.wikimediacloud.org:5000/v3 auth_type = v3password password = REDACTED project_domain_name = default project_name = admin tenant_name = admin user_domain_id = default user_domain_name = default username = novaadmin region_name = codfw1dev-r [nova] region_name = codfw1dev-r auth_url = http://openstack.codfw1dev.wikimediacloud.org:8774/v2.1 auth_type = v3password password = REDACTED project_domain_name = default project_name = admin tenant_name = admin user_domain_id = default user_domain_name = default username = novaadmin [oslo_concurrency] lock_path = /var/lock/manila [oslo_messaging_amqp] [oslo_messaging_kafka] [oslo_messaging_notifications] [oslo_messaging_rabbit] rabbit_ha_queues = true [oslo_middleware] [oslo_policy] policy_file = policy.json [ssl]
I want to believe this works in this way (to be confirmed yet):
- DHSS=true, meaning manila creates nova VMs to host NFS shares
- such VMs will be attached to the admin-managed neutron network (i.e, our lan flat network, no tenant networks)
Comment Actions
When trying the config described above, I hit another problem. I think this is simply manila trying to create the service VM and failing because keystone auth.
/var/log/manila/manila-share.log
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 193, in wrapped 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/utils.py", line 568, in wrapper 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 1878, in create_share_instance 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server detail=message_field.Detail.NO_SHARE_SERVER) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server raise value 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 1860, in create_share_instance 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server share_group=share_group_ref, 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 655, in _provide_share_server_for_share 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return _wrapped_provide_share_server_for_share() 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 651, in _wrapped_provide_share_server_for_share 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server context, compatible_share_server, metadata)) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 751, in _create_share_server_in_backend 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server share_server = self._setup_server(context, share_server, metadata) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 3926, in _setup_server 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server self.driver.deallocate_network(context, share_server['id']) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server raise value 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 3875, in _setup_server 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server network_info, metadata=metadata) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/driver.py", line 945, in setup_server 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return self._setup_server(*args, **kwargs) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/generic.py", line 897, in _setup_server 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server self.admin_context, network_info) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 455, in set_up_service_instance 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server context, instance_name, network_info) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 549, in _create_service_instance 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server service_image_id = self._get_service_image(context) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 534, in _get_service_image 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server images = [image.id for image in self.image_api.image_list(context) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 534, in <listcomp> 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server images = [image.id for image in self.image_api.image_list(context) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 583, in __next__ 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return self._next() 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 572, in _next 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server obj, resp = next(self._self_wrapped) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 183, in list 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server for image, resp in paginate(url, page_size, limit): 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 110, in paginate 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server resp, body = self.http_client.get(next_url, headers=req_id_hdr) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 366, in request 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server **kwargs) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 780, in request 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1191, in get_auth_headers 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server token = self.get_token(session) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session) 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 144, in get_auth_ref 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server message='Authentication cannot be scoped to multiple' 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.auth.AuthorizationFailure: Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped 2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server
I think the problem is simply misconfigured credentials somewhere.
Steps I made:
root@cloudcontrol2001-dev:~# manila type-list +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+ | ID | Name | visibility | is_default | required_extra_specs | optional_extra_specs | Description | +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+ | febfcfe1-9d42-4ac1-aa85-90f63aac7c2b | default_share_type | public | - | driver_handles_share_servers : True | | None | +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+ root@cloudcontrol2001-dev:~# manila share-network-show manila-share-network +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | id | 929a7860-7b73-4544-a20e-45942c64dffe | | name | manila-share-network | | project_id | admin | | created_at | 2021-09-21T15:28:49.000000 | | updated_at | None | | description | None | | share_network_subnets | [{'id': 'cc5ba5f5-bacb-4150-a176-884bcdb2ce91', 'availability_zone': None, 'created_at': '2021-09-21T15:28:49.000000', 'updated_at': None, 'segmentation_id': None, 'neutron_net_id': '05a5494a-184f-4d5c-9e98-77ae61c56daa', 'neutron_subnet_id': '7adfcebe-b3d0-4315-92fe-e8365cc80668', 'ip_version': None, 'cidr': None, 'network_type': None, 'mtu': None, 'gateway': None}] | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ root@cloudcontrol2001-dev:~# openstack network show 05a5494a-184f-4d5c-9e98-77ae61c56daa +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2018-03-16T21:38:53Z | | description | | | dns_domain | None | | id | 05a5494a-184f-4d5c-9e98-77ae61c56daa | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 1500 | | name | lan-flat-cloudinstances2b | | port_security_enabled | True | | project_id | admin | | provider:network_type | flat | | provider:physical_network | cloudinstances2b | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 3 | | router:external | Internal | | segments | None | | shared | True | | status | ACTIVE | | subnets | 7adfcebe-b3d0-4315-92fe-e8365cc80668 | | tags | | | updated_at | 2020-01-02T14:27:46Z | +---------------------------+--------------------------------------+ root@cloudcontrol2001-dev:~# manila create NFS 1 --name myshare2 --description "My Manila share" --share-type default_share_type --share-network manila-share-network +---------------------------------------+--------------------------------------+ | Property | Value | +---------------------------------------+--------------------------------------+ | id | 917a34aa-e018-4435-96c1-f9df77a09128 | | size | 1 | | availability_zone | None | | created_at | 2021-09-23T11:42:21.000000 | | status | creating | | name | myshare2 | | description | My Manila share | | project_id | admin | | snapshot_id | None | | share_network_id | 929a7860-7b73-4544-a20e-45942c64dffe | | share_proto | NFS | | metadata | {} | | share_type | febfcfe1-9d42-4ac1-aa85-90f63aac7c2b | | is_public | False | | snapshot_support | False | | task_state | None | | share_type_name | default_share_type | | access_rules_status | active | | replication_type | None | | has_replicas | False | | user_id | novaadmin | | create_share_from_snapshot_support | False | | revert_to_snapshot_support | False | | share_group_id | None | | source_share_group_snapshot_member_id | None | | mount_snapshot_support | False | | progress | None | | share_server_id | None | | host | | +---------------------------------------+--------------------------------------+ root@cloudcontrol2001-dev:~# manila share-server-list +--------------------------------------+----------------------------------------+--------+----------------------+------------+----------------------------+--------------------------------------+ | Id | Host | Status | Share Network | Project Id | Updated_at | Share Network Subnet Id | +--------------------------------------+----------------------------------------+--------+----------------------+------------+----------------------------+--------------------------------------+ | baa04ea6-f837-4eb7-9936-1eec3a11a886 | cloudcontrol2001-dev@wmcscinderbackend | error | manila-share-network | admin | 2021-09-23T11:29:52.000000 | cc5ba5f5-bacb-4150-a176-884bcdb2ce91 | +--------------------------------------+----------------------------------------+--------+----------------------+------------+----------------------------+--------------------------------------+ root@cloudcontrol2001-dev:~# manila list +--------------------------------------+----------+------+-------------+--------+-----------+--------------------+----------------------------------------------------------+-------------------+ | ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone | +--------------------------------------+----------+------+-------------+--------+-----------+--------------------+----------------------------------------------------------+-------------------+ | 917a34aa-e018-4435-96c1-f9df77a09128 | myshare2 | 1 | NFS | error | False | default_share_type | cloudcontrol2001-dev@wmcscinderbackend#wmcscinderbackend | nova | +--------------------------------------+----------+------+-------------+--------+-----------+--------------------+----------------------------------------------------------+-------------------+
Comment Actions
I think you found this already, but this issue is somewhat discussed on https://bugs.launchpad.net/openstack-ansible/+bug/1506285 -- the issue is with specifying domain_name in the [glance] section.
With that commented out, we get to what I think is the next problem: keystone returning a completely bogus response.
2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token 2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token 2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access 2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session) 2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 197, in get_auth_ref 2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server raise exceptions.InvalidResponse(response=resp) 2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.response.InvalidResponse: Invalid response from server.
No idea what that's about. The exception is a side-effect of a failure in _provide_share_server_for_share
Comment Actions
Nice, I though I was breaking something xd, will debug with keystone misbehaving in mind
Comment Actions
It seems that we might have to pass a version explicitly or something:
2021-09-24 08:06:42.926 19922 ERROR keystoneauth.identity.v3.base [req-a4026bc8-992b-4639-91c8-509362c4c1c6 novaadmin admin - - -] Got response: <Response [300]>
resp_data:{'versions': [{'id': 'v2.11', 'status': 'CURRENT', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.10', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.9', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.8', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.7', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.6', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.5', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.4', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.3', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.2', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.1', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.0', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}]}Comment Actions
Added the /v2 ending to the auth endpoint for glance, and now the error is different ("keystoneauth1.exceptions.http.Unauthorized: Unrecognized schema in response body. (HTTP 401)"), looking
Comment Actions
okok, seems to be a regular unauthorized issue :/, maybe thereś something wrong in the versions of the libs though, as it complains about unrecognized schema...
2021-09-24 08:19:32.148 15803 ERROR keystoneauth.session [req-9719bfda-4b86-4126-a6fa-f7b9bc4eac3b novaadmin admin - - -] Got error <Response [401]>: {'Content-Type': 'application/json', 'WWW-Authenticate': 'Keystone uri="http://openstack.codfw1dev.wikimediacloud.org:5000"', 'Connection': 'close'}
{'message': 'This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.<br /><br />\n\n\n', 'code': '401 Unauthorized', 'title': 'Unauthorized'}Comment Actions
okok, so finally used the v3 endopint instead of the v2 for auth (saw it being used somewhere else):
auth_url = http://openstack.codfw1dev.wikimediacloud.org:5000/v3
And now the error is about not finding an image, nice:
2021-09-24 08:24:01.363 24366 ERROR oslo_messaging.rpc.server [req-f5f7d02b-e2ca-44cd-992e-64fc5c18f2c8 novaadmin admin - - -] Exception during message handling: manila.exception.ServiceInstanceException: Image with name '670e1578-5849-40c0-ad16-a0accb42363f' was not found or is not in 'active' state.
looking
Comment Actions
it's there, and active:
root@cloudcontrol2001-dev:~# openstack image list | grep 670e1578-5849-40c0-ad16-a0accb42363f | 670e1578-5849-40c0-ad16-a0accb42363f | manila-service-image-1.3.0-75-g631a854 | active |
Comment Actions
And the next step, auth with the novaclient \o/ progress!
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/novaclient/base.py", line 363, in _create 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 401, in post 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/novaclient/client.py", line 72, in request 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server **kwargs) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 554, in request 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 780, in request 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1191, in get_auth_headers 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server token = self.get_token(session) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 188, in get_auth_ref 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server authenticated=False, log=False, **rkwargs) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1139, in post 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 976, in request 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, method, url) 2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-bfca3b24-c100-4c6b-a63e-03f29833ed3c)
Comment Actions
Okok, I setup the novaclient with the v3 auth api endpoint and it worked.
Now the instance was started, and I can see the console through virsh:
root@cloudvirt2003-dev:~# virsh virsh # console 121 Connected to domain 'i-00000c58' Escape character is ^] (Ctrl + ]) ubuntu login:
but it does not reply to ssh from the cloudcontrol:
2021-09-24 08:59:23.368 17050 DEBUG manila.share.drivers.service_instance [req-1e04705b-5164-467a-aa63-0946c0a9d4ea novaadmin admin - - -] [Errno 110] ETIMEDOUT _test_server_connection /usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py:673 2021-09-24 08:59:23.369 17050 DEBUG manila.share.drivers.service_instance [req-1e04705b-5164-467a-aa63-0946c0a9d4ea novaadmin admin - - -] Server 172.16.128.89 is not available via SSH. Waiting... _test_server_connection /usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py:675 2021-09-24 08:59:28.371 17050 DEBUG manila.share.drivers.service_instance [req-1e04705b-5164-467a-aa63-0946c0a9d4ea novaadmin admin - - -] Checking server availability. _check_server_availability /usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py:659
looking
Comment Actions
Change 723470 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: refresh configuration file
Comment Actions
Change 723470 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: refresh configuration file
Comment Actions
Change 723492 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: separate manila-share service into a different role
Comment Actions
Submitted another bug report for manila, in this case a feature request: https://bugs.launchpad.net/manila/+bug/1944980
Comment Actions
Change 723492 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: separate manila-share service into a different role
Comment Actions
Change 723521 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila_sharecontroller: require Debian Bullseye
Comment Actions
Change 723521 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila_sharecontroller: require Debian Bullseye
Comment Actions
I stop at this point: figuring out hiera for rabbit @ manila-share-controller-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud to see if the manila-share service can work from there.
Comment Actions
The manila-share package (and manila-common) fails to install on the VM because there is a manila user defined on LDAP.
Comment Actions
hey @Andrew could you help me rename the unix user manila in codfw1dev to something else? perhaps manila-srv. Or even completely delete the user, given I don't think we need it.
The LDAP dir for codfw1dev seems to be on cloudservices2002-dev.wikimedia.org
NOTE: it was registered by me using labtestwikitech.wikimedia.org
Comment Actions
This has been done using this documentation: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Testing_deployment#LDAP Now manila-share and manila-common install well on the VM.
Next step: figure out rabbitmq connectivity.
Comment Actions
Change 724350 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: refresh config file
Comment Actions
Change 724350 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: refresh config file
Comment Actions
Change 724356 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: configuration template cleanups
Comment Actions
Change 724391 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: galera: allow optional access to the database from manila share
Comment Actions
Change 724391 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: galera: allow optional access to the database from manila share
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-09-28T11:30:02Z] <arturo> [codfw1dev] create floating IP 185.15.57.5 for manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org (T291257)
Comment Actions
Change 724445 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: keystone: allow manila-share auth as novaadmin
Comment Actions
I stop at this point:
- manila-share running on manila-share-controller-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud needs keystone & rabbit & database credentials for the service to work.
- I initially attempted to replicate credentials from cloudcontrols, but was grumpy about it (also see https://gerrit.wikimedia.org/r/724445)
- @Andrew volunteer to try to work out some additional privilege separation
- current secrets are at labs/private.git: hieradata/cloudinfra-codfw1dev.yaml on cloudinfra-internal-puppetmaster-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud. All the secrets ends in /etc/manila/manila.conf in the share controller VM.
Comment Actions
Change 724500 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] manila: use manila-srv service user rather than novaadmin for auth
Comment Actions
Change 724500 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] manila: use manila-srv service user rather than novaadmin for auth
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-09-29T09:41:00Z] <arturo> [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured (T291257)
Comment Actions
Change 724725 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: use manilainfra project to host service instances
Comment Actions
Change 724725 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: use manilainfra project to host service instances
Comment Actions
Change 724726 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: manila: refresh services on config file change
Comment Actions
Change 724726 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: manila: refresh services on config file change
Comment Actions
Latest updates:
- changed the internal tenant from service to be manilainfra, which is a bit more in line with others we have been using lately.
- manila-share is able to create the service VM. It apparently ignores the service_instance_name_template = manila-service-instance-%s setting, because the VMs have uuids as names.
- the service VM ends up with 2 neutron IPs. I can manually check that SSH works in one of them, but manila-share tries the wrong one, example:
2021-09-29 12:12:00.744 135757 ERROR oslo_messaging.rpc.server manila.exception.ServiceInstanceException: SSH connection has not been established to 172.16.128.103 in 300s. Giving up. 2021-09-29 12:12:00.744 135757 ERROR oslo_messaging.rpc.server
root@cloudcontrol2001-dev:~# openstack server show 0fa41323-0c0f-4adf-8b6d-aede186b8412 | grep addresses | addresses | lan-flat-cloudinstances2b=172.16.128.115, 172.16.128.103 | root@manila-share-controller-01:~# telnet 172.16.128.103 22 Trying 172.16.128.103... ^C root@manila-share-controller-01:~# telnet 172.16.128.115 22 Trying 172.16.128.115... Connected to 172.16.128.115. Escape character is '^]'. SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.3
Comment Actions
Reported by upstream about the service instance name: https://bugs.launchpad.net/manila/+bug/1945463
Comment Actions
I believe what's happening with the 2 ports things is that the service VM recvs SSH packets on one interface but replies on other. The neutron port firewall may be filtering this (the return packet). Need to investigate a bit more.
Comment Actions
Change 726564 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: drop manila code
Comment Actions
Change 726564 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: drop manila code
Comment Actions
Change 726569 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[labs/private@master] hieradata: drop openstack manila keys
Comment Actions
Change 726569 merged by Arturo Borrero Gonzalez:
[labs/private@master] hieradata: drop openstack manila keys
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-10-05T09:39:36Z] <arturo> [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) T291257
Comment Actions
Change 724445 abandoned by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: keystone: allow manila-share auth as novaadmin
Reason:
no longer interested in openstack manila