Page MenuHomePhabricator

Cloud: NFS: PoC: manila with generic driver using DHSS=true
Closed, ResolvedPublic

Description

Details

ProjectBranchLines +/-Subject
labs/privatemaster+0 -4
operations/puppetproduction+9 -601
operations/puppetproduction+12 -8
operations/puppetproduction+7 -7
operations/puppetproduction+18 -21
operations/puppetproduction+14 -0
operations/puppetproduction+39 -25
operations/puppetproduction+0 -3
operations/puppetproduction+10 -5
operations/puppetproduction+143 -77
operations/puppetproduction+4 -0
operations/puppetproduction+33 -125
operations/puppetproduction+6 -0
operations/puppetproduction+1 -0
operations/puppetproduction+27 -3
labs/privatemaster+1 -0
operations/puppetproduction+28 -7
operations/puppetproduction+8 -5
labs/privatemaster+1 -0
operations/puppetproduction+2 -2
operations/puppetproduction+1 -0
labs/privatemaster+1 -0
operations/puppetproduction+3 -3
operations/puppetproduction+10 -0
operations/puppetproduction+5 -5
operations/puppetproduction+6 -5
operations/puppetproduction+459 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-cloud) [2021-09-17T11:35:16Z] <arturo> [codfw1dev] install manila on cloudcontrol2001-dev (T291257)

Change 721805 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: bootstrap manila component

https://gerrit.wikimedia.org/r/721805

Change 721805 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: bootstrap manila component

https://gerrit.wikimedia.org/r/721805

Change 722355 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: don't show config file diff

https://gerrit.wikimedia.org/r/722355

Change 722355 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: don't show config file diff

https://gerrit.wikimedia.org/r/722355

Change 722357 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: correct variable expansion in template

https://gerrit.wikimedia.org/r/722357

Change 722357 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: correct variable expansion in template

https://gerrit.wikimedia.org/r/722357

Change 722567 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: introduce rabbit configuration

https://gerrit.wikimedia.org/r/722567

Change 722568 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: fix datatype for db_host config value

https://gerrit.wikimedia.org/r/722568

Change 722569 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hieradata: openstack: manila: introduce placeholder for rabbit password

https://gerrit.wikimedia.org/r/722569

Change 722567 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: introduce rabbit configuration

https://gerrit.wikimedia.org/r/722567

Change 722568 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: fix datatype for db_host config value

https://gerrit.wikimedia.org/r/722568

Change 722569 merged by Arturo Borrero Gonzalez:

[labs/private@master] hieradata: openstack: manila: introduce placeholder for rabbit password

https://gerrit.wikimedia.org/r/722569

Change 722581 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] hieradata: openstack: codfw1dev: add missing manila key

https://gerrit.wikimedia.org/r/722581

Change 722581 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] hieradata: openstack: codfw1dev: add missing manila key

https://gerrit.wikimedia.org/r/722581

Mentioned in SAL (#wikimedia-cloud) [2021-09-21T10:49:00Z] <arturo> [codfw1dev] create manila database on cloudcontrol-dev nodes (galera) T291257

Mentioned in SAL (#wikimedia-cloud) [2021-09-21T10:57:05Z] <arturo> [codfw1dev] created manila user @ labtestwikitech (T291257)

Mentioned in SAL (#wikimedia-cloud) [2021-09-21T11:06:10Z] <arturo> [codfw1dev] created manila project (T291257)

Mentioned in SAL (#wikimedia-cloud) [2021-09-21T11:06:28Z] <arturo> [codfw1dev] give manila user admin role @ manila project (T291257)

Change 722588 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hieradata: openstack: manila: add user password placeholder

https://gerrit.wikimedia.org/r/722588

Change 722590 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: fix typo in hiera key for rabbit pass

https://gerrit.wikimedia.org/r/722590

Change 722591 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: introduce support for manila user

https://gerrit.wikimedia.org/r/722591

Change 722590 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: fix typo in hiera key for rabbit pass

https://gerrit.wikimedia.org/r/722590

Change 722588 merged by Arturo Borrero Gonzalez:

[labs/private@master] hieradata: openstack: manila: add user password placeholder

https://gerrit.wikimedia.org/r/722588

Change 722591 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: introduce support for manila user

https://gerrit.wikimedia.org/r/722591

Mentioned in SAL (#wikimedia-cloud) [2021-09-21T11:32:01Z] <arturo> [codfw1dev] populated manila DB & created service endpoints (T291257)

Mentioned in SAL (#wikimedia-cloud) [2021-09-21T11:45:04Z] <arturo> [codfw1dev] created rabbitmq user (T291257)

Mentioned in SAL (#wikimedia-cloud) [2021-09-21T12:13:11Z] <arturo> [codfw1dev] trying to create a manila service image (T291257)

Change 722602 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: configure additional bits for the service image

https://gerrit.wikimedia.org/r/722602

Change 722603 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hieradata: openstack: manila: add service instance password

https://gerrit.wikimedia.org/r/722603

Change 722602 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: configure additional bits for the service image

https://gerrit.wikimedia.org/r/722602

Change 722603 merged by Arturo Borrero Gonzalez:

[labs/private@master] hieradata: openstack: manila: add service instance password

https://gerrit.wikimedia.org/r/722603

Change 722607 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: manila: enable services

https://gerrit.wikimedia.org/r/722607

In the first iteration I'll be trying to use base images from https://opendev.org/openstack/manila-image-elements for the service instance as recommended in the upstream docs here https://docs.openstack.org/manila/victoria/install/post-install.html#post-install
This image is configured to use user/pass auth by default, but we should revisit that.

Change 722607 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: codfw1dev: manila: enable services

https://gerrit.wikimedia.org/r/722607

Change 722639 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: install python3-manilaclient

https://gerrit.wikimedia.org/r/722639

Change 722639 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: install python3-manilaclient

https://gerrit.wikimedia.org/r/722639

Change 722645 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: install manila-data package

https://gerrit.wikimedia.org/r/722645

Note:

manila-share is refusing to start con cloudcontrol2001-dev, with stack trace:

2021-09-21 16:11:05.544 10696 ERROR oslo_service.service Traceback (most recent call last):
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 807, in run_service
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     service.start()
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/manila/service.py", line 129, in start
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     self.manager.init_host()
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 193, in wrapped
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     return f(self, *args, **kwargs)
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 330, in init_host
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     (self.driver.service_instance_manager.network_helper.
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     return f(*args, **kwargs)
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 247, in network_helper
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     self._network_helper.setup_connectivity_with_service_instances()
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 970, in setup_connectivity_with_service_instances
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     self.service_network_id)
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     return f(*args, **kwargs)
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 804, in service_network_id
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     self._service_network_id = self._get_service_network_id()
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     return f(*args, **kwargs)
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 825, in _get_service_network_id
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     self.admin_project_id, service_network_name)['id']
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/manila/network/neutron/api.py", line 252, in network_create
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service     message=e.message)
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service manila.exception.NetworkException: Unable to create the network. No tenant network is available for allocation.
2021-09-21 16:11:05.544 10696 ERROR oslo_service.service Neutron server returns request_ids: ['req-f8729973-d668-49cc-be36-e6b05865e245']

Reading the reference source code led me to:

/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py
def _get_service_network_id(self):
    """Finds existing or creates new service network."""
    service_network_name = self.get_config_option("service_network_name")
    networks = []
    for network in self.neutron_api.get_all_admin_project_networks():
        if network['name'] == service_network_name:
            networks.append(network)
    if len(networks) > 1:
        raise exception.ServiceInstanceException(
            _('Ambiguous service networks.'))
    elif not networks:
        return self.neutron_api.network_create(
            self.admin_project_id, service_network_name)['id']
    else:
        return networks[0]['id']

it doesn't find a network, because:

/usr/lib/python3/dist-packages/manila/network/neutron/api.py
def get_all_admin_project_networks(self):
    search_opts = {'tenant_id': self.admin_project_id, 'shared': False}
    nets = self.client.list_networks(**search_opts).get('networks', [])
    return nets

It turns out that we created our network with shared=True:

root@cloudcontrol2001-dev:~# openstack network show 05a5494a-184f-4d5c-9e98-77ae61c56daa
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2018-03-16T21:38:53Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | 05a5494a-184f-4d5c-9e98-77ae61c56daa |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | None                                 |
| is_vlan_transparent       | None                                 |
| mtu                       | 1500                                 |
| name                      | lan-flat-cloudinstances2b            |
| port_security_enabled     | True                                 |
| project_id                | admin                                |
| provider:network_type     | flat                                 |
| provider:physical_network | cloudinstances2b                     |
| provider:segmentation_id  | None                                 |
| qos_policy_id             | None                                 |
| revision_number           | 3                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   | 7adfcebe-b3d0-4315-92fe-e8365cc80668 |
| tags                      |                                      |
| updated_at                | 2020-01-02T14:27:46Z                 |
+---------------------------+--------------------------------------+

Apparently manila requires the base flat network to be shared=false? I need to investigate this a bit more. Why do we have our network created with shared=true?

If this approach doesn't work, I think I can try with other network plugins, or even with DHSS=false so we would need to create NFS VMs by hand and manila would only manage the shares/exports.

Change 722645 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: install manila-data package

https://gerrit.wikimedia.org/r/722645

From https://docs.openstack.org/manila/victoria/admin/shared-file-systems-share-networks.html#shared-file-systems-share-networks

You must always specify a share network when creating a share with a share type that requests hard multi-tenancy, i.e., has extra-spec ‘driver_handles_share_servers=True’.

I filled upstream bug requesting the documentation be extended: https://bugs.launchpad.net/manila/+bug/1944696

From reading the manila source code, I draw the following conclusions:

Using the generic driver there are 2 options:

  • use DHSS=true --> manila creates a nova VM and then injects shares configurations on it

It apparently will read the following specific config options in manila.conf:

service_image_name = xxxx
service_instance_name_template = xxxx
manila_service_keypair_name = xxxx
path_to_public_key = xxxx
service_instance_security_group = xxxx
service_instance_flavor_id = xxxx
service_network_name = xxxx
service_network_cidr = xxxx
service_network_division_mask = xxxx
interface_driver = xxxx
connect_share_server_to_tenant_network = xxxx
admin_network_id = xxxx
admin_subnet_id = xxxx
NOTE: within this mode, there are apparently 2 options, using tenant-networks or an admin-managed network (which is what would be of our interest). This logic however, can only be seen on the source code and is not explained in the docs (see https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/service_instance.py)

To use the admin-controlled neutron network, the config should have (see https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/service_instance.py#L779):

connect_share_server_to_tenant_network = true
admin_network_id = xxxx
admin_subnet_id = xxxx
interface_driver = manila.network.linux.interface.NoopInterfaceDriver
  • use DHSS=false --> the service instance should be created by hand, and then manila will inject share configuration on it. It will still request IP addresses per share.

It apparently will read the following config options in manila.conf:

service_instance_name_or_id = xxxx
service_net_name_or_ip = xxxx
tenant_net_name_or_ip = xxxx
service_instance_user = xxxx
service_instance_password = xxxx
path_to_private_key = xxxx
max_time_to_build_instance = xxxx
limit_ssh_access = xxxx

Other global options for this driver can be seen at https://opendev.org/openstack/manila/src/branch/master/manila/share/drivers/generic.py

I was able to start the manila-share agent with this configuration file:

[DEFAULT]
debug = true
osapi_share_base_URL = openstack.codfw1dev.wikimediacloud.org
auth_strategy = keystone
state_path = /var/lib/manila
my_ip = 208.80.153.59
osapi_share_listen = 208.80.153.59
osapi_share_listen_port = 18786
osapi_share_workers = 1
osapi_share_use_ssl = false
transport_url = rabbit://manila:REDACTED@cloudcontrol2001-dev.wikimedia.org:5672,manila:REDACTED@cloudcontrol2003-dev.wikimedia.org:5672,manila:REDACTED@cloudcontrol2004-dev.wikimedia.org:5672
network_plugin_ipv4_enabled = true
network_plugin_ipv6_enabled = false
neutron_net_id =05a5494a-184f-4d5c-9e98-77ae61c56daa
neutron_subnet_id = 7adfcebe-b3d0-4315-92fe-e8365cc80668
enabled_share_backends = wmcscinderbackend

[wmcscinderbackend]
share_backend_name = wmcscinderbackend
share_driver = manila.share.drivers.generic.GenericShareDriver
driver_handles_share_servers = True
enabled_share_protocols = NFS
interface_driver = manila.network.linux.interface.NoopInterfaceDriver
share_name_template = share-%s
share_snapshot_name_template = share-snapshot-%s
quota_shares = 50
quota_snapshots = 50
quota_gigabytes = 1000
quota_snapshot_gigabytes = 1000
quota_share_networks = 0
quota_share_replicas = 0
quota_replica_gigabytes = 0
max_gigabytes = 10000
share_mount_template = mount -vt %(proto)s %(options)s %(export)s %(path)s
share_unmount_template = umount -v %(path)s
volume_name_template = manila-share-%s
volume_snapshot_name_template = manila-snapshot-%s
share_mount_path = /shares
share_helpers = NFS=manila.share.drivers.helpers.NFSHelper
share_volume_fstype = ext4
volume_api_class = manila.volume.cinder.API
cinder_volume_type = 31fc5bfc-bc32-40e8-8865-90cfdf67a977
service_instance_user = manila
service_instance_password = REDACTED
service_net_name_or_ip = lan-flat-cloudinstances2b
service_image_name = 670e1578-5849-40c0-ad16-a0accb42363f
service_instance_flavor_id = cb15aca6-3ff5-4623-8d21-0d37d3092835
connect_share_server_to_tenant_network = True
admin_network_id = 05a5494a-184f-4d5c-9e98-77ae61c56daa
admin_subnet_id = 7adfcebe-b3d0-4315-92fe-e8365cc80668
automatic_share_server_cleanup = false

[cinder]
region_name = codfw1dev-r
auth_url = http://openstack.codfw1dev.wikimediacloud.org:8776/v3/
auth_type = v3password
password = REDACTED
project_domain_name = default
project_name = admin
auth_type = v3password
default_domain_name = admin
domain_name = default
password = REDACTED
project_domain_name = default
project_name = admin
user_domain_id = default
user_domain_name = default
username = novaadmin

[database]
connection = mysql+pymysql://manila:REDACTED@openstack.codfw1dev.wikimediacloud.org/manila

[glance]
api_microversion = 2
region_name = codfw1dev-r
auth_url = http://openstack.codfw1dev.wikimediacloud.org:9292
auth_type = v3password
default_domain_name = admin
domain_name = default
password = REDACTED
project_domain_name = default
project_name = admin
user_domain_id = default
user_domain_name = default
username = novaadmin

[keystone_authtoken]
auth_host = openstack.codfw1dev.wikimediacloud.org
auth_protocol = http
www_authenticate_uri = http://openstack.codfw1dev.wikimediacloud.org:5000
auth_url = http://openstack.codfw1dev.wikimediacloud.org:35357
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = manila
username = manila
password = REDACTED
memcached_servers = cloudcontrol2001-dev.wikimedia.org:11211,cloudcontrol2003-dev.wikimedia.org:11211,cloudcontrol2004-dev.wikimedia.org:11211
region_name = codfw1dev-r

[neutron]
service_metadata_proxy = true
metadata_proxy_shared_secret = REDACTED
auth_url = http://openstack.codfw1dev.wikimediacloud.org:5000/v3
auth_type = v3password
password = REDACTED
project_domain_name = default
project_name = admin
tenant_name = admin
user_domain_id = default
user_domain_name = default
username = novaadmin
region_name = codfw1dev-r

[nova]
region_name = codfw1dev-r
auth_url = http://openstack.codfw1dev.wikimediacloud.org:8774/v2.1
auth_type = v3password
password = REDACTED
project_domain_name = default
project_name = admin
tenant_name = admin
user_domain_id = default
user_domain_name = default
username = novaadmin

[oslo_concurrency]
lock_path = /var/lock/manila

[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]

[oslo_messaging_rabbit]
rabbit_ha_queues = true

[oslo_middleware]

[oslo_policy]
policy_file = policy.json
[ssl]

I want to believe this works in this way (to be confirmed yet):

  • DHSS=true, meaning manila creates nova VMs to host NFS shares
  • such VMs will be attached to the admin-managed neutron network (i.e, our lan flat network, no tenant networks)

When trying the config described above, I hit another problem. I think this is simply manila trying to create the service VM and failing because keystone auth.

/var/log/manila/manila-share.log
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 193, in wrapped
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/utils.py", line 568, in wrapper
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 1878, in create_share_instance
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     detail=message_field.Detail.NO_SHARE_SERVER)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     raise value
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 1860, in create_share_instance
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     share_group=share_group_ref,
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 655, in _provide_share_server_for_share
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return _wrapped_provide_share_server_for_share()
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 360, in inner
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 651, in _wrapped_provide_share_server_for_share
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     context, compatible_share_server, metadata))
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 751, in _create_share_server_in_backend
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     share_server = self._setup_server(context, share_server, metadata)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 3926, in _setup_server
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     self.driver.deallocate_network(context, share_server['id'])
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     raise value
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/manager.py", line 3875, in _setup_server
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     network_info, metadata=metadata)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/driver.py", line 945, in setup_server
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return self._setup_server(*args, **kwargs)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/drivers/generic.py", line 897, in _setup_server
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     self.admin_context, network_info)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 455, in set_up_service_instance
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     context, instance_name, network_info)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 549, in _create_service_instance
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     service_image_id = self._get_service_image(context)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 534, in _get_service_image
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     images = [image.id for image in self.image_api.image_list(context)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py", line 534, in <listcomp>
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     images = [image.id for image in self.image_api.image_list(context)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 583, in __next__
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return self._next()
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 572, in _next
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     obj, resp = next(self._self_wrapped)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 183, in list
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     for image, resp in paginate(url, page_size, limit):
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 110, in paginate
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     resp, body = self.http_client.get(next_url, headers=req_id_hdr)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return self.request(url, 'GET', **kwargs)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 366, in request
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     **kwargs)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return self.session.request(url, method, **kwargs)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 780, in request
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     auth_headers = self.get_auth_headers(auth)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1191, in get_auth_headers
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return auth.get_headers(self, **kwargs)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     token = self.get_token(session)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     return self.get_access(session).auth_token
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     self.auth_ref = self.get_auth_ref(session)
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 144, in get_auth_ref
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server     message='Authentication cannot be scoped to multiple'
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.auth.AuthorizationFailure: Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped
2021-09-23 11:42:21.765 7816 ERROR oslo_messaging.rpc.server

I think the problem is simply misconfigured credentials somewhere.

Steps I made:

root@cloudcontrol2001-dev:~# manila type-list
+--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+
| ID                                   | Name               | visibility | is_default | required_extra_specs                | optional_extra_specs | Description |
+--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+
| febfcfe1-9d42-4ac1-aa85-90f63aac7c2b | default_share_type | public     | -          | driver_handles_share_servers : True |                      | None        |
+--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+
root@cloudcontrol2001-dev:~# manila share-network-show manila-share-network
+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property              | Value                                                                                                                                                                                                                                                                                                                                                                             |
+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                    | 929a7860-7b73-4544-a20e-45942c64dffe                                                                                                                                                                                                                                                                                                                                              |
| name                  | manila-share-network                                                                                                                                                                                                                                                                                                                                                              |
| project_id            | admin                                                                                                                                                                                                                                                                                                                                                                             |
| created_at            | 2021-09-21T15:28:49.000000                                                                                                                                                                                                                                                                                                                                                        |
| updated_at            | None                                                                                                                                                                                                                                                                                                                                                                              |
| description           | None                                                                                                                                                                                                                                                                                                                                                                              |
| share_network_subnets | [{'id': 'cc5ba5f5-bacb-4150-a176-884bcdb2ce91', 'availability_zone': None, 'created_at': '2021-09-21T15:28:49.000000', 'updated_at': None, 'segmentation_id': None, 'neutron_net_id': '05a5494a-184f-4d5c-9e98-77ae61c56daa', 'neutron_subnet_id': '7adfcebe-b3d0-4315-92fe-e8365cc80668', 'ip_version': None, 'cidr': None, 'network_type': None, 'mtu': None, 'gateway': None}] |
+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@cloudcontrol2001-dev:~# openstack network show 05a5494a-184f-4d5c-9e98-77ae61c56daa
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2018-03-16T21:38:53Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | 05a5494a-184f-4d5c-9e98-77ae61c56daa |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | None                                 |
| is_vlan_transparent       | None                                 |
| mtu                       | 1500                                 |
| name                      | lan-flat-cloudinstances2b            |
| port_security_enabled     | True                                 |
| project_id                | admin                                |
| provider:network_type     | flat                                 |
| provider:physical_network | cloudinstances2b                     |
| provider:segmentation_id  | None                                 |
| qos_policy_id             | None                                 |
| revision_number           | 3                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   | 7adfcebe-b3d0-4315-92fe-e8365cc80668 |
| tags                      |                                      |
| updated_at                | 2020-01-02T14:27:46Z                 |
+---------------------------+--------------------------------------+
root@cloudcontrol2001-dev:~# manila create NFS 1 --name myshare2 --description "My Manila share" --share-type default_share_type --share-network manila-share-network
+---------------------------------------+--------------------------------------+
| Property                              | Value                                |
+---------------------------------------+--------------------------------------+
| id                                    | 917a34aa-e018-4435-96c1-f9df77a09128 |
| size                                  | 1                                    |
| availability_zone                     | None                                 |
| created_at                            | 2021-09-23T11:42:21.000000           |
| status                                | creating                             |
| name                                  | myshare2                             |
| description                           | My Manila share                      |
| project_id                            | admin                                |
| snapshot_id                           | None                                 |
| share_network_id                      | 929a7860-7b73-4544-a20e-45942c64dffe |
| share_proto                           | NFS                                  |
| metadata                              | {}                                   |
| share_type                            | febfcfe1-9d42-4ac1-aa85-90f63aac7c2b |
| is_public                             | False                                |
| snapshot_support                      | False                                |
| task_state                            | None                                 |
| share_type_name                       | default_share_type                   |
| access_rules_status                   | active                               |
| replication_type                      | None                                 |
| has_replicas                          | False                                |
| user_id                               | novaadmin                            |
| create_share_from_snapshot_support    | False                                |
| revert_to_snapshot_support            | False                                |
| share_group_id                        | None                                 |
| source_share_group_snapshot_member_id | None                                 |
| mount_snapshot_support                | False                                |
| progress                              | None                                 |
| share_server_id                       | None                                 |
| host                                  |                                      |
+---------------------------------------+--------------------------------------+
root@cloudcontrol2001-dev:~# manila share-server-list
+--------------------------------------+----------------------------------------+--------+----------------------+------------+----------------------------+--------------------------------------+
| Id                                   | Host                                   | Status | Share Network        | Project Id | Updated_at                 | Share Network Subnet Id              |
+--------------------------------------+----------------------------------------+--------+----------------------+------------+----------------------------+--------------------------------------+
| baa04ea6-f837-4eb7-9936-1eec3a11a886 | cloudcontrol2001-dev@wmcscinderbackend | error  | manila-share-network | admin      | 2021-09-23T11:29:52.000000 | cc5ba5f5-bacb-4150-a176-884bcdb2ce91 |
+--------------------------------------+----------------------------------------+--------+----------------------+------------+----------------------------+--------------------------------------+

root@cloudcontrol2001-dev:~# manila list
+--------------------------------------+----------+------+-------------+--------+-----------+--------------------+----------------------------------------------------------+-------------------+
| ID                                   | Name     | Size | Share Proto | Status | Is Public | Share Type Name    | Host                                                     | Availability Zone |
+--------------------------------------+----------+------+-------------+--------+-----------+--------------------+----------------------------------------------------------+-------------------+
| 917a34aa-e018-4435-96c1-f9df77a09128 | myshare2 | 1    | NFS         | error  | False     | default_share_type | cloudcontrol2001-dev@wmcscinderbackend#wmcscinderbackend | nova              |
+--------------------------------------+----------+------+-------------+--------+-----------+--------------------+----------------------------------------------------------+-------------------+

I think you found this already, but this issue is somewhat discussed on https://bugs.launchpad.net/openstack-ansible/+bug/1506285 -- the issue is with specifying domain_name in the [glance] section.

With that commented out, we get to what I think is the next problem: keystone returning a completely bogus response.

2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server     return self.get_access(session).auth_token
2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server     self.auth_ref = self.get_auth_ref(session)
2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 197, in get_auth_ref
2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server     raise exceptions.InvalidResponse(response=resp)
2021-09-24 05:35:01.611 9116 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.response.InvalidResponse: Invalid response from server.

No idea what that's about. The exception is a side-effect of a failure in _provide_share_server_for_share

I think you found this already, but this issue is somewhat discussed on https://bugs.launchpad.net/openstack-ansible/+bug/1506285 -- the issue is with specifying domain_name in the [glance] section.

With that commented out, we get to what I think is the next problem: keystone returning a completely bogus response.

Nice, I though I was breaking something xd, will debug with keystone misbehaving in mind

It seems that we might have to pass a version explicitly or something:

2021-09-24 08:06:42.926 19922 ERROR keystoneauth.identity.v3.base [req-a4026bc8-992b-4639-91c8-509362c4c1c6 novaadmin admin - - -] Got response: <Response [300]>
resp_data:{'versions': [{'id': 'v2.11', 'status': 'CURRENT', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.10', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.9', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.8', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.7', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.6', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.5', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.4', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.3', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.2', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.1', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}, {'id': 'v2.0', 'status': 'SUPPORTED', 'links': [{'rel': 'self', 'href': 'http://openstack.codfw1dev.wikimediacloud.org:9292/v2/'}]}]}

Added the /v2 ending to the auth endpoint for glance, and now the error is different ("keystoneauth1.exceptions.http.Unauthorized: Unrecognized schema in response body. (HTTP 401)"), looking

okok, seems to be a regular unauthorized issue :/, maybe thereś something wrong in the versions of the libs though, as it complains about unrecognized schema...

2021-09-24 08:19:32.148 15803 ERROR keystoneauth.session [req-9719bfda-4b86-4126-a6fa-f7b9bc4eac3b novaadmin admin - - -] Got error <Response [401]>: {'Content-Type': 'application/json', 'WWW-Authenticate': 'Keystone uri="http://openstack.codfw1dev.wikimediacloud.org:5000"', 'Connection': 'close'}
 {'message': 'This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.<br /><br />\n\n\n', 'code': '401 Unauthorized', 'title': 'Unauthorized'}

okok, so finally used the v3 endopint instead of the v2 for auth (saw it being used somewhere else):

auth_url = http://openstack.codfw1dev.wikimediacloud.org:5000/v3

And now the error is about not finding an image, nice:

2021-09-24 08:24:01.363 24366 ERROR oslo_messaging.rpc.server [req-f5f7d02b-e2ca-44cd-992e-64fc5c18f2c8 novaadmin admin - - -] Exception during message handling: manila.exception.ServiceInstanceException: Image with name '670e1578-5849-40c0-ad16-a0accb42363f' was not found or is not in 'active' state.

looking

it's there, and active:

root@cloudcontrol2001-dev:~# openstack image list | grep 670e1578-5849-40c0-ad16-a0accb42363f
| 670e1578-5849-40c0-ad16-a0accb42363f | manila-service-image-1.3.0-75-g631a854     | active      |

It turns out that it wants the name, but not the id xd

And the next step, auth with the novaclient \o/ progress!

2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/novaclient/base.py", line 363, in _create
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.post(url, body=body)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 401, in post
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     return self.request(url, 'POST', **kwargs)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/novaclient/client.py", line 72, in request
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     **kwargs)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 554, in request
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     return self.session.request(url, method, **kwargs)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 780, in request
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     auth_headers = self.get_auth_headers(auth)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1191, in get_auth_headers
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     return auth.get_headers(self, **kwargs)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     token = self.get_token(session)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     return self.get_access(session).auth_token
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     self.auth_ref = self.get_auth_ref(session)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 188, in get_auth_ref
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     authenticated=False, log=False, **rkwargs)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1139, in post
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     return self.request(url, 'POST', **kwargs)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 976, in request
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, method, url)
2021-09-24 08:39:31.072 23493 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-bfca3b24-c100-4c6b-a63e-03f29833ed3c)

Okok, I setup the novaclient with the v3 auth api endpoint and it worked.

Now the instance was started, and I can see the console through virsh:

root@cloudvirt2003-dev:~# virsh
virsh # console 121
Connected to domain 'i-00000c58'
Escape character is ^] (Ctrl + ])

ubuntu login:

but it does not reply to ssh from the cloudcontrol:

2021-09-24 08:59:23.368 17050 DEBUG manila.share.drivers.service_instance [req-1e04705b-5164-467a-aa63-0946c0a9d4ea novaadmin admin - - -] [Errno 110] ETIMEDOUT _test_server_connection /usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py:673
2021-09-24 08:59:23.369 17050 DEBUG manila.share.drivers.service_instance [req-1e04705b-5164-467a-aa63-0946c0a9d4ea novaadmin admin - - -] Server 172.16.128.89 is not available via SSH. Waiting... _test_server_connection /usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py:675
2021-09-24 08:59:28.371 17050 DEBUG manila.share.drivers.service_instance [req-1e04705b-5164-467a-aa63-0946c0a9d4ea novaadmin admin - - -] Checking server availability. _check_server_availability /usr/lib/python3/dist-packages/manila/share/drivers/service_instance.py:659

looking

Change 723470 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: refresh configuration file

https://gerrit.wikimedia.org/r/723470

Change 723470 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: refresh configuration file

https://gerrit.wikimedia.org/r/723470

Change 723492 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: separate manila-share service into a different role

https://gerrit.wikimedia.org/r/723492

Submitted another bug report for manila, in this case a feature request: https://bugs.launchpad.net/manila/+bug/1944980

Change 723492 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: separate manila-share service into a different role

https://gerrit.wikimedia.org/r/723492

Change 723521 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila_sharecontroller: require Debian Bullseye

https://gerrit.wikimedia.org/r/723521

Change 723521 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila_sharecontroller: require Debian Bullseye

https://gerrit.wikimedia.org/r/723521

I stop at this point: figuring out hiera for rabbit @ manila-share-controller-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud to see if the manila-share service can work from there.

The manila-share package (and manila-common) fails to install on the VM because there is a manila user defined on LDAP.

hey @Andrew could you help me rename the unix user manila in codfw1dev to something else? perhaps manila-srv. Or even completely delete the user, given I don't think we need it.

The LDAP dir for codfw1dev seems to be on cloudservices2002-dev.wikimedia.org

NOTE: it was registered by me using labtestwikitech.wikimedia.org

hey @Andrew could you help me rename the unix user manila in codfw1dev to something else? perhaps manila-srv. Or even completely delete the user, given I don't think we need it.

The LDAP dir for codfw1dev seems to be on cloudservices2002-dev.wikimedia.org

NOTE: it was registered by me using labtestwikitech.wikimedia.org

This has been done using this documentation: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Testing_deployment#LDAP Now manila-share and manila-common install well on the VM.

Next step: figure out rabbitmq connectivity.

Change 724350 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: refresh config file

https://gerrit.wikimedia.org/r/724350

Change 724350 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: refresh config file

https://gerrit.wikimedia.org/r/724350

Change 724356 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: configuration template cleanups

https://gerrit.wikimedia.org/r/724356

Change 724391 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: galera: allow optional access to the database from manila share

https://gerrit.wikimedia.org/r/724391

Change 724391 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: galera: allow optional access to the database from manila share

https://gerrit.wikimedia.org/r/724391

Mentioned in SAL (#wikimedia-cloud) [2021-09-28T11:30:02Z] <arturo> [codfw1dev] create floating IP 185.15.57.5 for manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org (T291257)

Change 724445 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: keystone: allow manila-share auth as novaadmin

https://gerrit.wikimedia.org/r/724445

I stop at this point:

  • manila-share running on manila-share-controller-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud needs keystone & rabbit & database credentials for the service to work.
  • I initially attempted to replicate credentials from cloudcontrols, but was grumpy about it (also see https://gerrit.wikimedia.org/r/724445)
  • @Andrew volunteer to try to work out some additional privilege separation
  • current secrets are at labs/private.git: hieradata/cloudinfra-codfw1dev.yaml on cloudinfra-internal-puppetmaster-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud. All the secrets ends in /etc/manila/manila.conf in the share controller VM.

Change 724500 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] manila: use manila-srv service user rather than novaadmin for auth

https://gerrit.wikimedia.org/r/724500

Change 724500 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] manila: use manila-srv service user rather than novaadmin for auth

https://gerrit.wikimedia.org/r/724500

Mentioned in SAL (#wikimedia-cloud) [2021-09-29T09:41:00Z] <arturo> [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured (T291257)

Change 724725 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: use manilainfra project to host service instances

https://gerrit.wikimedia.org/r/724725

Change 724725 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: use manilainfra project to host service instances

https://gerrit.wikimedia.org/r/724725

Change 724726 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: manila: refresh services on config file change

https://gerrit.wikimedia.org/r/724726

Change 724726 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: manila: refresh services on config file change

https://gerrit.wikimedia.org/r/724726

Latest updates:

  • changed the internal tenant from service to be manilainfra, which is a bit more in line with others we have been using lately.
  • manila-share is able to create the service VM. It apparently ignores the service_instance_name_template = manila-service-instance-%s setting, because the VMs have uuids as names.
  • the service VM ends up with 2 neutron IPs. I can manually check that SSH works in one of them, but manila-share tries the wrong one, example:
2021-09-29 12:12:00.744 135757 ERROR oslo_messaging.rpc.server manila.exception.ServiceInstanceException: SSH connection has not been established to 172.16.128.103 in 300s. Giving up.
2021-09-29 12:12:00.744 135757 ERROR oslo_messaging.rpc.server
root@cloudcontrol2001-dev:~# openstack server show 0fa41323-0c0f-4adf-8b6d-aede186b8412 | grep addresses
| addresses                           | lan-flat-cloudinstances2b=172.16.128.115, 172.16.128.103                      |

root@manila-share-controller-01:~# telnet 172.16.128.103 22
Trying 172.16.128.103...
^C
root@manila-share-controller-01:~# telnet 172.16.128.115 22
Trying 172.16.128.115...
Connected to 172.16.128.115.
Escape character is '^]'.
SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.3

I believe what's happening with the 2 ports things is that the service VM recvs SSH packets on one interface but replies on other. The neutron port firewall may be filtering this (the return packet). Need to investigate a bit more.

I believe what's happening with the 2 ports things is that the service VM recvs SSH packets on one interface but replies on other. The neutron port firewall may be filtering this (the return packet). Need to investigate a bit more.

Reported bug upstream https://bugs.launchpad.net/manila/+bug/1946002

After this evaluation we decided we wont be using Openstack Manila for now.

Change 726564 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: drop manila code

https://gerrit.wikimedia.org/r/726564

Change 726564 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: drop manila code

https://gerrit.wikimedia.org/r/726564

Change 726569 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hieradata: drop openstack manila keys

https://gerrit.wikimedia.org/r/726569

aborrero changed the task status from In Progress to Open.Tue, Oct 5, 9:34 AM

Change 726569 merged by Arturo Borrero Gonzalez:

[labs/private@master] hieradata: drop openstack manila keys

https://gerrit.wikimedia.org/r/726569

Mentioned in SAL (#wikimedia-cloud) [2021-10-05T09:39:36Z] <arturo> [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) T291257