Page MenuHomePhabricator

Permission error while trying to create magnum cluster
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • create "jump" bastion box in "duct" CloudVPS project
  • create application credentials for duct with role "member" and "Unrestricted (dangerous)" checked
  • download openrc.sh for application credential
  • scp openrc.sh for application credential onto "jump"
  • ssh onto "jump"
  • sudo apt install python3-magnumclient
  • source app-cred-duct-magnum-openrc.sh
  • openstack coe cluster create duct --cluster-template k8s23 --master-count 1 --node-count 1
  • watch openstack coe cluster list
  • openstack coe cluster show duct

What happens?:

status changes to "CREATE_FAILED" with faults

default-master: 'Resource CREATE failed: Forbidden: resources.kube_masters.resources[0].resources.kube_master_eth0:
  ((rule: create_port and (rule: create_port: fixed_ips and (rule: create_port:
  fixed_ips: subnet_id))) and (rule: create_port: allowed_address_pairs and (rule:
  create_port: allowed_address_pairs: ip_address))) is disallowed by policy

  Neutron server returns request_ids: [''req-14215cec-59f7-4d23-9793-dc3ab7cea49b'']'
default-worker: 'Resource CREATE failed: Forbidden: resources.kube_masters.resources[0].resources.kube_master_eth0:
  ((rule: create_port and (rule: create_port: fixed_ips and (rule: create_port:
  fixed_ips: subnet_id))) and (rule: create_port: allowed_address_pairs and (rule:
  create_port: allowed_address_pairs: ip_address))) is disallowed by policy

  Neutron server returns request_ids: [''req-14215cec-59f7-4d23-9793-dc3ab7cea49b'']'

What should have happened instead?:

The cluster should have been successfully created.

Software version (skip for WMF-hosted wikis like Wikipedia):

kindrobot@jump:~$ openstack --version
openstack 5.8.0

Other information (browser name/version, screenshots, etc.):
Full output of openstack coe cluster show duct -f yaml

status: CREATE_FAILED
health_status: null
cluster_template_id: bdaab57a-b784-483b-aa48-2600f228557f
node_addresses: []
uuid: a37ad549-8c24-4fb1-9451-aefa8f5bddfe
stack_id: 88ca1d41-0817-446e-a128-f5875fa21741
status_reason: default-master failed, default-worker failed
created_at: '2023-04-03T19:15:20+00:00'
updated_at: '2023-04-03T19:15:53+00:00'
coe_version: v1.23.15-rancher1-linux-amd64
labels:
  cloud_provider_enabled: 'true'
  hyperkube_prefix: docker.io/rancher/
  kube_tag: v1.23.15-rancher1-linux-amd64
labels_overridden: {}
labels_skipped: {}
labels_added: {}
fixed_network: lan-flat-cloudinstances2b
fixed_subnet: cloud-instances2-b-eqiad
floating_ip_enabled: false
faults:
  default-master: 'Resource CREATE failed: Forbidden: resources.kube_masters.resources[0].resources.kube_master_eth0:
    ((rule: create_port and (rule: create_port: fixed_ips and (rule: create_port:
    fixed_ips: subnet_id))) and (rule: create_port: allowed_address_pairs and (rule:
    create_port: allowed_address_pairs: ip_address))) is disallowed by policy

    Neutron server returns request_ids: [''req-14215cec-59f7-4d23-9793-dc3ab7cea49b'']'
  default-worker: 'Resource CREATE failed: Forbidden: resources.kube_masters.resources[0].resources.kube_master_eth0:
    ((rule: create_port and (rule: create_port: fixed_ips and (rule: create_port:
    fixed_ips: subnet_id))) and (rule: create_port: allowed_address_pairs and (rule:
    create_port: allowed_address_pairs: ip_address))) is disallowed by policy

    Neutron server returns request_ids: [''req-14215cec-59f7-4d23-9793-dc3ab7cea49b'']'
keypair: null
api_address: null
master_addresses: []
master_lb_enabled: false
create_timeout: 60
node_count: 1
discovery_url: https://discovery.etcd.io/9e08bedb2aac98ef6ad59211c548f1c8
docker_volume_size: 20
master_count: 1
container_version: 1.12.6
name: duct
master_flavor_id: g3.cores2.ram4.disk20
flavor_id: g3.cores2.ram4.disk20
health_status_reason: {}
project_id: duct

Event Timeline

This looks like Magnum trying to manage network stuff (namely managing ports) that's restricted to admins only in our particular OpenStack deployment. :/

rook added a project: PAWS.

This is the same error that occurs when trying to deploy magnum with terraform

rook moved this task from Backlog to Dysfunction on the Openstack-Magnum board.

I am testing this today (in your project 'duct') sorry in advance for the noise and whatever cruft gets left behind.

to my surprise, reverting to the default upstream neutron policies does not seem to change this result.

Change 930676 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Heat and Magnum: include service token with subcalls

https://gerrit.wikimedia.org/r/930676

Change 930676 merged by Andrew Bogott:

[operations/puppet@production] Heat and Magnum: include service token with subcalls

https://gerrit.wikimedia.org/r/930676

Change 930681 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] neutron policy: policy rules to permit members to create magnum clusters

https://gerrit.wikimedia.org/r/930681

Change 930681 merged by Andrew Bogott:

[operations/puppet@production] neutron policy: policy rules to permit members to create magnum clusters

https://gerrit.wikimedia.org/r/930681

Andrew claimed this task.

I can't make promises about the magnum cluster itself, but this particular auth issue should be fixed. Please re-open if you see it again.

Note that I had to increase your cores/RAM quotas slightly to actually create a a cluster as the project was mostly full already. I'm going to leave them where they are (+2 cores and +4G ram) so you don't have to bump up against that second problem.

Change 930683 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] neutron policy: more policy rule changes to support our shared network

https://gerrit.wikimedia.org/r/930683

Change 930683 merged by Andrew Bogott:

[operations/puppet@production] neutron policy: more policy rule changes to support our shared network

https://gerrit.wikimedia.org/r/930683