openstack: clarify default security group semantics
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• aborrero
	Sep 18 2024, 4:07 PM

Description

There are at least 3 or 4 conflicting mechanisms for introducing a set of neutron security group rules at project creation:

the native neutron default security group rules mechanism. This is https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/default-security-group-rule.html
the custom keystone hooks we have. This is https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/openstack/files/caracal/keystone/wmfkeystonehooks/wmfkeystonehooks.py
tofu-infra. This is https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra
stuff created via wmcs cookbooks

We have been using them inconsistently, and each has advantages and limitations.

The problem is that they don't mix well. For example:

neutron wont notice the rules created by our custom keystone hook as being 'default' rules
tofu-infra wont be able to track rules creates by either the neutron native mechanism or the keystone hook (or the cookbooks)

I think my proposal is to declare tofu-infra as the primary source of truth and drop the logic from all other places.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	openstack: keystone: dont add default security rules via wmfkeystonehooks	operations/puppet	production	+192 -328

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T370037 Cloud VPS: extend tofu-infra coverage
Resolved	• aborrero	T374835 cloud: tofu-infra: support neutron security groups
Resolved	• aborrero	T375111 openstack: clarify default security group semantics

Event Timeline

• aborrero created this task.Sep 18 2024, 4:07 PM

I agree on declaring tofu-infra as the source of truth and I'd like if we could drop the keystone hooks and the cookbook-created rules.

I see two possible solutions:

create a Tofu module with a set of default rules, and include the module as a dependency of modules/projects, so that when Tofu creates a project it also creates a security group including the default rules
create a Tofu module with a "native neutron default security group", so that Tofu only manages that resource (the "default SG") while Neutron creates the actual rules based on the Tofu-managed defaults

I think I prefer 1. as it means if you change one of the defaults, you could run "tofu apply" to apply the change to all projects. Using 2. the change would only apply to new projects created in the future.

Of course we will need some testing and it might be more complicated than it looks. :D

• aborrero changed the task status from Open to In Progress.Sep 19 2024, 8:10 AM

• aborrero triaged this task as Medium priority.

• aborrero moved this task from Backlog to Doing on the User-aborrero board.

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50

secgroups: add optional default security group

In T375111#10159813, @CodeReviewBot wrote:

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50

secgroups: add optional default security group

this has a somewhat nice abstraction to your point @fnegri

However, I'm finding that openstack wont let me delete a default security group from a project (it will create another with the same name)

So I think the strategy could be:

Assume neutron will: always create a default SG, always self manage this group, with the very basic connectivity rules (egress, same SG)
Assume we will never be able to track this neutron default SG in tofu-infra.
Stop adding "our" default rules via the custom keystone hook
Add "our" default rules via tofu-infra in a sg named cloud-vps-default (naming TBD ...)

Mentioned in SAL (#wikimedia-cloud) [2024-09-19T09:51:05Z] <arturo> [codfw1dev] play with neutron default security group rules (delete, create them, etc) T375111

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51

secgroups: enable delete_default_rules

mmm not that easy.

When creating a VM, they will be automatically added to the default secgroup.
If we don't have the SSH rules in this group, but in another cloudvps-default we would force an additional step when creating a VM: to add a this security group.

We may be forced to keep using this neutron default security group after all.

Somewhat good news is that I believe we can import it to tofu-infra after it has been automatically created by neutron.

Let me try to put things a bit more clear on what the problem is, and what we would like to achieve.

Use cases

Use case 1:
1.1: When a project is created, a default security group is created automatically.
1.2: When this default security group is created, default rules are added to it from a template.
1.3: This default security group is applied to all ports (VMs) automatically, without user selecting it, or performing any special action.

Use case 2:
2.1: To allow VMs for basic interoperability within our cloud (i.e, SSH access from bastions etc) we need to customize the template of default security group rules.

Use case 3:
3.1: When doing changes to the template of default security group rules, we most likely would want to "backfill" them to existing default security groups in already existing projects

Current state

Default security group creation and default sg rules are not tracked with any IaC tool.

The use case 1, is natively supported by neutron, and it currently operates nromally.

The use case 2, we use a custom keystone hook, to inject a set of rules into the default sg created by neutron.

The use case 3, we don't have anything.

The opentofu openstack provider does not support tracking the template of default sg rules.

Because the default sg is created automatically by neutron on new project creation, the only way to track this in opentofu is to 'import' it after being created.

So, nowadays, we would have a hard time implementing use cases 1, 2 or 3 using opentofu.

Desired state

At very least, we would operate default security group and sg rules using tofu-infra, i.e use cases #1 and #2.

Use case #3 could also be integrated via tofu-infra, but we would need to support #1 and #2 first. Also, given #3 is less frequent, I'm fine creating a script "on the fly", or having a cookbook.

some additional links, this is the neutron code that creates the default sg:

I wish we could have a config option to disable it.

@aborrero thanks for all your investigation! I dug through a few docs and internet pages and maybe I found a potential approach (to be verified!)

manually delete all rules from the "default security group rules". this means that for every new project, openstack will create a security group but it will contain no rules
when we create a new project with tofu, we use tofu to add the policies we need to that "default" SG that openstack created

What I'm unsure about is how we can add the rules to that group given the SG was created by openstack and not with tofu. Maybe we can leverage the provider "data source" to retrieve the ID of the SG that was created, similar to what I found in this post:

# Allow SSH ingress to default security group
data "openstack_networking_secgroup_v2" "default" {
  name        = "default"
}
resource "openstack_networking_secgroup_rule_v2" "sg_rule_ssh" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = "0.0.0.0/0"
  security_group_id = data.openstack_networking_secgroup_v2.default.id
}

Obviously I was missing this data tofu functionality when considering the options earlier. I like this approach, I believe it covers all uses cases, and I will try to implement it.

I have integrated the idea by @fnegri in the patch at https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 with the following semantics:

neutron will still create the default sg in each project (all points in use case #1)
via tofu-infra, a project can be set to manage_default_secgroup: true, which will make tofu to inject templated rules into the default security group (use case #2)
this, most likely, will conflict with currently defined rules, so when running tofu apply for the first time, all rules in the default sg of the newly managed project should be deleted _by hand_ (two clicks in horizon)
from that moment on, we can use tofu-infra to backfill default sg rules if they change (use case #3)

I think the only downside being the manual deletion on the first tofu apply is more than good enough.

We can extend this workflow in the following way:

delete neutron native default security group rules
when a new project is created, a default sg will be created, but empty
because new project creation is managed via tofu-infra, we can set all new projects to have the default security group managed by tofu
in this case, no conflicts will happen, no rules would need to be deleted by hand