I would maybe force --branch main if you select --apply (or at least require --force to apply a different branch)

We should also make use of Spicerack's locking functionality to prevent two people from running apply at the same time:
https://doc.wikimedia.org/spicerack/master/introduction.html#distributed-locking

maybe we can only allow plan for non-default branch.

In T370414#9994549, @fnegri wrote:

We should also make use of Spicerack's locking functionality to prevent two people from running apply at the same time:
https://doc.wikimedia.org/spicerack/master/introduction.html#distributed-locking

I have read this and I have some questions:

Spicerack supports also distributed locking to prevent some actions from being executed multiple times in parallel in the environments with etcd configured. Each lock can be defined with arbitraty concurrency and TTL (time to live). That means that each lock can either be exclusive or allow a given number of parallel executions. The locks are saved in etcd.

• Do we make use of the mentioned etcd backend from cloudcumin hosts?
• Is this etcd backend available if we run the cookbook from our laptops?

I'm thinking on the following command line options and semantics:

• cookbook wmcs.openstack.tofu <-- a no argument execution runs the plan in the main branch in all deployments.
• cookbook wmcs.openstack.tofu --mr 123 --plan <-- run plan in the branch associated with a gitlab merge request
• cookbook wmcs.openstack.tofu --mr 123 --deployment codfw1dev --plan <-- run plan in the branch associated with a merge request, only for a given deployment
• cookbook wmcs.openstack.tofu --apply <-- run plan + apply in the main branch. No --mr option is allowed if --apply is selected.

I think I'll make the --plan argument optional (and default to true) so we always run the plan command.

Do we make use of the mentioned etcd backend from cloudcumin hosts?

I was asking @Volans right at this moment :) The answer is that Spicerack locking needs etcd, and cloudcumins are not allowed to access the etcd backend used by prod cumins.

So we can either set up a separate etcd cluster (maybe in cloudcontrols? or on new ganeti vms?), or we could explore adding a separate namespace in the prod etcd, with a different user limited to that namespace.

Is this etcd backend available if we run the cookbook from our laptops?

I expect this cookbook would be usually run from cloudcumins, and we could always run it with --no-locks if we really need to run it locally. Technically we might be able to connect to etcd through the socks proxy that is used when running cookbooks locally, but I'm not sure if it would work.

Change #1055420 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/wmcs-cookbooks@main] cookbooks: add new wmcs.openstack.tofu cookbook

https://gerrit.wikimedia.org/r/1055420

I just noticed this:

± tofu apply --help
[..]
  -lock=false            Don't hold a state lock during the operation. This is
                         dangerous if others might concurrently run commands
                         against the same workspace.

It seems tofu itself is protected against this. We may not need any lock in the cookbooks after all.

It seems tofu itself is protected against this. We may not need any lock in the cookbooks after all.

Yes Tofu has a locking system but unfortunately that requires storing the state in AWS or other supported backends:
https://opentofu.org/docs/language/state/locking/

I think Spicerack locking might be the easiest alternative for us to implement, but given it's not currently enabled in cloudcumins I'm fine with merging the cookbook without locking for now, and implement locking in a separate task.

Change #1055420 merged by Arturo Borrero Gonzalez:

[cloud/wmcs-cookbooks@main] cookbooks: add new wmcs.openstack.tofu cookbook

https://gerrit.wikimedia.org/r/1055420

Next cookbook patch:

• git checkout the gitlab MR into a different directory
• avoid touching / rebasing at all the main /srv/tofu-infra repo for MRs

Also:

• when planing a MR, write results to the gitlab MR as a comment

when planing a MR, write results to the gitlab MR as a comment

I would rather do it from the CI itself (T370652: tofu-infra: introduce additional gitlab-ci automation), but I have no objections if you want to implement it in the cookbook as well.

Change #1058573 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/wmcs-cookbooks@main] wmcs.openstack.tofu: write plan to gitlab MR as note

https://gerrit.wikimedia.org/r/1058573

Change #1058581 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudcumins: deploy gitlab token for tofu-infra

https://gerrit.wikimedia.org/r/1058581

Change #1058581 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudcumins: deploy gitlab token for tofu-infra

https://gerrit.wikimedia.org/r/1058581

Change #1058573 merged by Arturo Borrero Gonzalez:

[cloud/wmcs-cookbooks@main] wmcs.openstack.tofu: write plan to gitlab MR as note

https://gerrit.wikimedia.org/r/1058573

Change #1071600 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/wmcs-cookbooks@main] wmcs.openstack.tofu: don't collapse MR notes with plans if they are small

https://gerrit.wikimedia.org/r/1071600

Change #1071600 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] wmcs.openstack.tofu: don't collapse MR notes with plans if they are small

https://gerrit.wikimedia.org/r/1071600