root@cloudcumin1001:~# cookbook wmcs.vps.create_project --user nmw03 --cluster-name eqiad1 --project wlmaz --description 'Interactive map for W
iki Loves Monuments Azerbaijan'
...
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00,  2.36hosts/s]
FAIL |                                                                                                         |   0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'sudo -i rm -rf -...27e-f326da036bf6'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
INFO: running tofu for deployment 'codfw1dev', node 'cloudcontrol2005-dev.codfw.wmnet'
----- OUTPUT of 'sudo -i mkdir -p...acd-7f98fb591189' -----
================
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00,  1.27hosts/s]
FAIL |                                                                                                         |   0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'sudo -i mkdir -p...acd-7f98fb591189'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
INFO: checking out gitlab MR branch in /tmp/tofu_infra_mr287_254bbaef-d4d1-4480-aacd-7f98fb591189
----- OUTPUT of 'sudo -i tofu -ch...f98fb591189 init' -----

Initializing the backend...

Successfully configured the backend "s3"! OpenTofu will automatically
use this backend unless the backend configuration changes.
Error refreshing state: operation error S3: HeadObject, https response error StatusCode: 403, RequestID: tx00000dd41a4820c2bc8aa-006945569f-123016c-default, HostID: , api error Forbidden: Forbidden
================
100.0% (1/1) of nodes failed to execute command 'sudo -i tofu -ch...f98fb591189 init': cloudcontrol2005-dev.codfw.wmnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: 'sudo -i tofu -ch...f98fb591189 init'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
----- OUTPUT of 'sudo -i rm -rf -...acd-7f98fb591189' -----
================
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00,  1.27hosts/s]
FAIL |                                                                                                         |   0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'sudo -i rm -rf -...acd-7f98fb591189'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Exception raised while executing cookbook wmcs.vps.create_project:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/common.py", line 826, in _wrapped_run
    return object.__getattribute__(self, __name)(*args, **kwargs)
  File "/srv/deployment/wmcs-cookbooks/cookbooks/wmcs/vps/create_project.py", line 372, in run
    OpenstackTofuRunner(
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/common.py", line 826, in _wrapped_run
    return object.__getattribute__(self, __name)(*args, **kwargs)
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/common.py", line 844, in run
    return self.run_with_proxy()
  File "/srv/deployment/wmcs-cookbooks/cookbooks/wmcs/openstack/tofu.py", line 292, in run_with_proxy
    self._run(cluster_name=str(cluster_name))
  File "/srv/deployment/wmcs-cookbooks/cookbooks/wmcs/openstack/tofu.py", line 273, in _run
    self._tofu_plan(node, plan_file=plan_file, op_dir=op_dir)
  File "/srv/deployment/wmcs-cookbooks/cookbooks/wmcs/openstack/tofu.py", line 178, in _tofu_plan
    run_one_raw(
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/common.py", line 279, in run_one_raw
    return run_one_raw_needed_to_be_able_to_mock(
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/common.py", line 217, in run_one_raw_needed_to_be_able_to_mock
    result = next(node.run_sync(command, **run_sync_params))
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 556, in run_sync
    return self._execute(
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 763, in _execute
    raise RemoteExecutionError(ret, "Cumin execution failed", worker.get_results())
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project wlmaz in eqiad1