Steps to replicate the issue (include links if applicable):
$ ssh cloudcumin1001 fnegri@cloudcumin1001:~$ sudo cookbook wmcs.openstack.tofu
What happens?:
Planning failed. OpenTofu encountered an error while generating this plan.
╷
│ Error: Error retrieving openstack_dns_zone_v2 7a2210a4-3a55-48d8-9713-0256d7d9bc1b: Expected HTTP response code [200] when accessing [GET https://openstack.codfw1dev.wikimediacloud.org:29001/v2/zones/7a2210a4-3a55-48d8-9713-0256d7d9bc1b], but got 504 instead: {"code": 504, "type": "timeout", "request_id": "req-1306d84a-e117-4bc1-ba3b-5441e5730424"}
│
│ with module.project["cloudinfra-codfw1dev"].openstack_dns_zone_v2.zone["1.0.0.0.0.0.1.a.0.8.c.e.2.0.a.2.ip6.arpa."],
│ on modules/project/dns.tf line 21, in resource "openstack_dns_zone_v2" "zone":
│ 21: resource "openstack_dns_zone_v2" "zone" {
│
╵
╷
│ Error: Error retrieving openstack_dns_zone_v2 4c754100-1790-4858-a583-9de93c9e8b3d: Expected HTTP response code [200] when accessing [GET https://openstack.codfw1dev.wikimediacloud.org:29001/v2/zones/4c754100-1790-4858-a583-9de93c9e8b3d], but got 504 instead: {"code": 504, "type": "timeout", "request_id": "req-1285f282-c142-435d-959d-467757f87652"}
│
│ with module.project["cloudinfra-codfw1dev"].openstack_dns_zone_v2.zone["codfw1dev.wikimedia.cloud."],
│ on modules/project/dns.tf line 21, in resource "openstack_dns_zone_v2" "zone":
│ 21: resource "openstack_dns_zone_v2" "zone" {
│
╵
╷
│ Error: Error retrieving openstack_dns_zone_v2 1d558d0d-999c-4547-ad9a-e6bcdf125f4e: Expected HTTP response code [200] when accessing [GET https://openstack.codfw1dev.wikimediacloud.org:29001/v2/zones/1d558d0d-999c-4547-ad9a-e6bcdf125f4e], but got 504 instead: {"code": 504, "type": "timeout", "request_id": "req-620ca6cb-69e2-42be-bc99-d385ecfe44c7"}
│
│ with module.project["cloudinfra-codfw1dev"].openstack_dns_zone_v2.zone["svc.codfw1dev.wikimedia.cloud."],
│ on modules/project/dns.tf line 21, in resource "openstack_dns_zone_v2" "zone":
│ 21: resource "openstack_dns_zone_v2" "zone" {EDIT: This is now failing with a different error (see comments below):
root@cloudcontrol2005-dev:/srv/tofu-infra# TF_LOG=1 tofu init [...] 2026-01-07T09:56:14.586Z [DEBUG] backend-s3: HTTP Response Received: aws.region=codfw1dev-r aws.s3.bucket=admin:tofu-state aws.s3.key=repos/cloud/cloud-vps/tofu-infra rpc.method=HeadObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_aws.signing_region="" http.response.header.date="Wed, 07 Jan 2026 09:56:14 GMT" http.response.body="" http.duration=62 http.response.header.x_amz_request_id=tx00000b360a5a3ceb59297-00695e2dbe-123016c-default http.response.header.content_type=application/xml http.response.header.content_security_policy="default-src; font-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'" http.status_code=403 http.response_content_length=210 http.response.header.accept_ranges=bytes 2026-01-07T09:56:14.587Z [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 403, RequestID: tx00000b360a5a3ceb59297-00695e2dbe-123016c-default, HostID: , api error Forbidden: Forbidden: aws.region=codfw1dev-r aws.s3.bucket=admin:tofu-state aws.s3.key=repos/cloud/cloud-vps/tofu-infra rpc.method=HeadObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 Error refreshing state: operation error S3: HeadObject, https response error StatusCode: 403, RequestID: tx00000b360a5a3ceb59297-00695e2dbe-123016c-default, HostID: , api error Forbidden: Forbidden