Page MenuHomePhabricator
Create Task
Maniphest T398643

[tofu-cloudvps] cloudvps_puppet_prefix.hiera settings show dirty diffs based on YAML canonicalization
Closed, ResolvedPublicBUG REPORT
Actions

Description

@taavi has mentioned this before as a known issue. The Puppet ENC api normalizes YAML input by parsing it into Python native data structures with yaml.safe_load and then creating a canonical stored form by yaml.safe_dump of that native data. This round trip from YAML to Python and then back to YAML can change the quoting of strings and various other minor structural differences in the YAML that do not change the semantic content of the configuration, but do create dirty diffs when comparing input to output.

╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.haproxy.cloudvps_puppet_prefix.haproxy,
│ provider "provider[\"terraform.wmcloud.org/registry/cloudvps\"]" produced an
│ unexpected new value: .hiera: was
│ cty.StringVal("\"profile::zuul::haproxy::kubernetes_hosts\":\n-
│ \"172.16.10.149\"\n"), but now
│ cty.StringVal("profile::zuul::haproxy::kubernetes_hosts:\n    -
│ 172.16.10.149\n").
│
│ This is a bug in the provider, which should be reported in the provider's own
│ issue tracker.
╵

In this particular input case the Tofu side is using Tofu's yamlencode function to produce Tofu canonical YAML. As the slightly obfuscated diff here shows this form is more aggressive with quoting strings and does not indent list elements under a map key. There are likely more small differences that would be revealed by a more complex input.

Can we find a way for the golang to produce the same canonicalized form as Python does so that we can preprocess the input to Tofu and eliminate (or at least greatly reduce) round trip dirty diffs?

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Upgrade cloudvps provider and wrap YAML imports to avoid dirty diffsrepos/releng/zuul/tofu-provisioning!51bd808work/bd808/T398643-yaml-dirty-diffsmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

• bd808 created this task.Jul 3 2025, 3:42 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptJul 3 2025, 3:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• bd808 added a comment.Jul 3 2025, 3:49 PM

One workaround for this that I could imagine is adding a new /v1/yaml (route name is easily debatable) route to the puppet-enc.py service to do the yaml.safe_load → yaml.safe_dump canonicalization. The Tofu could then use this to compute the canonical form of any YAML input for it's internal model.

• bd808 updated the task description. (Show Details)Jul 3 2025, 3:50 PM
taavi added a comment.Jul 3 2025, 3:52 PM

I'm fairly sure the provider actually transforms the YAML string into a JSON object that it sends to the API, and does the same operation in reverse. So the fix here would be to change the provider's JSON object->YAML string conversation to match Tofu's yamlencode(), or allow providing a Tofu object to the provider in the first place instead of forcing it to be converted to YAML in the middle.

• bd808 added a comment.Jul 3 2025, 5:12 PM
In T398643#10973146, @taavi wrote:

I'm fairly sure the provider actually transforms the YAML string into a JSON object that it sends to the API, and does the same operation in reverse.

Ah, that might make things even more complicated. Fun times!

The input is:

hiera = yamlencode({
  "profile::zuul::haproxy::kubernetes_hosts" : [
    for node in var.kubernetes_api_nodes : node
  ]
})

That yamlencode seems to produce YAML that looks like:

"profile::zuul::haproxy::kubernetes_hosts":
- "172.16.10.149"

The stored canonical YAML for my example of the zuul-haproxy- prefix in the zuul project is:

profile::zuul::haproxy::kubernetes_hosts:
- 172.16.10.149

The API response is the canonical form wrapped in a JSON string:

somebody@7635f8bcdb14:/srv/app$ OS_TOKEN=$(openstack token issue -f yaml | yq .id)
somebody@7635f8bcdb14:/srv/app$ curl -H "Accept: application/json" -H "X-Auth-Token: $OS_TOKEN" https://puppet-enc.cloudinfra.wmcloud.org/v1/c26d9d326bdf464fa1025939ded7e5a2/prefix/zuul-haproxy-/hiera
{"hiera":"profile::zuul::haproxy::kubernetes_hosts:\n- 172.16.10.149\n"}

The YAML causing the dirty diff is:

profile::zuul::haproxy::kubernetes_hosts:
    - 172.16.10.149

This is not the same YAML as the yaml.safe_dump canonical form returned by the API or the input. It appears to be the result of yaml.Unmarshal([]byte(m.Hiera.ValueString()), &data) in hasEqualHiera as @taavi notes. The YAML package in use there is gopkg.in/yaml.v3. OpenTofu gets its yamlencode from the YAMLEncodeFunc of the github.com/zclconf/go-cty-yaml library.

• bd808 added a comment.Jul 3 2025, 5:48 PM
In T398643#10973350, @bd808 wrote:

This is not the same YAML as the yaml.safe_dump canonical form returned by the API or the input. It appears to be the result of yaml.Unmarshal([]byte(m.Hiera.ValueString()), &data) in hasEqualHiera as @taavi notes. The YAML package in use there is gopkg.in/yaml.v3. OpenTofu gets its yamlencode from the YAMLEncodeFunc of the github.com/zclconf/go-cty-yaml library.

Wait.. that's not right. Unmarshal would be turning a string into a parsed data structure. m.Hiera should be the yamlencode output string. The yaml.Unmarshal call is turning that into a map[string]any structure. This is later compared against the map[string]any structure returned from parsing the API's JSON response.

Where does the YAML encoded string from the dirty diff report come from? I think that is actually the data, err := yaml.Marshal(prefix.Hiera) from PuppetPrefixResourceModel.readFrom.

Now I'm confused again though because it seems that PuppetPrefixResourceModel.hasEqualHiera is already comparing gloang structures rather than strings which should avoid dirty diff issues in the YAML representation.

CodeReviewBot added a project: Patch-For-Review.Jul 8 2025, 3:24 PM

bd808 opened https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/17

kubernetes: build a v1.28 cluster

CodeReviewBot added a comment.Jul 8 2025, 5:05 PM

bd808 merged https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/17

kubernetes: build a v1.28 cluster

Maintenance_bot removed a project: Patch-For-Review.Jul 8 2025, 5:31 PM
• joanna_borun triaged this task as Medium priority.Jul 9 2025, 2:06 PM
CodeReviewBot added a project: Patch-For-Review.Jul 11 2025, 12:26 AM

bd808 opened https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/27

bastion: Match expected YAML formatting

CodeReviewBot added a comment.Jul 11 2025, 12:27 AM

bd808 merged https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/27

bastion: Match expected YAML formatting

• bd808 removed a project: Patch-For-Review.Jul 11 2025, 12:30 AM
• bd808 added a comment.Jul 17 2025, 9:37 PM

https://developer.hashicorp.com/terraform/plugin/sdkv2/best-practices/detecting-drift might be helpful when working on this issue.

• bd808 added a parent task: T396936: Provision Kubernetes cluster and bastion using OpenTofu and Magnum.Jul 17 2025, 9:38 PM
• bd808 mentioned this in T399882: Flannel networking broken in Magnum cluster because upstream containers are missing.Jul 17 2025, 11:41 PM
• bd808 added a comment.Aug 6 2025, 10:20 PM
In T398643#11015358, @bd808 wrote:

https://developer.hashicorp.com/terraform/plugin/sdkv2/best-practices/detecting-drift might be helpful when working on this issue.

The more modern plugin framework version of this is apparently implementing a custom type and semantic equality handling. https://github.com/hashicorp/terraform-plugin-framework-jsontypes/blob/b723dad5906df021c15a0ae6968a173480303739/jsontypes/normalized_value.go#L52 is an example of such a custom type with a semantic equals method for JSON strings.

• bd808 added a comment.Aug 6 2025, 11:57 PM

I spent time over the last two days poking at this which mostly means I spent time figuring out how to compile the provider and use a locally modified version in my tofu runtime. I made reasonable progress on that initial goal and have a workflow where I edit things in terraform-cloudvps, compile the terraform-provider-cloudvps binary, copy that binary to another directory, and run tofu with a configuration override to load the custom binary. I have used this workflow to add a lot of debug logging in internal/provider/puppet_prefix_resource.go that uses spew.Sdump() to log internal data structures. So far all good.

Now the "interesting" thing is that I no longer seem to be able to recreate the validation error. Both the released 0.3.1 provider and my locally modified version seem to be giving tofu the right signals to make tofu understand that the YAML strings are semantically equivalent. This is very confusing as I am unsure what changed to make this true. I am seeing logs like this now that show the internal logic at work:

2025-08-06T23:30:31.937Z [DEBUG] provider.terraform-provider-cloudvps: Value switched to prior value due to semantic equality logic: @caller=/Users/bd808/projects/wmf/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.13.0/internal/fwschemadata/value_semantic_equality.go:91 tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_req_id=cb153c4c-bb7d-9e5e-6d5e-65684903fda1 tf_resource_type=cloudvps_puppet_prefix @module=sdk.framework tf_attribute_path=hiera tf_rpc=ReadResource timestamp=2025-08-06T23:30:31.937Z
• bd808 added a comment.Aug 7 2025, 11:19 PM

I can haz a reproduction!

│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.puppetserver.cloudvps_puppet_prefix.bd808-T398643, provider "provider[\"terraform.wmcloud.org/registry/cloudvps\"]" produced an unexpected new value: .hiera: was cty.StringVal("\"bd808\": \"wuz here\"\n"), but now cty.StringVal("bd808: wuz here\n").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

I was able to reproduce by adding this tofu config:

diff --git i/modules/puppetserver/main.tf w/modules/puppetserver/main.tf
index 0ecb3d2..372067a 100644
--- i/modules/puppetserver/main.tf
+++ w/modules/puppetserver/main.tf
@@ -96,3 +96,11 @@ resource "cloudvps_puppet_prefix" "project" {
     "puppetmaster" : "${openstack_compute_instance_v2.puppetserver[var.active_puppetserver].name}.zuul.eqiad1.wikimedia.cloud"
   })
 }
+
+resource "cloudvps_puppet_prefix" "bd808-T398643" {
+  name = "zuul-bd808-T398643-"
+  roles = []
+  hiera = yamlencode({
+    "bd808": "wuz here",
+  })
+}

This indicates that the problematic action is the initial creation of a prefix object. Because of quirks of the upstream API a prefix creation is implemented as a create followed by an update to set the roles and hiera members.

Some additional logs from the failed run and my hacked up provider:

2025-08-07T21:28:51.346Z [TRACE] provider.terraform-provider-cloudvps: Calling provider defined Resource Create: tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 tf_rpc=ApplyResourceChange @caller=/Users/bd808/projects/wmf/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.13.0/internal/fwserver/server_createresource.go:100 @module=sdk.framework tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_resource_type=cloudvps_puppet_prefix timestamp=2025-08-07T21:28:51.345Z
2025-08-07T21:28:51.350Z [DEBUG] provider.terraform-provider-cloudvps: New prefix created:
  hiera=
  | (map[string]interface {}) (len=1) {
  |  (string) (len=5) "bd808": (string) (len=8) "wuz here"
  | }
   tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 name=zuul-bd808-T398643-
  roles=
  | ([]string) <nil>
   @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:74 @module=cloudvps id=0 tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_resource_type=cloudvps_puppet_prefix tf_rpc=ApplyResourceChange timestamp=2025-08-07T21:28:51.350Z
2025-08-07T21:28:51.737Z [DEBUG] provider.terraform-provider-cloudvps: PuppetPrefixResourceModel.readFrom() input: @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:117 @module=cloudvps
  prefix=
  | (*puppet.Prefix)(0xc000116500)({
  |  Id: (int) 22071,
  |  Name: (string) (len=19) "zuul-bd808-T398643-",
  |  Roles: ([]string) {
  |  },
  |  Hiera: (map[string]interface {}) {
  |  }
  | })
   tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_resource_type=cloudvps_puppet_prefix tf_rpc=ApplyResourceChange
  m=
  | (*provider.PuppetPrefixResourceModel)(0xc000304380)({
  |  Id: (basetypes.Int64Value) <unknown>,
  |  Name: (basetypes.StringValue) "zuul-bd808-T398643-",
  |  Roles: (basetypes.ListValue) [],
  |  Hiera: (basetypes.StringValue) "\"bd808\": \"wuz here\"\n"
  | })
   tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 timestamp=2025-08-07T21:28:51.737Z
2025-08-07T21:28:51.737Z [TRACE] provider.terraform-provider-cloudvps: prefix.Hiera != nil: tf_rpc=ApplyResourceChange @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:134 @module=cloudvps tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_resource_type=cloudvps_puppet_prefix timestamp=2025-08-07T21:28:51.737Z
2025-08-07T21:28:51.739Z [DEBUG] provider.terraform-provider-cloudvps: PuppetPrefixResourceModel.hasEqualHiera():
  m=
  | (*provider.PuppetPrefixResourceModel)(0xc000304380)({
  |  Id: (basetypes.Int64Value) 22071,
  |  Name: (basetypes.StringValue) "zuul-bd808-T398643-",
  |  Roles: (basetypes.ListValue) [],
  |  Hiera: (basetypes.StringValue) "\"bd808\": \"wuz here\"\n"
  | })
  
  prefix=
  | (*puppet.Prefix)(0xc000116500)({
  |  Id: (int) 22071,
  |  Name: (string) (len=19) "zuul-bd808-T398643-",
  |  Roles: ([]string) {
  |  },
  |  Hiera: (map[string]interface {}) {
  |  }
  | })
   tf_resource_type=cloudvps_puppet_prefix @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:90 tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 tf_rpc=ApplyResourceChange @module=cloudvps tf_provider_addr=terraform.wmcloud.org/registry/cloudvps timestamp=2025-08-07T21:28:51.739Z
2025-08-07T21:28:51.740Z [DEBUG] provider.terraform-provider-cloudvps: comparing structs using reflect.DeepEqual: tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 tf_rpc=ApplyResourceChange @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:109 @module=cloudvps tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_resource_type=cloudvps_puppet_prefix
  from-ENC=
  | (map[string]interface {}) {
  | }
  
  from-Tofu=
  | (map[string]interface {}) (len=1) {
  |  (string) (len=5) "bd808": (string) (len=8) "wuz here"
  | }
   timestamp=2025-08-07T21:28:51.739Z
2025-08-07T21:28:51.740Z [DEBUG] provider.terraform-provider-cloudvps: PuppetPrefixResourceModel.readFrom() output: @module=cloudvps tf_provider_addr=terraform.wmcloud.org/registry/cloudvps
  m=
  | (*provider.PuppetPrefixResourceModel)(0xc000304380)({
  |  Id: (basetypes.Int64Value) 22071,
  |  Name: (basetypes.StringValue) "zuul-bd808-T398643-",
  |  Roles: (basetypes.ListValue) [],
  |  Hiera: (basetypes.StringValue) ""
  | })
   tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 tf_resource_type=cloudvps_puppet_prefix tf_rpc=ApplyResourceChange @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:169 timestamp=2025-08-07T21:28:51.740Z
2025-08-07T21:28:51.740Z [DEBUG] provider.terraform-provider-cloudvps: Created a prefix:
  prefix=
  | (*provider.PuppetPrefixResourceModel)(0xc000304380)({
  |  Id: (basetypes.Int64Value) 22071,
  |  Name: (basetypes.StringValue) "zuul-bd808-T398643-",
  |  Roles: (basetypes.ListValue) [],
  |  Hiera: (basetypes.StringValue) ""
  | })
   tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 tf_resource_type=cloudvps_puppet_prefix @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:338 tf_rpc=ApplyResourceChange @module=cloudvps timestamp=2025-08-07T21:28:51.740Z
2025-08-07T21:28:52.519Z [DEBUG] provider.terraform-provider-cloudvps: PuppetPrefixResourceModel.readFrom() input:
  m=
  | (*provider.PuppetPrefixResourceModel)(0xc000304380)({
  |  Id: (basetypes.Int64Value) 22071,
  |  Name: (basetypes.StringValue) "zuul-bd808-T398643-",
  |  Roles: (basetypes.ListValue) [],
  |  Hiera: (basetypes.StringValue) ""
  | })
   tf_rpc=ApplyResourceChange @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:117 @module=cloudvps
  prefix=
  | (*puppet.Prefix)(0xc0005a0840)({
  |  Id: (int) 22071,
  |  Name: (string) (len=19) "zuul-bd808-T398643-",
  |  Roles: ([]string) {
  |  },
  |  Hiera: (map[string]interface {}) (len=1) {
  |   (string) (len=5) "bd808": (string) (len=8) "wuz here"
  |  }
  | })
   tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 tf_resource_type=cloudvps_puppet_prefix timestamp=2025-08-07T21:28:52.519Z
2025-08-07T21:28:52.522Z [DEBUG] provider.terraform-provider-cloudvps: PuppetPrefixResourceModel.hasEqualHiera(): @module=cloudvps tf_req_id=c431226c-6019-5e88-8c96-dace21127f53
  prefix=
  | (*puppet.Prefix)(0xc0005a0840)({
  |  Id: (int) 22071,
  |  Name: (string) (len=19) "zuul-bd808-T398643-",
  |  Roles: ([]string) {
  |  },
  |  Hiera: (map[string]interface {}) (len=1) {
  |   (string) (len=5) "bd808": (string) (len=8) "wuz here"
  |  }
  | })
   tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_resource_type=cloudvps_puppet_prefix tf_rpc=ApplyResourceChange @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:90
  m=
  | (*provider.PuppetPrefixResourceModel)(0xc000304380)({
  |  Id: (basetypes.Int64Value) 22071,
  |  Name: (basetypes.StringValue) "zuul-bd808-T398643-",
  |  Roles: (basetypes.ListValue) [],
  |  Hiera: (basetypes.StringValue) ""
  | })
   timestamp=2025-08-07T21:28:52.522Z
2025-08-07T21:28:52.522Z [DEBUG] provider.terraform-provider-cloudvps: comparing structs using reflect.DeepEqual:
  from-Tofu=
  | (map[string]interface {}) {
  | }
   @module=cloudvps
  from-ENC=
  | (map[string]interface {}) (len=1) {
  |  (string) (len=5) "bd808": (string) (len=8) "wuz here"
  | }
   tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 tf_resource_type=cloudvps_puppet_prefix @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:109 tf_rpc=ApplyResourceChange timestamp=2025-08-07T21:28:52.522Z
2025-08-07T21:28:52.522Z [DEBUG] provider.terraform-provider-cloudvps: PuppetPrefixResourceModel.readFrom() output: @module=cloudvps
  m=
  | (*provider.PuppetPrefixResourceModel)(0xc000304380)({
  |  Id: (basetypes.Int64Value) 22071,
  |  Name: (basetypes.StringValue) "zuul-bd808-T398643-",
  |  Roles: (basetypes.ListValue) [],
  |  Hiera: (basetypes.StringValue) "bd808: wuz here\n"
  | })
   tf_resource_type=cloudvps_puppet_prefix tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 tf_rpc=ApplyResourceChange @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:169 timestamp=2025-08-07T21:28:52.522Z
2025-08-07T21:28:52.522Z [DEBUG] provider.terraform-provider-cloudvps: Updated prefix: tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 @module=cloudvps
  prefix=
  | (*provider.PuppetPrefixResourceModel)(0xc000304380)({
  |  Id: (basetypes.Int64Value) 22071,
  |  Name: (basetypes.StringValue) "zuul-bd808-T398643-",
  |  Roles: (basetypes.ListValue) [],
  |  Hiera: (basetypes.StringValue) "bd808: wuz here\n"
  | })
   tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_resource_type=cloudvps_puppet_prefix tf_rpc=ApplyResourceChange @caller=/Users/bd808/projects/wmf/cloud/cloud-vps/terraform-cloudvps/internal/provider/puppet_prefix_resource.go:372 timestamp=2025-08-07T21:28:52.522Z
2025-08-07T21:28:52.523Z [TRACE] provider.terraform-provider-cloudvps: Called provider defined Resource Create: @module=sdk.framework tf_provider_addr=terraform.wmcloud.org/registry/cloudvps tf_req_id=c431226c-6019-5e88-8c96-dace21127f53 @caller=/Users/bd808/projects/wmf/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.13.0/internal/fwserver/server_createresource.go:102 tf_rpc=ApplyResourceChange tf_resource_type=cloudvps_puppet_prefix timestamp=2025-08-07T21:28:52.522Z
2025-08-07T21:28:52.527Z [TRACE] maybeTainted: module.puppetserver.cloudvps_puppet_prefix.bd808-T398643 encountered an error during creation, so it is now marked as tainted

Weirdly I am not seeing any log line explaining where the error during creation happened.

• bd808 added a comment.Aug 7 2025, 11:35 PM

Once a tainted cloudvps_puppet_prefix node is in the state all subsequent tofu plan stages want to replace the tainted node via destroy + create cycle. Since the create step reproduces the taint issue the next plan will seek to do the same thing.

Removing the taint with tofu untaint module.puppetserver.cloudvps_puppet_prefix.bd808-T398643 once seems to be enough to break the cycle. The next plan following the taint removal calls for an update rather than a replacement. When applied that change fixes the tracked state such that it no longer shows a diff on subsequent runs.

• bd808 mentioned this in T401839: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie.Aug 13 2025, 3:57 PM
• bd808 added a comment.Aug 13 2025, 8:18 PM

I tried to get delve setup so I could set a breakpoint at node_resource_apply_instance.go:533. I ended up not having enough patience to get that to work, so instead I compiled a tofu binary that added some logging there.

diff --git i/internal/tofu/node_resource_apply_instance.go w/internal/tofu/node_resource_apply_instance.go
index 1045fcd71c..c395032370 100644
--- i/internal/tofu/node_resource_apply_instance.go
+++ w/internal/tofu/node_resource_apply_instance.go
@@ -22,6 +22,8 @@ import (
     "github.com/opentofu/opentofu/internal/states"
     "github.com/opentofu/opentofu/internal/tfdiags"
     "github.com/opentofu/opentofu/internal/tracing"
+
+    "github.com/davecgh/go-spew/spew"
 )

 // NodeApplyableResourceInstance represents a resource instance that is
@@ -534,6 +536,7 @@ func maybeTainted(addr addrs.AbsResourceInstance, state *states.ResourceInstance
     if state == nil || change == nil || err == nil {
         return state
     }
+    log.Printf("[bd808] maybeTainted: %s err=%s", addr, spew.Sdump(err))
     if state.Status == states.ObjectTainted {
         log.Printf("[TRACE] maybeTainted: %s was already tainted, so nothing to do", addr)
         return state
2025-08-13T20:02:13.707Z [INFO]  [bd808] maybeTainted: module.puppetserver.cloudvps_puppet_prefix.bd808-T398643 err=(tfdiags.diagnosticsAsError) Provider produced inconsistent result after apply: When applying changes to module.puppetserver.cloudvps_puppet_prefix.bd808-T398643, provider "provider[\"tofu.wmcloud.org/registry/cloudvps\"]" produced an unexpected new value: .hiera: was cty.StringVal("\"bd808\": \"wuz here\"\n"), but now cty.StringVal("bd808: wuz here\n").                                                                                        This is a bug in the provider, which should be reported in the provider's own issue tracker.                                                                    2025-08-13T20:02:13.707Z [TRACE] maybeTainted: module.puppetserver.cloudvps_puppet_prefix.bd808-T398643 encountered an error during creation, so it is now marked as tainted

I sure spent a long time to get all the way back to the original error message. I did learn a few things along the way I think.

• bd808 added a comment.Aug 26 2025, 3:52 AM
In T398643#10973146, @taavi wrote:

I'm fairly sure the provider actually transforms the YAML string into a JSON object that it sends to the API, and does the same operation in reverse. So the fix here would be to change the provider's JSON object->YAML string conversation to match Tofu's yamlencode(), or allow providing a Tofu object to the provider in the first place instead of forcing it to be converted to YAML in the middle.

I have a locally modified terraform-cloudvps provider that is cleanly round tripping by converting the API client's object response to YAML via the function same yamlencode function as Tofu. When loading data from a YAML file dirty diffs can be avoided by either encoding exactly as yamlencode or more simply by using a call chain like yamlencode(yamldecode(file(...))). I will need to spend a bit more time cleaning the patch up. I am pretty sure I went overboard on logging as I was stumbling through understanding the problem.

• bd808 changed the task status from Open to In Progress.Aug 26 2025, 3:53 AM
• bd808 claimed this task.
Restricted Application added a project: User-bd808. · View Herald TranscriptAug 26 2025, 3:53 AM
CodeReviewBot added a project: Patch-For-Review.Aug 26 2025, 6:12 PM

bd808 opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/terraform-cloudvps/-/merge_requests/10

puppet_prefix: Generate YAML with yamlencode equivalent

CodeReviewBot added a comment.Aug 28 2025, 12:19 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/10

puppet_prefix: Generate YAML with yamlencode equivalent

CodeReviewBot added a comment.Aug 28 2025, 8:54 PM

bd808 opened https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/51

Draft: hiera: Wrap YAML file imports to avoid dirty diffs

fnegri edited projects, added: cloud-services-team (FY2025/2026-Q1-Q2); removed: cloud-services-team.Sep 5 2025, 8:40 AM
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2025/2026-Q1-Q2) board.
• bd808 added a comment.Sep 9 2025, 11:46 PM

@taavi Can I help you prepare a new release of tofu-cloudvps so that I can call this {{done}}?

taavi added a comment.Sep 10 2025, 3:47 PM

{{done}} I just tagged and published a 0.4.0 release.

CodeReviewBot added a comment.Sep 10 2025, 8:16 PM

bd808 merged https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/51

Upgrade cloudvps provider and wrap YAML imports to avoid dirty diffs

• bd808 closed this task as Resolved.Sep 10 2025, 9:14 PM
fnegri moved this task from In progress to Done on the cloud-services-team (FY2025/2026-Q1-Q2) board.Sep 22 2025, 9:01 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits