Page MenuHomePhabricator

Creation of Hiera Puppet Prefix via OpenTofu fails
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

Familiarity with Terraform/OpenTofu is assumed.

Create a new folder and place the following file inside:

main.tf
terraform {
  required_providers {
    cloudvps = {
      source  = "terraform.wmcloud.org/registry/cloudvps"
      version = "~> 0.3"
    }
  }

  required_version = "~> 1.9.0"
}

provider "cloudvps" {
  os_project_id = "account-creation-assistance"
}

resource "cloudvps_puppet_prefix" "testinstances" {
  name = "accounts-testinstance"

  hiera = <<-EOT
    profile::systemd::timesyncd::ntp_servers:
      - ntp.ubuntu.com
  EOT
}

Create a new application credential in Horizon and download the openrc file. Source it into your terminal.

tofu init and tofu plan -out tfplan. You should get a plan output that looks something like this:

  # cloudvps_puppet_prefix.testinstances will be created
  + resource "cloudvps_puppet_prefix" "testinstances" {
      + hiera = <<-EOT
            profile::systemd::timesyncd::ntp_servers:
              - ntp.ubuntu.com
        EOT
      + id    = (known after apply)
      + name  = "accounts-testinstance"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Finally, tofu apply tfplan to apply the changes and trigger the error.

What happens?:

cloudvps_puppet_prefix.testinstances: Creating...
╷
│ Error: Unable to update prefix
│ 
│   with cloudvps_puppet_prefix.testinstances,
│   on testinstances.tf line 64, in resource "cloudvps_puppet_prefix" "testinstances":
│   64: resource "cloudvps_puppet_prefix" "testinstances" {
│ 
│ Unable to update prefix 21866: Expected HTTP response code [200] when
│ accessing [PUT
│ https://puppet-enc.cloudinfra.wmcloud.org/v1/account-creation-assistance/prefix/id/21866],
│ but got 500 instead: <!doctype html>
│ <html lang=en>
│ <title>500 Internal Server Error</title>
│ <h1>Internal Server Error</h1>
│ <p>The server encountered an internal error and was unable to complete your
│ request. Either the server is overloaded or there is an error in the
│ application.</p>
╵
Error: OpenTofu exited with code 1.
Error: Process completed with exit code 1.

The prefix is created in Horizon, but the Hiera configuration isn't applied.

What should have happened instead?:

The provider should have created the Puppet prefix with the Hiera configuration correctly defined. If there is a genuine problem with the Hiera configuration given, then a proper error should be returned.

In either case, since the prefix itself is created, I would expect the resource to be added to state in some form.

Software version:

$ terraform -version
OpenTofu v1.9.1
on linux_amd64
+ provider terraform.wmcloud.org/registry/cloudvps v0.3.0

Other information (browser name/version, screenshots, etc.):

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc [None req-bc681680-3458-47d4-80a2-21b4c5364c69 stwalkerster account-creation-assistance None None default default] Exception on /v1/account-creation-assistance/prefix/id/21873 [PUT]: TypeError: 'NoneType' object is not iterable
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc Traceback (most recent call last):
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc   File "/usr/lib/python3/dist-packages/flask/app.py", line 2525, in wsgi_app
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc     response = self.full_dispatch_request()
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc   File "/usr/lib/python3/dist-packages/flask/app.py", line 1822, in full_dispatch_request
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc     rv = self.handle_user_exception(e)
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc   File "/usr/lib/python3/dist-packages/flask/app.py", line 1820, in full_dispatch_request
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc     rv = self.dispatch_request()
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc   File "/usr/lib/python3/dist-packages/flask/app.py", line 1796, in dispatch_request
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc   File "/usr/lib/python3/dist-packages/flask_keystone/__init__.py", line 338, in wrapped_f
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc     return f(*args, **kwargs)
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc   File "/usr/local/lib/python3.9/dist-packages/puppet-enc.py", line 635, in update_prefix_by_id
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc     to_remove = set(current_roles) - set(roles)
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc TypeError: 'NoneType' object is not iterable
2025-06-28 17:19:09.861 3678954 ERROR uwsgi_file__usr_local_lib_python3_9_dist-packages_puppet-enc

Based on the stack trace, it seems like the provider (or the go-cloudvps library, not sure which one is the problem here) sends "roles": null when roles haven't been set on a prefix, and the API can't handle that. Both of those should be fixed.

Change #1164945 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: puppet-enc: Return helpful error for invalid role data

https://gerrit.wikimedia.org/r/1164945

Change #1164945 merged by Majavah:

[operations/puppet@production] openstack: puppet-enc: Return helpful error for invalid role data

https://gerrit.wikimedia.org/r/1164945

As expected, the above patch changes the API to return a proper error:

cloudvps_puppet_prefix.testinstances: Creating...
╷
│ Error: Unable to update prefix
│ 
│   with cloudvps_puppet_prefix.testinstances,
│   on test.tf line 1, in resource "cloudvps_puppet_prefix" "testinstances":
│    1: resource "cloudvps_puppet_prefix" "testinstances" {
│ 
│ Unable to update prefix 21875: Expected HTTP response code [200] when accessing [PUT https://puppet-enc.cloudinfra.wmcloud.org/v1/metricsinfra/prefix/id/21875], but got 400 instead: {"error":"Provided roles should be a list"}

Still looking at the provider code.

taavi triaged this task as Medium priority.Jun 30 2025, 10:06 AM

Mentioned in SAL (#wikimedia-cloud) [2025-06-30T14:23:19Z] <taavi> publish v0.3.1 of the provider with the fix for T398117

I published v0.3.1 of the provider with this fix included.