For the past few times when we've created a Cloud VPS project via the cookbook, the creation failed which left tofu in a state that needed manual fixing. This raises several questions:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Keystone hooks: speed up domain creation | operations/puppet | production | +9 -5 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T398712 Cloud VPS project creation cookbook times out really often | |||
| Resolved | fnegri | T401161 SystemdUnitDown The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. |
I'm looking for a few examples of this on logstash.
On July 4, The keystone hooks for the project 'wikidata-deleted' took 2:09 with the majority of the time spent in DNS domain creation
On July 2, 'dumpstorrents' 2:23, again most time spent (more than 2 minutes) in domain creation
On June 12, 'zuul', 1:32
Meanwhile, projects created by magnum (which don't get dns entries) take less than 2 seconds for creation.
So, if the slowdown is happening during the actual project creation stage, domain creation (or more likely, domain transfer between projects) is the slow bit.
It's also possible that what looks like a timeout during creation is actually a timeout during the addition of an initial user to a project (which would be quite slow if the user is being added to the bastion project for the first time.)
@taavi, are you seeing delays of more than 2 or so minutes? And do you happen to know which stage of the cookbook is failing? And is the timeout happening in the API call, or happening because tofu is impatient?
Yes.
And do you happen to know which stage of the cookbook is failing? And is the timeout happening in the API call, or happening because tofu is impatient?
The Tofu run fails to create the project because HAProxy returns a HTTP 504 (iirc) error for the create project keystone API call.
Change #1182188 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Keystone hooks: speed up domain creation
Change #1182188 merged by Andrew Bogott:
[operations/puppet@production] Keystone hooks: speed up domain creation