Openstack: many orphaned (or seemingly orphaned) VMs in codfw1dev
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Andrew
	Nov 19 2024, 5:40 PM

Description

I'm trying to enumerate VMs in codfw1dev and finding many VMs that are associated with projects that can't be found, for example:

k8s-dev
pawsdev
andrewtestproject
f93ed978bc5943c4b9c3fb1bfc70d3b4

Most likely the projects were deleted before being emptied first, in which case we just need to clean up the resources they used to contain. Less likely, something interesting is happening causing me and my sript to be unable to see those projects...

I recall deleting a few projects a few months ago but I don't think I deleted all of these.

Related Objects
Search...

Status	Assigned	Task
Open	None	T53494 Use Beta cluster as a true canary for code deployments (epic)
Declined	None	T87220 Minimize infrastructure differences between Beta Cluster and production
Open	None	T211677 Support IPv6 in beta
Open	None	T404466 Support IPv6-only VMs
Open	None	T392688 Enable IPv6 on Cloud VPS infrastructure services
Resolved	taavi	T379175 Enable IPv6 for the Cloud VPS web proxy
Open	None	T270694 CloudVPS: introduce tenant networks
Resolved	• aborrero	T364725 Migrate Cloud VPS instances to VXLAN based networks
Open	None	T392509 Enable IPv6 for Toolforge services
Resolved	taavi	T392506 Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service
Resolved	taavi	T211575 Enable IPv6 on toolforge.org
Open	None	T220306 Add IPv6 monitoring
Resolved	• aborrero	T37947 Enable IPv6 on CloudVPS
Resolved	• aborrero	T379356 openstack: nova-fullstack: add support for IPv6
Resolved	• aborrero	T380208 openstack: codfw1dev: fullstack tests failing
Resolved	• aborrero	T380303 Openstack: many orphaned (or seemingly orphaned) VMs in codfw1dev

Event Timeline

Andrew created this task.Nov 19 2024, 5:40 PM

This is due to https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/commit/ca8b21897b2e084197b9ea85dbaf17f109661fa8

fnegri subscribed.Nov 19 2024, 6:03 PM

Here are the orphaned VMs:

f93ed978bc5943c4b9c3fb1bfc70d3b4: f950d7b4-ac86-4d12-b19a-096c0d52cd19
f93ed978bc5943c4b9c3fb1bfc70d3b4: 4e67370c-1db0-411a-b2a4-e3107f8a9057
pawsdev: ef794b03-03b8-46f4-9901-d9a7e6beaaa7
pawsdev: b3bbe68d-064b-46a9-b2ea-95eda53292b4
k8s-dev: 9f57d4fb-506d-4c6c-a2a3-566b57abe095
andrewtestproject: 84864e4b-5ac8-4c56-bbc8-3a0dbeaea1cb
taavi-test-project: 7837b970-51ec-404a-8d81-3d71ab01c923
taavi-test-project: 6c7ff713-8af7-4761-a996-4ce0dbf09618

I'm going to clean those up now but likely there are other resources left around (mostly dns domains)

Andrew mentioned this in T380310: opentofu shouldn't delete openstack resources.Nov 19 2024, 6:18 PM

Hmph, 84864e4b-5ac8-4c56-bbc8-3a0dbeaea1cb is wedged in a way that was harmless until andrewtestproject was deleted and now I need to do some database-diving.

Andrew closed this task as Resolved.Nov 20 2024, 3:32 AM

Openstack: many orphaned (or seemingly orphaned) VMs in codfw1devClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Openstack: many orphaned (or seemingly orphaned) VMs in codfw1dev
Closed, ResolvedPublic
Actions

Related Objects
Search...