CloudVPS: figure out DNS zone ownership transfers and setup
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aborrero
	Feb 18 2020, 11:05 AM

Description

According to our docs https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS DNS zones should belong to the cloudinfra project. I tried a couple mechanism for the initial setup of the zones but I'm not convinced:

Create zone in the admin project using the cmdline, then manually update the database to set the owner project for the domain

This was how wmcloud.org was set until today. @Krenair discovered the zone lacked the SOA records, thus an invalid zone that I finally deleted to recreate.
Not sure what happened here, but a workflow requiring manual updates to the database is far from elegant.

Create zone in the admin project, using the cmdline. Then try using the openstack zone transfer native mechanism for updating the zone ownership.

I couldn't figure out how this work. The 2 commands below resulted in the zone still being owned by the admin project.

openstack zone transfer request create --target-project-id cloudinfra 6ddcb082-69d6-43f4-9993-5c6bdc27dfc9
openstack zone transfer accept request --transfer-id  cb12641f-7cdc-4b0f-85c6-5ecaae43bd3b --key XXXXXX --sudo-project-id cloudinfra

This task is about reviewing this workflow and documenting it in wikitech, probably here:
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Designate

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		aborrero	T245173 CloudVPS: DNS improvements
		Resolved		aborrero	T245494 CloudVPS: figure out DNS zone ownership transfers and setup

Event Timeline

aborrero created this task.Feb 18 2020, 11:05 AM

I just did this for eqiad1.wikimedia.cloud. and it seemed to work, at least this once

# openstack zone transfer request create --target-project-id cloudinfra 67603ef4-3d64-40d6-90d3-5b7776a99034
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                               |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| created_at        | 2020-02-18T15:35:43.000000                                                                                                          |
| description       | None                                                                                                                                |
| id                | 0a30d1fe-51ea-4da4-8a27-80c16e681ff6                                                                                                |
| key               | LOYRJY7Q                                                                                                                            |
| links             | {u'self': u'http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/tasks/transfer_requests/0a30d1fe-51ea-4da4-8a27-80c16e681ff6'} |
| project_id        | admin                                                                                                                               |
| status            | ACTIVE                                                                                                                              |
| target_project_id | cloudinfra                                                                                                                          |
| updated_at        | None                                                                                                                                |
| zone_id           | 67603ef4-3d64-40d6-90d3-5b7776a99034                                                                                                |
| zone_name         | eqiad1.wikimedia.cloud.                                                                                                             |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------+

# OS_PROJECT_ID=cloudinfra openstack zone transfer accept request --key LOYRJY7Q --transfer-id 0a30d1fe-51ea-4da4-8a27-80c16e681ff6 
+--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                    | Value                                                                                                                                                                                                                                     |
+--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at               | 2020-02-18T15:38:17.000000                                                                                                                                                                                                                |
| id                       | f8077a78-9f0d-4cbc-9ac7-59873f5657fa                                                                                                                                                                                                      |
| key                      | LOYRJY7Q                                                                                                                                                                                                                                  |
| links                    | {u'self': u'http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/tasks/transfer_accepts/f8077a78-9f0d-4cbc-9ac7-59873f5657fa', u'zone': u'http://cloudservices1003.wikimedia.org:9001/v2/zones/67603ef4-3d64-40d6-90d3-5b7776a99034'} |
| project_id               | cloudinfra                                                                                                                                                                                                                                |
| status                   | COMPLETE                                                                                                                                                                                                                                  |
| updated_at               | 2020-02-18T15:38:17.000000                                                                                                                                                                                                                |
| zone_id                  | 67603ef4-3d64-40d6-90d3-5b7776a99034                                                                                                                                                                                                      |
| zone_transfer_request_id | 0a30d1fe-51ea-4da4-8a27-80c16e681ff6                                                                                                                                                                                                      |
+--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

# OS_PROJECT_ID=cloudinfra openstack zone show 67603ef4-3d64-40d6-90d3-5b7776a99034
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| action         | NONE                                 |
| attributes     | {}                                   |
| created_at     | 2019-10-18T15:57:51.000000           |
| description    | None                                 |
| email          | root@wmflabs.org                     |
| id             | 67603ef4-3d64-40d6-90d3-5b7776a99034 |
| masters        |                                      |
| name           | eqiad1.wikimedia.cloud.              |
| pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
| project_id     | cloudinfra                           |
| serial         | 1581676303                           |
| status         | ACTIVE                               |
| transferred_at | None                                 |
| ttl            | 3600                                 |
| type           | PRIMARY                              |
| updated_at     | 2020-02-18T15:38:17.000000           |
| version        | 6                                    |
+----------------+--------------------------------------+

I remember trying with either --sudo-project-id <project> and --os-project-id <project> in the cmdline with no luck.

Will try again tomorrow with your exact same commands!

I just now moved wmcloud.org to cloud-infra and tools.wmcloud.org to tools using the above process. Seems ok.

Ok, thanks for dealing with this.

I just updated the docs at https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Designate#DNS_zone_creating_/_transfers feel free to update it if required.

Closing task now, thanks again!

Please can someone sort out the codfw1dev.wmcloud.org. zone in cloudinfra-codfw1dev? It's broken in the same way wmcloud.org was and I don't have the kind of access needed to fix this anymore. I need it for T242607: Create in-cloud puppetmaster for codfw1dev (labs cloud-wide puppetmasters need a public IP to be exposed so horizon can talk to encapi, and also so designate can SSH in to clean up certs of deleted instances).
Also while we're here, shouldn't there be a cloudinfra-codfw1dev.codfw1dev.wmcloud.org. zone?

I updated all of the recordsets under codfw1dev.wmcloud.org. to belong to cloudinfra-codfw1dev -- I'm pretty sure that's all that was broken but lmk if there are other issues.

I also created cloudinfra-codfw1dev.codfw1dev.wmcloud.org. and fixed up wmcs-makedomain to work in codfw1dev.

codfw1dev.wmcloud.org. is still broken as I can't seem to create records under it - now I get HTTP 500s :(
I've also noticed codfw1dev.wikimedia.cloud. is lacking NS/SOA records like this one - presumably another record ownership problem from a manual database update?
I have successfully created my puppetmaster.cloudinfra-codfw1dev.codfw1dev.wmcloud.org. record in designate though it doesn't show up to external DNS queries, possibly because of lack of NS records for cloudinfra-codfw1dev under codfw1dev.wmcloud.org?
Edit: Nope that's not it, in eqiad1 the wmflabs.org domain has no such requirement (to make NS records for subdomains, presumably because they're all on the same DNS servers)

In T245494#5909824, @Krenair wrote:

codfw1dev.wmcloud.org. is still broken as I can't seem to create records under it - now I get HTTP 500s :(

Me too. I deleted that domain (and subdomains) and recreated it. It's working for me now, with luck it will work for you as well. I'm pretty sure I didn't break anything that was previously working by wiping it out and recreating.

I've also noticed codfw1dev.wikimedia.cloud. is lacking NS/SOA records like this one - presumably another record ownership problem from a manual database update?

This should be fixed (via mysql hacking)

I have successfully created my puppetmaster.cloudinfra-codfw1dev.codfw1dev.wmcloud.org. record in designate though it doesn't show up to external DNS queries, possibly because of lack of NS records for cloudinfra-codfw1dev under codfw1dev.wmcloud.org?

Maybe xfr is laggy there? It seems to work now:

labtestandrew@test-instance-99:~$ dig +short puppetmaster.cloudinfra-codfw1dev.codfw1dev.wmcloud.org.
172.16.128.20

Edit: Nope that's not it, in eqiad1 the wmflabs.org domain has no such requirement (to make NS records for subdomains, presumably because they're all on the same DNS servers)

In T245494#5913633, @Andrew wrote:

In T245494#5909824, @Krenair wrote:

codfw1dev.wmcloud.org. is still broken as I can't seem to create records under it - now I get HTTP 500s :(

Me too. I deleted that domain (and subdomains) and recreated it. It's working for me now, with luck it will work for you as well. I'm pretty sure I didn't break anything that was previously working by wiping it out and recreating.

Works now, thanks.

I've also noticed codfw1dev.wikimedia.cloud. is lacking NS/SOA records like this one - presumably another record ownership problem from a manual database update?

This should be fixed (via mysql hacking)

Now I get 500s for trying to make test records under that. It's not a zone that I need to modify right now but having broken zones in use is unideal

I have successfully created my puppetmaster.cloudinfra-codfw1dev.codfw1dev.wmcloud.org. record in designate though it doesn't show up to external DNS queries, possibly because of lack of NS records for cloudinfra-codfw1dev under codfw1dev.wmcloud.org?

Maybe xfr is laggy there? It seems to work now:
labtestandrew@test-instance-99:~$ dig +short puppetmaster.cloudinfra-codfw1dev.codfw1dev.wmcloud.org.
172.16.128.20

Wfm too now, thanks.

Now I get 500s for trying to make test records under that. It's not a zone that I need to modify right now but having
broken zones in use is unideal

I was able to reproduce this, and it now looks fixed to me. The issue was that I changed the ownership of the SOA and NS recordsets but did not change the ownership of the associated records (which caused a crash when the designate code tried to dereference an empty record array).

The main lesson here is -- trying to move domains via sql is difficult and bad. Please lmk if you hit any more of these, they should be fixable in roughly the same way.

• Bstorm mentioned this in T211096: PAWS: Rebuild and upgrade Kubernetes.May 21 2020, 12:02 AM

CloudVPS: figure out DNS zone ownership transfers and setupClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

CloudVPS: figure out DNS zone ownership transfers and setup
Closed, ResolvedPublic
Actions

Related Objects
Search...