Page MenuHomePhabricator

[cloud-vps] creating a new project can override existing DNS entries in the wmcloud.org domain
Open, MediumPublicBUG REPORT

Description

This happened with the pixel project that was created last week (T360091).

When a project is created, a DNS zone is automatically created, using the project name (in this case, pixel.wmcloud.org).

There was an existing DNS record pixel.wmcloud.org pointing to a VM in a different project, and that DNS stopped working when the new project was created. The fix was to delete the new DNS zone associated with the new project.

I'm not sure where the existing DNS was configured, but I think it's something that could happen again in the future when we create a new project.

Event Timeline

While we find if there's a better way to prevent this, I've added a note to the project creation steps to check for DNS clashes:
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Projects_lifecycle#Creating_a_new_project

aborrero subscribed.

The proper fix for this is to have different domains for each thing.

Otherwise there will always be potential conflicts.

Example ideas:

  • dedicate wmcloud.org oly for the shared proxy. For projects, use the svc subdomain.
    • example <project>.svc.<deploy>.wmcloud.org, or <project>.svc.wmcloud.org in eqiad1
    • in the proxy we only need to forbid creating entries like *.svc in the name to avoid the resolution hijack
  • figure out a new subdomain for all proxy entries. For example w.<deploy>.wmcloud.org, with the w keyword.
    • create an apex DNS record so that all FQDNs in that subdomain point to the nova-proxy floating IP.
    • we can still hand over <project>.<deploy>.wmcloud.org for each tenant, without conflicts (as long as there is no tenant named w or whatever keyword we decide)
  • including the project name in the proxy FQDN: myproxyname.<project>.eqiad1.wmcloud.org or myproxyname.<project>.wmcloud.org
    • this can create conflicts in how we handle DNS zones and delegation within designate, given zones <project>.wmcloud.org usually belong to each project, while the nova-proxy cannot create records in there.
    • we wont be able to create apex DNS records, precisely because all DNS records in <project>.wmcloud.org belong to each project
  • dedicate wmcloud.org only for the shared proxy. Stop handing over <project>.wmcloud.org for each project, and figure out a completely new domain that we can hand per-project without conflict
    • example wikicloud.org, then hand over <project>.<deploy>.wikicloud.org per project
aborrero renamed this task from [cloud-vps] creating a new project can override existing DNS entries to [cloud-vps] creating a new project can override existing DNS entries in the wmcloud.org domain.Oct 31 2024, 10:35 AM

We knew this could happen when we started delegating the project name related subdomain to the project. It feels like a pretty small problem to me. Treating existing web proxies as a deny list for new Cloud VPS project names does not seem burdensome to me for either the community or WMCS admins.