Page MenuHomePhabricator

Designate should support split horizon resolution to yield private IP of instances behind a public DNS entry
Closed, ResolvedPublic

Description

Instances of the Beta Cluster and the Continuous Integration projects, have to resolve public DNS records such as en.wikipedia.beta.wmflabs.org . By default dnsmasq (and Designate) yield the public IP which is not reacheable by instances due to NAT.

To workaround it, Brandon Black added aliases in dnsmasq which causes it to yield the internal private IP when the entries are queried from labs instance. This way host resolving the public DNS entry ends up with the private IP.

The related puppet configuration is in operations/puppet.git modules/openstack/manifests/nova/network.pp

$nova_dnsmasq_aliases = {
    # eqiad
    'deployment-cache-text02'   => {public_ip  => '208.80.155.135',
                                    private_ip => '10.68.16.16' },
    'deployment-cache-upload02' => {public_ip  => '208.80.155.136',
                                    private_ip => '10.68.17.51' },
    'deployment-cache-bits01'   => {public_ip  => '208.80.155.137',
                                    private_ip => '10.68.16.12' },
    'deployment-stream'         => {public_ip  => '208.80.155.138',
                                    private_ip => '10.68.17.106' },
    'deployment-cache-mobile03' => {public_ip  => '208.80.155.139',
                                    private_ip => '10.68.16.13' },
    'relic'                     => {public_ip  => '208.80.155.197',
                                    private_ip => '10.68.16.162' },
    'tools-webproxy'            => {public_ip  => '208.80.155.131',
                                    private_ip => '10.68.17.145' },
    'udplog'                    => {public_ip  => '208.80.155.191',
                                    private_ip => '10.68.16.58' },

    # A wide variety of hosts are reachable via a public web proxy.
    'labs_shared_proxy' => {public_ip  => '208.80.155.156',
                            private_ip => '10.68.16.65'},
}

Both the Beta cluster and the Continuous integration projects require that functionality. That is hardcoded in dnsmasq but maybe Designate natively supports split horizon (ie yield different results based on client).

Event Timeline

hashar raised the priority of this task from to High.
hashar updated the task description. (Show Details)
hashar added subscribers: gerritbot, Aklapper, Andrew, hashar.

This is a showstopper for Toolforge as well; many tools connect to the proxies from within Labs.

It seems Designate supports PowerDNS and BIND, both support DNS split horizon. From http://designate.readthedocs.org/en/latest/architecture.html#dns-backend

DNS Backend

Backends are drivers for a particular DNS server. Designate supports multiple backend implementations, PowerDNS, BIND and MySQL BIND, you are also free to implement your own backend to fit your needs, as well as extensions to provide extra functionality to complement existing backends.

Before I dive headlong into the pdns config docs... can we just accomplish this by puppetizing /etc/hosts and adding local ips for .wfmlabs.org hosts? That's easy, and would be less obscure than doing it in the dns server as well.

We can but that's totally terrible :) Also how will we generate them? We would need an entry for every public IP's domain, and change them when that changes, and then if puppet is broken those will be broken too...

DNS masq aliases work on the IP addresses regardless of the DNS entry being queried. For tools webproxy, there is a large chunk of *.wmflabs.org entries created by users, Beta has wildcard DNS entries such as *.wikipedia.beta.wmflabs.org and I don't think /etc/hosts support that.

I'll investigate a proper way to do split horizon dns with the pdns/designate combo

Coren has, um, volunteered to make this happen in the pdns.

As to whether this happens in pdns/ldap (the old server) or pdns/mysql (the new one) I'm unclear. Ideally everything would be configured in the new server via the designate API so that designate knows what's happening.

I'm going to solve this by setting up a labs-specific recursor which will swizzle IPs the way dnsmaq does.

There's now a proper recursor running for labs that supports aliases much as dnsmasq did. It is labs-recursor0.wikimedia.org.