| • aborrero | |
| Sep 5 2024, 4:02 PM |
| F58979530: image.png | |
| Apr 4 2025, 11:55 AM |
This existed mostly to prevent VMs reaching to floating IPs from within the virtual network, because weird network things would happen.
However, the network weirdness might be solved now, so we may not need the labs-ip-aliases mechanism no more.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| ci: Switch back to Digital Ocean runners by default | repos/releng/zuul/tofu-provisioning!12 | bd808 | work/bd808/arrrrrgh | main |
Some things I did when Andrew asked me to double check that things seemed to work without split horizon DNS remapping:
root@abogott-T374129:~# hostname -f abogott-T374129.testlabs.eqiad1.wikimedia.cloud root@abogott-T374129:~# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:e0:38:a3 brd ff:ff:ff:ff:ff:ff altname enp0s3 inet 172.16.0.47/21 metric 100 brd 172.16.7.255 scope global dynamic ens3 valid_lft 83628sec preferred_lft 83628sec inet6 fe80::f816:3eff:fee0:38a3/64 scope link valid_lft forever preferred_lft forever root@abogott-T374129:~# host 185.15.56.77 77.56.15.185.in-addr.arpa is an alias for 77.0-25.56.15.185.in-addr.arpa. 77.0-25.56.15.185.in-addr.arpa domain name pointer instance-abogott-T374129.testlabs.wmcloud.org. root@abogott-T374129:~# ping 185.15.56.77 PING 185.15.56.77 (185.15.56.77) 56(84) bytes of data. 64 bytes from 185.15.56.77: icmp_seq=1 ttl=63 time=1.72 ms 64 bytes from 185.15.56.77: icmp_seq=2 ttl=63 time=0.758 ms ^C --- 185.15.56.77 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.758/1.241/1.724/0.483 ms root@abogott-T374129:~# traceroute -I 185.15.56.77 traceroute to 185.15.56.77 (185.15.56.77), 30 hops max, 60 byte packets 1 instance-abogott-T374129.testlabs.wmcloud.org (185.15.56.77) 1.005 ms 0.976 ms 0.970 ms 2 instance-abogott-T374129.testlabs.wmcloud.org (185.15.56.77) 0.826 ms 0.821 ms * root@abogott-T374129:~# traceroute -T 185.15.56.77 traceroute to 185.15.56.77 (185.15.56.77), 30 hops max, 60 byte packets 1 instance-abogott-T374129.testlabs.wmcloud.org (185.15.56.77) 0.563 ms 0.489 ms 0.432 ms 2 instance-abogott-T374129.testlabs.wmcloud.org (185.15.56.77) 0.797 ms 0.754 ms 0.633 ms root@abogott-T374129:~#
The first traceroute tests I did were not as clean, but then Andrew opened up UDP and ICMP in the service group applied to the host and things got better.
Change #1105442 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] pdns recursor: support injecting extra hostnames into recursor config
Change #1105443 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] cloud-vps dns recursor: remove labs-ip-aliaser
Change #1105442 merged by Andrew Bogott:
[operations/puppet@production] pdns recursor: support injecting extra hostnames into recursor config
Change #1105443 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps dns recursor: remove labs-ip-aliaser
Change #1109156 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Pdns recursor: Make the extrarecursorhosts file world-readable
Change #1109156 merged by Andrew Bogott:
[operations/puppet@production] Pdns recursor: Make the extrarecursorhosts file world-readable
Removing the aliaser in codfw1dev broke the connection between the enc-api and the puppetserver. So this needs more research, it may be that the aliaser is still needed or it might be that there's something more interesting happening.
Change #1109176 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] cloud-vps dns recursor: remove labs-ip-aliaser
Here is an example of not being able to reach a public IP without the name being aliased. enc-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud and puppet-enc.cloudinfra-codfw1dev.codfw1dev.wmcloud.org are the same host but the latter is a floating IP:
root@cloudinfra-cloudvps-puppetserver-1:~# curl https://enc-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud curl: (60) SSL: no alternative certificate subject name matches target host name 'enc-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud' More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. root@cloudinfra-cloudvps-puppetserver-1:~# curl https://puppet-enc.cloudinfra-codfw1dev.codfw1dev.wmcloud.org <hangs>
root@cloudinfra-cloudvps-puppetserver-1:~# traceroute --icmp --port=443 enc-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud traceroute to enc-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud (172.16.128.97), 30 hops max, 60 byte packets 1 enc-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud (172.16.128.97) 5.637 ms * * (works) root@cloudinfra-cloudvps-puppetserver-1:~# traceroute --tcp --port=443 enc-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud traceroute to enc-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud (172.16.128.97), 30 hops max, 60 byte packets 1 enc-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud (172.16.128.97) 2.258 ms 2.207 ms 2.183 ms (works) root@cloudinfra-cloudvps-puppetserver-1:~# traceroute --icmp --port=443 puppet-enc.cloudinfra-codfw1dev.codfw1dev.wmcloud.org traceroute to puppet-enc.cloudinfra-codfw1dev.codfw1dev.wmcloud.org (185.15.57.5), 30 hops max, 60 byte packets 1 puppet-enc.cloudinfra-codfw1dev.codfw1dev.wmcloud.org (185.15.57.5) 1.989 ms 1.953 ms 1.946 ms 2 instance-enc-1.cloudinfra-codfw1dev.codfw1dev.wmcloud.org (185.15.57.5) 4.085 ms * * (works) root@cloudinfra-cloudvps-puppetserver-1:~# traceroute --tcp --port=443 puppet-enc.cloudinfra-codfw1dev.codfw1dev.wmcloud.org traceroute to puppet-enc.cloudinfra-codfw1dev.codfw1dev.wmcloud.org (185.15.57.5), 30 hops max, 60 byte packets 1 puppet-enc.cloudinfra-codfw1dev.codfw1dev.wmcloud.org (185.15.57.5) 0.519 ms 0.465 ms * 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * (does not work!)
the particular problem you are describing in your last comment is that both origin and destination virtual machines have floating IPs.
cloudinfra-cloudvps-puppetserver-1 has IP address 172.16.128.65 and floating IP 185.15.57.4
env-1 has IP address 172.16.128.97 and floating IP 185.15.57.5
When a communication happens between the two, the source IP address that they will see from each other is the floating IP and not the VM address.
This means the security group should allow this communication which is otherwise rejected.
I added this security group rule to the enc-1 VM (allow 443/tcp from 185.15.57.4)
And traffic started flowing. See before and after:
aborrero@cloudinfra-cloudvps-puppetserver-1:~$ curl https://185.15.57.5 -k curl: (28) Failed to connect to 185.15.57.5 port 443 after 131028 ms: Couldn't connect to server aborrero@cloudinfra-cloudvps-puppetserver-1:~$ curl https://185.15.57.5 -k <!doctype html> <html lang=en> <title>404 Not Found</title> <h1>Not Found</h1> <p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
If we make this change, users will have to adjust firewalls on floating-IP-attached VMs in two ways:
After quite a lot of discussion and testing, @taavi has pointed out a real problem with switching to direct traffic from a VM to a floating IP, which is that the source info is completely lost and we have no ability to audit traffic.
So, ripping out this aliaser will have to wait. What we really want is for people to use v6 for internal traffic (using CNAMES instead of floating IPs for failover) at which time the aliaser won't matter much.
If/when we make that change, here's a bit of wikitext for the announcement page:
== Summary == Currently most cloud-internal traffic using a floating IP (185.x.x.x) address as the destination address is mangled to instead target the associated internal (172.x.x.x) address. Soon, that mangling will stop and traffic destined for a VM bound to a floating IP will use that actual 185.x.x.x address as the destination. This will break some internal traffic that relied on firewall or security group rules. The correct fix for this is to make sure that all cloud-internal traffic happens via IPv6 which avoids the natting and mangling associated with floating IPs. == Who Should Care About This == This change will affect: # VMs which 1) are bound to a floating IP and 2) receive inbound network traffic initiated from elsewhere within toolforge or cloud-vps # VMs, services, or tools which communicate with any of the VMs covered by case #1 using a public (.org) hostname or a public (185.x.x.x) IP address. This change will not affect web traffic initiated outside of cloud-vps, and will also not affect traffic that /is/ initiated from within cloud-vps but uses an internal 172.x.x.x address as the destination.
bd808 opened https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/12
ci: Switch back to Digital Ocean runners by default
bd808 merged https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/12
ci: Switch back to Digital Ocean runners by default