• All ~/.kube/config files use https://k8s-master.tools.wmflabs.org:6443
• k8s-master.tools.wmflabs.org resolves to 10.68.17.142 on all tools servers
• k8s-master.tools.wmflabs.org resolves to 208.80.155.247 externally
• Neither IP are referenced in operations/dns.git or operations/puppet.git
• None of the ports allowed by the security group to connect externally (0/0) are actually open on tools-k8s-master-01
• Packet capture on labnet1001 shows:
  • 208.80.155.204 is used as NAT'ed address for external connections (because the iptable rules for it come before .247's)
  • All incoming connections were to closed ports (probably portscanners). No connections established whatsoever.
• No mention of external address in documentation

I'm not sure tools-k8s-master-01 should even have a floating IP but I'm probably missing an user case here. Does anyone have any historical context for this?

In any case, if a floating IP needs to be assigned it seems 208.80.155.247 is the correct one.

Mentioned in SAL (#wikimedia-cloud) [2018-12-04T20:03:27Z] <gtirloni> removed floating IPs from tools-k8s-master-01 (T164123)

This was discussed today and the best guess is that some tenant (possibly PAWS) wanted to expose an endpoint publicly but either ended up abandoning the idea or the tenant migrated to Cloud VPS and the public IPs remained associated.

In either case, they don't seem to be used and we don't know who would want to use them. I've removed both floating IPs and deleted the the "k8s-master.tools.wmflabs.org" DNS entry.

Mentioned in SAL (#wikimedia-cloud) [2018-12-04T22:47:49Z] <bstorm_> gtirloni added back main floating IP for tools-k8s-master-01 and removed unnecessary ones to stop k8s outage T164123

When I was investigating what was using those floating IPs, I focused on network captures. Unfortunately, I missed the fact that k8s-master.tools.wmflabs.org resolved to 10.68.17.142 internally. So while the floating IPs are probably not needed, that hostname is needed very much. It's apparently used in the Kubernetes certificate and many other places (e.g. maintainkube, toolschecker.py, etc). We'll need to investigate it much deeper to find all the places where it exists.

For the time being, the issue reported in this task is resolved. The k8s master only has one floating IP now 208.80.155.247.

Incident caused by removing the DNS entry: https://wikitech.wikimedia.org/wiki/Incident_documentation/20181204-Toolforge-Kubernetes