Page MenuHomePhabricator

tools-k8s-master-01 has two floating IPs
Closed, ResolvedPublic


We should probably remove both but I'm not entirely sure. At least one of these is extraneous. [floating] [floating] has address

Event Timeline

chasemp triaged this task as Medium priority.
Andrew subscribed.

Worst case, this will get resolved when we move regions

Andrew lowered the priority of this task from Medium to Low.Nov 15 2018, 3:35 PM
  • All ~/.kube/config files use
  • resolves to on all tools servers
  • resolves to externally
  • Neither IP are referenced in operations/dns.git or operations/puppet.git
  • None of the ports allowed by the security group to connect externally (0/0) are actually open on tools-k8s-master-01
  • Packet capture on labnet1001 shows:
    • is used as NAT'ed address for external connections (because the iptable rules for it come before .247's)
    • All incoming connections were to closed ports (probably portscanners). No connections established whatsoever.
  • No mention of external address in documentation

I'm not sure tools-k8s-master-01 should even have a floating IP but I'm probably missing an user case here. Does anyone have any historical context for this?

In any case, if a floating IP needs to be assigned it seems is the correct one.

Mentioned in SAL (#wikimedia-cloud) [2018-12-04T20:03:27Z] <gtirloni> removed floating IPs from tools-k8s-master-01 (T164123)

This was discussed today and the best guess is that some tenant (possibly PAWS) wanted to expose an endpoint publicly but either ended up abandoning the idea or the tenant migrated to Cloud VPS and the public IPs remained associated.

In either case, they don't seem to be used and we don't know who would want to use them. I've removed both floating IPs and deleted the the "" DNS entry.

GTirloni edited projects, added Toolforge; removed Cloud-Services.

Mentioned in SAL (#wikimedia-cloud) [2018-12-04T22:47:49Z] <bstorm_> gtirloni added back main floating IP for tools-k8s-master-01 and removed unnecessary ones to stop k8s outage T164123

When I was investigating what was using those floating IPs, I focused on network captures. Unfortunately, I missed the fact that resolved to internally. So while the floating IPs are probably not needed, that hostname is needed very much. It's apparently used in the Kubernetes certificate and many other places (e.g. maintainkube,, etc). We'll need to investigate it much deeper to find all the places where it exists.

For the time being, the issue reported in this task is resolved. The k8s master only has one floating IP now