tools-k8s-master-01 has two floating IPs
Closed, ResolvedPublic

Description

We should probably remove both but I'm not entirely sure. At least one of these is extraneous.

208.80.155.204 [floating]
208.80.155.247 [floating]

https://tools.wmflabs.org/openstack-browser/server/tools-k8s-master-01.tools.eqiad.wmflabs

k8s-master.tools.wmflabs.org
k8s-master.tools.wmflabs.org has address 208.80.155.247

chasemp created this task.Apr 28 2017, 10:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 28 2017, 10:19 PM
chasemp triaged this task as Normal priority.Apr 28 2017, 10:19 PM
chasemp claimed this task.
chasemp removed chasemp as the assignee of this task.Apr 26 2018, 7:46 PM
Andrew claimed this task.Aug 24 2018, 7:59 PM
Andrew removed Andrew as the assignee of this task.Nov 15 2018, 3:35 PM
Andrew added a subscriber: Andrew.

Worst case, this will get resolved when we move regions

Andrew lowered the priority of this task from Normal to Low.Nov 15 2018, 3:35 PM
  • All ~/.kube/config files use https://k8s-master.tools.wmflabs.org:6443
  • k8s-master.tools.wmflabs.org resolves to 10.68.17.142 on all tools servers
  • k8s-master.tools.wmflabs.org resolves to 208.80.155.247 externally
  • Neither IP are referenced in operations/dns.git or operations/puppet.git
  • None of the ports allowed by the security group to connect externally (0/0) are actually open on tools-k8s-master-01
  • Packet capture on labnet1001 shows:
    • 208.80.155.204 is used as NAT'ed address for external connections (because the iptable rules for it come before .247's)
    • All incoming connections were to closed ports (probably portscanners). No connections established whatsoever.
  • No mention of external address in documentation

I'm not sure tools-k8s-master-01 should even have a floating IP but I'm probably missing an user case here. Does anyone have any historical context for this?

In any case, if a floating IP needs to be assigned it seems 208.80.155.247 is the correct one.

GTirloni claimed this task.Sat, Dec 1, 5:43 AM
GTirloni moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Mentioned in SAL (#wikimedia-cloud) [2018-12-04T20:03:27Z] <gtirloni> removed floating IPs from tools-k8s-master-01 (T164123)

This was discussed today and the best guess is that some tenant (possibly PAWS) wanted to expose an endpoint publicly but either ended up abandoning the idea or the tenant migrated to Cloud VPS and the public IPs remained associated.

In either case, they don't seem to be used and we don't know who would want to use them. I've removed both floating IPs and deleted the the "k8s-master.tools.wmflabs.org" DNS entry.

GTirloni closed this task as Resolved.Tue, Dec 4, 8:08 PM
GTirloni edited projects, added Toolforge; removed Cloud-Services.

Mentioned in SAL (#wikimedia-cloud) [2018-12-04T22:47:49Z] <bstorm_> gtirloni added back main floating IP for tools-k8s-master-01 and removed unnecessary ones to stop k8s outage T164123

When I was investigating what was using those floating IPs, I focused on network captures. Unfortunately, I missed the fact that k8s-master.tools.wmflabs.org resolved to 10.68.17.142 internally. So while the floating IPs are probably not needed, that hostname is needed very much. It's apparently used in the Kubernetes certificate and many other places (e.g. maintainkube, toolschecker.py, etc). We'll need to investigate it much deeper to find all the places where it exists.

For the time being, the issue reported in this task is resolved. The k8s master only has one floating IP now 208.80.155.247.