Page MenuHomePhabricator

Determine dse-k8s-codfw Kubernetes IP ranges
Closed, ResolvedPublic

Description

We are setting up the dse-k8s-codfw etcd cluster and require several IP ranges to be assigned: As per: https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/New#Prerequisites

This was done for the dse-k8s-eqiad cluster in T310169: Determine IP ranges for dse-k8s cluster

  • POD IPs - The ipv4 pool in helmfile.d/admin_ng/values/staging*/calico-values.yaml
  • Service IPs The cluster_cidr in hieradata/common/kubernetes.yaml
  • The Kubernetes POD IP delegation in templates/10.in-addr.arpa

We shall be using the same ranges as the eqiad cluster defined here https://github.com/wikimedia/operations-puppet/blob/production/hieradata/common/kubernetes.yaml#L866-L871 and here https://github.com/wikimedia/operations-dns/blob/master/templates/10.in-addr.arpa#L995C1-L1024C58

Event Timeline

BTullis renamed this task from dse-k8s-codfw Kubernetes POD IP delegation to Determine dse-k8s-codfw Kubernetes IP ranges.Jul 21 2025, 9:35 AM
BTullis updated the task description. (Show Details)

The dse-k8s-eqiad cluster uses the following IPv4 ranges:

  • 10.67.24.0/21 - pod IPs
  • 10.67.32.0/20 - service IPs

We chose a relatively large (/20) range for the service IPs, because we were expecting to deploy knative-serving to this cluster, as per: T310169#7992185
This far, we haven't done this, but I would still be most comfortable if we could allocate ranges of the same size to the codfw cluster.

The ipv6 ranges are:

  • 2620:0:861:302::/64 - pod IPs
  • 2620:0:861:303::/116 - service IPs

@cmooney @ayounsi - is this request reasonable, from your point of view?

I see that in the 10.194.0.0/16 range, both the ml-serve and aux clusters have also gone with the /20 and /21 approach, with a /18 container for the whole cluster.

image.png (670×1 px, 168 KB)

BTullis triaged this task as Medium priority.Jul 21 2025, 10:03 AM

@BTullis in terms of 10.192.128.0/17 I think we can probably avoid allocating any addressing from that range for now.

In terms of the above precedent I would say it makes sense if you feel you'll outgrow the /20 and /21 allocation, and want a larger /18 allocation now so there is unused free space adjacent to the initial ranges. If you don't expect to outgrow the /20 and /21 I would think we are probably better off not following that pattern and just assigning those blocks without a parent "container" for optimal use.

What I'd therefore propose is:

10.192.80.0/20 - pod IPs
10.192.96.0/21 - service IPs
2620:0:860:307::/64 - pod IPs
2620:0:860:308::/64 - service IP allocation

If we want to assign a /116 network in netbox for service IPs we can nest it within the 2620:0:860:308::/64, but I think it's preferable to break the larger block up in chunks of /64 at maximum to avoid subnet hell :)

I don't think we need to allocate a new larger container for K8s IP spaces in codfw. And a /18 seems a little large to accommodate a /20 and /21 allocation so no need to jump to that I think.

That's great. Many thanks @cmooney. I have created them now.

image.png (448×1 px, 86 KB)

I took the liberty of swapping the /20 and /21 because the other clusters use the /20 for service IPs. As mentioned, this is due to the fact that we may wish to run knative-serving, which uses an increased number of service IPs.

I'll go ahead and create an ASN for this cluster, too.

That's great. Many thanks @cmooney. I have created them now.

great, thanks.

I took the liberty of swapping the /20 and /21 because the other clusters use the /20 for service IPs.

yep that was a typo on my part no probs at all.

Change #1171621 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Add ASN mapping and import policy for dse-k8s codfw

https://gerrit.wikimedia.org/r/1171621

@BTullis if you want to ping me when you've the cluster detail added to hieradata/common/kubernetes.yaml I can do the reverse DNS delegation bit (I've a script to generate it based on that yaml file).

Change #1171621 merged by jenkins-bot:

[operations/homer/public@master] Add ASN mapping and import policy for dse-k8s codfw

https://gerrit.wikimedia.org/r/1171621

@BTullis if you want to ping me when you've the cluster detail added to hieradata/common/kubernetes.yaml I can do the reverse DNS delegation bit (I've a script to generate it based on that yaml file).

@cmooney - We've merged those changes to hieradata/common/kubernetes.yaml now, so if you'd like to work out what our DNS delegation settings should be, that would be great. Thanks.

Change #1173433 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Add reverse delegations for codfw K8s dse ranges

https://gerrit.wikimedia.org/r/1173433

Change #1173433 merged by Cathal Mooney:

[operations/dns@master] Add reverse delegations for codfw K8s dse ranges

https://gerrit.wikimedia.org/r/1173433

Change #1177990 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/dns@master] Delegate Kubernetes POD IP reverse ranges for dse-k8s-codfw cluster This patch updates the current delegations based on data from the puppet repo hieradata/common/kubernetes.yaml file for the codfw cluster.

https://gerrit.wikimedia.org/r/1177990

Updating from slack conversations,
With the current setup, CoreDNS isn't really following proper DNS spec and answering queries for NS or SOA records.
Short comparison for eqiad vs what we have for codfw

stevemunene@cumin1003:~$ dig +nsid -4 -x 10.67.134.8 @wikikube-ctrl1002.eqiad.wmnet 

; <<>> DiG 9.18.33-1~deb12u2-Debian <<>> +nsid -4 -x 10.67.134.8 @wikikube-ctrl1002.eqiad.wmnet
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15838
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
; NSID: 63 6f 72 65 64 6e 73 2d 39 62 64 35 37 63 37 38 37 2d 36 6c 35 71 6b ("coredns-9bd57c787-6l5qk")
;; QUESTION SECTION:
;8.134.67.10.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
8.134.67.10.in-addr.arpa. 5	IN	PTR	10-67-134-8.istio-ingressgateway.istio-system.svc.cluster.local.

;; Query time: 4 msec
;; SERVER: 10.64.16.16#53(wikikube-ctrl1002.eqiad.wmnet) (UDP)
;; WHEN: Tue Aug 12 12:55:15 UTC 2025
;; MSG SIZE  rcvd: 181

stevemunene@cumin1003:~$  dig -4 NS 96.192.10.in-addr.arpa @dse-k8s-ctrl2001.codfw.wmnet. 
;; communications error to 10.192.32.6#53: timed out
;; communications error to 10.192.32.6#53: timed out
;; communications error to 10.192.32.6#53: timed out

; <<>> DiG 9.18.33-1~deb12u2-Debian <<>> -4 NS 96.192.10.in-addr.arpa @dse-k8s-ctrl2001.codfw.wmnet.
;; global options: +cmd
;; no servers could be reached

his is what we have for eqiad https://github.com/wikimedia/operations-dns/blob/master/templates/10.in-addr.arpa#L987-L1010 and for codfw but not responding as expected https://github.com/wikimedia/operations-dns/blob/master/templates/10.in-addr.arpa#L1388-L1411.
Looking for any missing links related to this or a step that was missed during the setup
WIP: 1177990: Delegate Kubernetes POD IP reverse ranges for dse-k8s-codfw cluster | https://gerrit.wikimedia.org/r/c/operations/dns/+/1177990

We finally got to bootstrap the etcd cluster, we can mark this as done