Page MenuHomePhabricator

Configure networking for the dse-k8s-codfw cluster
Closed, ResolvedPublic

Description

We have already reserved the IP ranges in netbox for the dse-k8s-codfw cluster, when we commissioned the cluster in eqiad.

We need to mark these ranges as active and configure the remaining settings before bootstrapping the cluster.

See these guidelines for details: https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/New#Networking

Step 0: DNS (Done)
Step 1: Add node to BGP (Done)
Step 2: Node installation (done)
Step 3: Add to conftool/LVS (Done)

Event Timeline

BTullis triaged this task as High priority.Jul 28 2025, 6:20 PM

Change #1178834 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] dse-k8s: add dse-k8s-codfw hosts to LVS

https://gerrit.wikimedia.org/r/1178834

Change #1179652 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] Update firewall rules to add dse-k8s-codfw

https://gerrit.wikimedia.org/r/1179652

Change #1179654 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] dse-k8s: Add helmfile configuration for dse-k8s-codfw

https://gerrit.wikimedia.org/r/1179654

Change #1179652 merged by Stevemunene:

[operations/puppet@production] Update firewall rules to add dse-k8s-codfw

https://gerrit.wikimedia.org/r/1179652

Change #1179723 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/dns@master] dns: Define a DNS A record for the dse-k8s-codfw ingress

https://gerrit.wikimedia.org/r/1179723

Change #1179723 merged by Stevemunene:

[operations/dns@master] dns: Define DNS records for the dse-k8s-codfw ingress

https://gerrit.wikimedia.org/r/1179723

Change #1180116 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] dse-k8s: Add dse-k8s-codfw to service list

https://gerrit.wikimedia.org/r/1180116

Change #1178834 merged by Stevemunene:

[operations/puppet@production] dse-k8s: add dse-k8s-codfw hosts to LVS

https://gerrit.wikimedia.org/r/1178834

Change #1180116 merged by Stevemunene:

[operations/puppet@production] dse-k8s: Add dse-k8s-codfw to service list

https://gerrit.wikimedia.org/r/1180116

Change #1179654 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s: Add helmfile configuration for dse-k8s-codfw

https://gerrit.wikimedia.org/r/1179654

Stevemunene updated the task description. (Show Details)

Set the BGP value to true for the dse-k8s-codfw ctrl and worker hosts then ran homer from a cumin host

stevemunene@cumin1003:~$ sudo homer "cr*codfw*" diff
INFO:homer.devices:Initialized 105 devices
INFO:homer:Generating diff for query cr*codfw*
INFO:homer.devices:Matched 2 device(s) for query 'cr*codfw*'
INFO:homer:Generating configuration for cr1-codfw.wikimedia.org
INFO:homer.transports.junos:Running commit check on cr1-codfw.wikimedia.org
INFO:homer:Generating configuration for cr2-codfw.wikimedia.org
INFO:homer.transports.junos:Running commit check on cr2-codfw.wikimedia.org
Changes for 1 devices: ['cr1-codfw.wikimedia.org']

[edit policy-options]
+   prefix-list kubedse-pod-ips4 {
+       10.192.96.0/21;
+   }
+   prefix-list kubedse-pod-ips6 {
+       2620:0:860:308::/64;
+   }
[edit policy-options]
+   policy-statement kubedse_import {
+       term pod_ips4 {
+           from {
+               family inet;
+               protocol bgp;
+               prefix-list-filter kubedse-pod-ips4 longer;
+           }
+           then accept;
+       }
+       term pod_ips6 {
+           from {
+               family inet6;
+               protocol bgp;
+               prefix-list-filter kubedse-pod-ips6 longer;
+           }
+           then accept;
+       }
+       then reject;
+   }
[edit protocols bgp]
     group k8s-aux-ipv6 { ... }
+    group Kubedse4 {
+        type external;
+        multihop {
+            /* T328523 */
+            no-nexthop-change;
+        }
+        local-address 208.80.153.192;
+        hold-time 30;
+        /* T328523 */
+        advertise-peer-as;
+        import kubedse_import;
+        family inet {
+            unicast {
+                prefix-limit {
+                    maximum 50;
+                    teardown {
+                        80;
+                        idle-timeout forever;
+                    }
+                }
+            }
+        }
+        /* T328523 */
+        export kubernetes_export;
+        peer-as 64613;
+        multipath;
+        neighbor 10.192.32.6 {
+            description dse-k8s-ctrl2001;
+        }
+        neighbor 10.192.48.13 {
+            description dse-k8s-ctrl2002;
+        }
+    }
+    group Kubedse6 {
+        type external;
+        multihop {
+            /* T328523 */
+            no-nexthop-change;
+        }
+        local-address 2620:0:860:ffff::1;
+        hold-time 30;
+        /* T328523 */
+        advertise-peer-as;
+        import kubedse_import;
+        family inet6 {
+            unicast {
+                prefix-limit {
+                    maximum 50;
+                    teardown {
+                        80;
+                        idle-timeout forever;
+                    }
+                }
+            }
+        }
+        /* T328523 */
+        export kubernetes_export;
+        peer-as 64613;
+        multipath;
+        neighbor 2620:0:860:103:10:192:32:6 {
+            description dse-k8s-ctrl2001;
+        }
+        neighbor 2620:0:860:104:10:192:48:13 {
+            description dse-k8s-ctrl2002;
+        }
+    }

---------------
Changes for 1 devices: ['cr2-codfw.wikimedia.org']

[edit policy-options]
+   prefix-list kubedse-pod-ips4 {
+       10.192.96.0/21;
+   }
+   prefix-list kubedse-pod-ips6 {
+       2620:0:860:308::/64;
+   }
[edit policy-options]
+   policy-statement kubedse_import {
+       term pod_ips4 {
+           from {
+               family inet;
+               protocol bgp;
+               prefix-list-filter kubedse-pod-ips4 longer;
+           }
+           then accept;
+       }
+       term pod_ips6 {
+           from {
+               family inet6;
+               protocol bgp;
+               prefix-list-filter kubedse-pod-ips6 longer;
+           }
+           then accept;
+       }
+       then reject;
+   }
[edit protocols bgp]
     group k8s-aux-ipv6 { ... }
+    group Kubedse4 {
+        type external;
+        multihop {
+            /* T328523 */
+            no-nexthop-change;
+        }
+        local-address 208.80.153.193;
+        hold-time 30;
+        /* T328523 */
+        advertise-peer-as;
+        import kubedse_import;
+        family inet {
+            unicast {
+                prefix-limit {
+                    maximum 50;
+                    teardown {
+                        80;
+                        idle-timeout forever;
+                    }
+                }
+            }
+        }
+        /* T328523 */
+        export kubernetes_export;
+        peer-as 64613;
+        multipath;
+        neighbor 10.192.32.6 {
+            description dse-k8s-ctrl2001;
+        }
+        neighbor 10.192.48.13 {
+            description dse-k8s-ctrl2002;
+        }
+    }
+    group Kubedse6 {
+        type external;
+        multihop {
+            /* T328523 */
+            no-nexthop-change;
+        }
+        local-address 2620:0:860:ffff::2;
+        hold-time 30;
+        /* T328523 */
+        advertise-peer-as;
+        import kubedse_import;
+        family inet6 {
+            unicast {
+                prefix-limit {
+                    maximum 50;
+                    teardown {
+                        80;
+                        idle-timeout forever;
+                    }
+                }
+            }
+        }
+        /* T328523 */
+        export kubernetes_export;
+        peer-as 64613;
+        multipath;
+        neighbor 2620:0:860:103:10:192:32:6 {
+            description dse-k8s-ctrl2001;
+        }
+        neighbor 2620:0:860:104:10:192:48:13 {
+            description dse-k8s-ctrl2002;
+        }
+    }

---------------
INFO:homer:Homer run completed successfully on 2 devices: ['cr1-codfw.wikimedia.org', 'cr2-codfw.wikimedia.org']

Change #1183691 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] dse-k8s: disable cluster_dns to allow core-dns deploy.

https://gerrit.wikimedia.org/r/1183691

Change #1183691 abandoned by Btullis:

[operations/puppet@production] dse-k8s: disable cluster_dns to allow core-dns deploy.

Reason:

Not needed now.

https://gerrit.wikimedia.org/r/1183691