Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Duplicate | None | T302728 Analytics Platform Future State Planing | |||
| Duplicate | None | T308317 Data Infrastructure as a Service MVP | |||
| Resolved | Gehel | T327267 Create a DSE Kubernetes cluster with support for persistent storage from Ceph | |||
| Resolved | • EChetty | T310196 K8 DSE Kubernetes Cluster | |||
| Resolved | • EChetty | T310174 Configure routing for dse-k8s cluster |
Event Timeline
Change 826525 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/homer/public@master] Add BGP neighbor data for the new dse-k8s cluster
Change 826525 merged by jenkins-bot:
[operations/homer/public@master] Add BGP neighbor data for the new dse-k8s cluster
Change 826572 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/homer/public@master] Add btullis to users to allow for router configuration
Change 826579 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/homer/public@master] Add BGP neighbor data for the new dse-k8s cluster
Change 826579 merged by jenkins-bot:
[operations/homer/public@master] Add BGP neighbor data for the new dse-k8s cluster
Pushing out this configuration to all eqiad switches/routers now.
btullis@cumin1001:~$ homer "*eqiad*" commit "T310174 enabling calico for dse-k8s cluster"
Change 827979 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/homer/public@master] Fix policy name discrepancy between dse_k8s and kubedse
Change 827979 merged by jenkins-bot:
[operations/homer/public@master] Fix policy name discrepancy between dse_k8s and kubedse
It took one more patch, but these changes to the switches and routers have all now been deployed.
Tentatively moving this ticket to done.
Change 826572 merged by jenkins-bot:
[operations/homer/public@master] Add btullis to users to allow for router configuration
Change 829037 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):
[operations/homer/public@master] Change CR template to only include Kubedse in BGP_Switch_in for eqiad
Change 829037 merged by jenkins-bot:
[operations/homer/public@master] Change CR template to only include Kubedse in BGP_Switch_in for eqiad
Thanks @BTullis, looks ok from my side too. I had to submit the above patch, but don't worry it's really just to do with the circular logic - I would have also failed to catch it. Longer term we probably need to look at that, the "switch_in" policy is probably us being a little too cautious (dates from eqiad expansion initial go-live when we were maybe more cautious about filtering what the switches sent, given it was a new setup).
Anyway thanks for looking at this. I'm hopeful as we move forward we can simplify all this somewhat but for this particular scenario hopefully we are now good!
Mentioned in SAL (#wikimedia-operations) [2022-09-05T09:25:40Z] <btullis> deployed calico to dse-k8s cluster T310174
Thanks @cmooney for the additional patch.
I had to make a small modification to /srv/private/hieradata/common/profile/kubernetes.yaml in the private repo, because I had accidentally omitted the calico user tokens.
Now I've deployed calico and everything seems fine.
root@deploy1002:~# kubectl -n kube-system get deployment,daemonset NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/calico-kube-controllers 1/1 1 1 82s deployment.apps/calico-typha 2/2 2 2 82s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/calico-node 10 10 10 10 10 kubernetes.io/os=linux 82s