Integrate the (8) existing dse-k8s worker nodes
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To

Authored By

	JArguello-WMF
	Jun 8 2022, 2:35 PM

Details

Subject	Repo	Branch	Lines +/-
Label the eight dse-k8s-worker nodes	operations/puppet	production	+24 -0
Enable the dse-k8s-worker nodes	operations/puppet	production	+92 -3
Add dummy tokens for dse_k8s_workers	labs/private	master	+6 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Duplicate	None	T302728 Analytics Platform Future State Planing
Duplicate	None	T308317 Data Infrastructure as a Service MVP
Resolved	Gehel	T327267 Create a DSE Kubernetes cluster with support for persistent storage from Ceph
Resolved	• EChetty	T310196 K8 DSE Kubernetes Cluster
Resolved	BTullis	T310177 Integrate the (8) existing dse-k8s worker nodes

Event Timeline

JArguello-WMF created this task.Jun 8 2022, 2:35 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 8 2022, 2:35 PM

BTullis added a parent task: T310196: K8 DSE Kubernetes Cluster.Jun 23 2022, 9:08 AM

• EChetty moved this task from Backlog to To be discussed on the Shared-Data-Infrastructure board.Jul 11 2022, 11:35 AM

BTullis renamed this task from Integrate the (4) existing dse-k8s worker nodes to Integrate the (8) existing dse-k8s worker nodes.Jul 14 2022, 11:18 AM

• EChetty moved this task from To be discussed to Estimated/Discussed on the Shared-Data-Infrastructure board.Jul 29 2022, 2:46 PM

• EChetty moved this task from Estimated/Discussed to To be discussed on the Shared-Data-Infrastructure board.Aug 1 2022, 3:01 PM

• EChetty added a project: Data-Engineering-Planning.Aug 1 2022, 3:02 PM

• EChetty moved this task from To be discussed to Estimated/Discussed on the Shared-Data-Infrastructure board.Aug 1 2022, 3:02 PM

• EChetty moved this task from Backlog to Shared Data Infra on the Data-Engineering-Planning board.Aug 15 2022, 8:44 AM

• EChetty moved this task from Estimated/Discussed to To be discussed on the Shared-Data-Infrastructure board.Aug 15 2022, 9:06 AM

• EChetty moved this task from To be discussed to Estimated/Discussed on the Shared-Data-Infrastructure board.Aug 16 2022, 1:22 PM

• EChetty assigned this task to BTullis.Aug 24 2022, 9:38 AM

• EChetty moved this task from Estimated/Discussed to Sprint 00 on the Shared-Data-Infrastructure board.

• EChetty edited projects, added Shared-Data-Infrastructure (Sprint 00); removed Shared-Data-Infrastructure.

• EChetty moved this task from Next Up to In Progress on the Shared-Data-Infrastructure (Sprint 00) board.

• EChetty set the point value for this task to 5.

Change 826240 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Enable the dse-k8s-worker nodes

https://gerrit.wikimedia.org/r/826240

gerritbot added a project: Patch-For-Review.Aug 24 2022, 11:01 AM

Change 826241 had a related patch set uploaded (by Btullis; author: Btullis):

[labs/private@master] Add dummy tokens for dse_k8s_workers

https://gerrit.wikimedia.org/r/826241

Change 826241 merged by Btullis:

[labs/private@master] Add dummy tokens for dse_k8s_workers

https://gerrit.wikimedia.org/r/826241

BTullis mentioned this in rLPRI5ebb89604092: Add dummy tokens for dse_k8s_workers.Aug 24 2022, 11:08 AM

Change 826240 merged by Btullis:

[operations/puppet@production] Enable the dse-k8s-worker nodes

https://gerrit.wikimedia.org/r/826240

Maintenance_bot removed a project: Patch-For-Review.Aug 25 2022, 1:30 PM

Nodes 1-4 have been integrated, but for some reason nodes 5-8 have not yet registered with the controller.

root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl get nodes
NAME                             STATUS   ROLES    AGE     VERSION
dse-k8s-ctrl1001.eqiad.wmnet     Ready    <none>   4m16s   v1.16.15
dse-k8s-ctrl1002.eqiad.wmnet     Ready    <none>   4m12s   v1.16.15
dse-k8s-worker1001.eqiad.wmnet   Ready    <none>   4m13s   v1.16.15
dse-k8s-worker1002.eqiad.wmnet   Ready    <none>   4m13s   v1.16.15
dse-k8s-worker1003.eqiad.wmnet   Ready    <none>   4m17s   v1.16.15
dse-k8s-worker1004.eqiad.wmnet   Ready    <none>   4m14s   v1.16.15

I have begun labelling the worker nodes.

Starting with labels for the reqion

root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1001.eqiad.wmnet failure-domain.beta.kubernetes.io/region=eqiad
node/dse-k8s-worker1001.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1002.eqiad.wmnet failure-domain.beta.kubernetes.io/region=eqiad
node/dse-k8s-worker1002.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1003.eqiad.wmnet failure-domain.beta.kubernetes.io/region=eqiad
node/dse-k8s-worker1003.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1004.eqiad.wmnet failure-domain.beta.kubernetes.io/region=eqiad
node/dse-k8s-worker1004.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1005.eqiad.wmnet failure-domain.beta.kubernetes.io/region=eqiad

Followed by labels for the rows

root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1001.eqiad.wmnet failure-domain.beta.kubernetes.io/zone=row-a
node/dse-k8s-worker1001.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1002.eqiad.wmnet failure-domain.beta.kubernetes.io/zone=row-b
node/dse-k8s-worker1002.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1003.eqiad.wmnet failure-domain.beta.kubernetes.io/zone=row-c
node/dse-k8s-worker1003.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1004.eqiad.wmnet failure-domain.beta.kubernetes.io/zone=row-d
node/dse-k8s-worker1004.eqiad.wmnet labeled

Change 828052 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Label the first four of the dse-k8s-worker nodes

https://gerrit.wikimedia.org/r/828052

gerritbot added a project: Patch-For-Review.Aug 30 2022, 4:07 PM

The remaining four nodes just needed a reboot before the kubelet service started.
I have carried on with the labeling.

root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1005.eqiad.wmnet failure-domain.beta.kubernetes.io/region=eqiad
node/dse-k8s-worker1005.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1006.eqiad.wmnet failure-domain.beta.kubernetes.io/region=eqiad
node/dse-k8s-worker1006.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1007.eqiad.wmnet failure-domain.beta.kubernetes.io/region=eqiad
node/dse-k8s-worker1007.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1008.eqiad.wmnet failure-domain.beta.kubernetes.io/region=eqiad
node/dse-k8s-worker1008.eqiad.wmnet labeled

Note the rack specific labels for servers in rows E-F

root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1005.eqiad.wmnet failure-domain.beta.kubernetes.io/zone=row-e1
node/dse-k8s-worker1005.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1006.eqiad.wmnet failure-domain.beta.kubernetes.io/zone=row-e3
node/dse-k8s-worker1006.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1007.eqiad.wmnet failure-domain.beta.kubernetes.io/zone=row-f1
node/dse-k8s-worker1007.eqiad.wmnet labeled
root@deploy1002:/srv/deployment-charts/helmfile.d/admin_ng# kubectl label nodes dse-k8s-worker1008.eqiad.wmnet failure-domain.beta.kubernetes.io/zone=row-f3
node/dse-k8s-worker1008.eqiad.wmnet labeled