Page MenuHomePhabricator
Create Task
Maniphest T313129

Configure etcd for dse-k8s cluster
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

We have three VMs for this role now, so the next step is to bootstrap an etcd cluster on them.

As per the instructions here: https://wikitech.wikimedia.org/wiki/Etcd#Bootstrapping_an_etcd_cluster

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

BTullis created this task.Jul 15 2022, 4:30 PM
BTullis added a parent task: T311131: Site: eqiad : 3 VMs requested for Etcd cluster in support of the new DSE Kubernetes cluster.
BTullis edited parent tasks, added: T310170: Deploy (3) etcd cluster of VMs for dse-k8s cluster; removed: T311131: Site: eqiad : 3 VMs requested for Etcd cluster in support of the new DSE Kubernetes cluster.Jul 15 2022, 4:32 PM
BTullis mentioned this in T310170: Deploy (3) etcd cluster of VMs for dse-k8s cluster.
• EChetty moved this task from Backlog to To be Discussed on the Data-Engineering-Planning board.Jul 25 2022, 10:01 AM
• EChetty assigned this task to BTullis.Aug 1 2022, 4:00 PM
• EChetty moved this task from Backlog to In Progress before Value Stream Kickoff (before Aug 15th) on the Shared-Data-Infrastructure board.
• EChetty edited projects, added Data-Engineering-Planning (Sprint 02); removed Data-Engineering-Planning.
• EChetty set the point value for this task to 3.
• EChetty moved this task from Ready to In progress on the Data-Engineering-Planning (Sprint 02) board.Aug 1 2022, 4:08 PM
gerritbot added a comment.Aug 2 2022, 12:18 PM

Change 819565 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/dns@master] Add DNS SRV records for dse-k8s etcd cluster

https://gerrit.wikimedia.org/r/819565

gerritbot added a project: Patch-For-Review.Aug 2 2022, 12:18 PM
gerritbot added a comment.Aug 3 2022, 9:31 AM

Change 820090 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add an option to use the PKI for etcd intra-cluster certificates

https://gerrit.wikimedia.org/r/820090

gerritbot added a comment.Aug 3 2022, 9:35 AM

Change 819565 merged by Btullis:

[operations/dns@master] Add DNS SRV records for dse-k8s etcd cluster

https://gerrit.wikimedia.org/r/819565

gerritbot added a comment.Aug 4 2022, 8:48 AM

Change 820090 merged by Btullis:

[operations/puppet@production] Add an option to use the PKI for etcd intra-cluster certificates

https://gerrit.wikimedia.org/r/820090

Maintenance_bot removed a project: Patch-For-Review.Aug 4 2022, 9:30 AM
gerritbot added a comment.Aug 4 2022, 11:01 AM

Change 820416 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Bootstrap etcd on the dse_k8s_etcd cluster

https://gerrit.wikimedia.org/r/820416

gerritbot added a project: Patch-For-Review.Aug 4 2022, 11:01 AM
gerritbot added a comment.Aug 8 2022, 9:18 AM

Change 821186 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/dns@master] Replace underscores with hyphens in dse-k8s-etcd cluster

https://gerrit.wikimedia.org/r/821186

gerritbot added a comment.Aug 8 2022, 9:25 AM

Change 821186 merged by Btullis:

[operations/dns@master] Replace underscores with hyphens in dse-k8s-etcd cluster

https://gerrit.wikimedia.org/r/821186

• EChetty moved this task from In progress to In code review on the Data-Engineering-Planning (Sprint 02) board.Aug 8 2022, 3:17 PM
• EChetty moved this task from In code review to In progress on the Data-Engineering-Planning (Sprint 02) board.
gerritbot added a comment.Aug 9 2022, 8:53 AM

Change 820416 merged by Btullis:

[operations/puppet@production] Bootstrap etcd on the dse_k8s_etcd cluster

https://gerrit.wikimedia.org/r/820416

gerritbot added a comment.Aug 9 2022, 9:38 AM

Change 821691 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Change ownership of etcd certificates for cfssl usecase

https://gerrit.wikimedia.org/r/821691

gerritbot added a comment.Aug 9 2022, 12:40 PM

Change 821691 merged by Btullis:

[operations/puppet@production] Change ownership of etcd certificates for cfssl usecase

https://gerrit.wikimedia.org/r/821691

gerritbot added a comment.Aug 9 2022, 1:19 PM

Change 821721 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Change the output directory that cfssl uses for etcd

https://gerrit.wikimedia.org/r/821721

gerritbot added a comment.Aug 9 2022, 3:09 PM

Change 821721 merged by Btullis:

[operations/puppet@production] Change the output directory that cfssl uses for etcd

https://gerrit.wikimedia.org/r/821721

gerritbot added a comment.Aug 9 2022, 5:21 PM

Change 821780 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Use the chained certificate for the etcd cfssl option

https://gerrit.wikimedia.org/r/821780

BTullis added a comment.Aug 10 2022, 9:20 AM

Status update, we have decided to use the cfssl based PKI for thie etcd cluster, as opposed to the PuppetCA certificates.
As such I have added a boolean option to the etcd::v3 profile called use_pki_certs.

When this option is true the get_cert function is called which automatically generates the required certificates for etcd to use.

However, since the etcd servers all use TLS client authentication to verify each other before the cluster can form, we will need to create a new intermediate as per the policy here: https://wikitech.wikimedia.org/wiki/PKI/Policy#When_to_create_a_new_intermediate_CA

I also need to verify that etcd can send the intermediate CA certificate chain along with the host's certificate, which is what https://gerrit.wikimedia.org/r/821780 aims to do.

gerritbot added a comment.Aug 10 2022, 9:43 AM

Change 821780 merged by Btullis:

[operations/puppet@production] Use the chained certificate for the etcd cfssl option

https://gerrit.wikimedia.org/r/821780

gerritbot added a comment.Aug 10 2022, 10:18 AM

Change 822053 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add a new intermediate CA for use with etcd

https://gerrit.wikimedia.org/r/822053

• EChetty moved this task from In progress to In code review on the Data-Engineering-Planning (Sprint 02) board.Aug 10 2022, 10:31 AM
gerritbot added a comment.Aug 10 2022, 10:42 AM

Change 822053 merged by Btullis:

[operations/puppet@production] Add a new intermediate CA for use with etcd

https://gerrit.wikimedia.org/r/822053

BTullis triaged this task as High priority.Aug 10 2022, 10:54 AM
BTullis moved this task from In code review to In progress on the Data-Engineering-Planning (Sprint 02) board.
gerritbot added a comment.Aug 10 2022, 1:58 PM

Change 822086 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Configure the new intermediate CA for etcd use

https://gerrit.wikimedia.org/r/822086

gerritbot added a comment.Aug 10 2022, 2:14 PM

Change 822089 had a related patch set uploaded (by Btullis; author: Btullis):

[labs/private@master] Add a dummy private key for the etcd intermediate CA

https://gerrit.wikimedia.org/r/822089

gerritbot added a comment.Aug 10 2022, 2:15 PM

Change 822089 merged by Btullis:

[labs/private@master] Add a dummy private key for the etcd intermediate CA

https://gerrit.wikimedia.org/r/822089

BTullis mentioned this in rLPRI9cc73c7a585b: Add a dummy private key for the etcd intermediate CA.Aug 10 2022, 2:25 PM
gerritbot added a comment.Aug 10 2022, 4:09 PM

Change 822086 merged by Btullis:

[operations/puppet@production] Configure the new intermediate CA for etcd use

https://gerrit.wikimedia.org/r/822086

gerritbot added a comment.Aug 10 2022, 4:25 PM

Change 822118 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Use specific etcd intermediate CA to generate etcd certs in PKI mode

https://gerrit.wikimedia.org/r/822118

gerritbot added a comment.Aug 10 2022, 4:54 PM

Change 822118 merged by Btullis:

[operations/puppet@production] Use specific etcd intermediate CA to generate etcd certs in PKI mode

https://gerrit.wikimedia.org/r/822118

BTullis added a comment.Aug 10 2022, 5:00 PM

Success:

btullis@dse-k8s-etcd1001:~$ etcdctl  -C https://dse-k8s-etcd1001.eqiad.wmnet:2379 cluster-health
member 5fdf30a4c955ab6 is healthy: got healthy result from https://dse-k8s-etcd1002.eqiad.wmnet:2379
member 352f48bc392d9ef9 is healthy: got healthy result from https://dse-k8s-etcd1001.eqiad.wmnet:2379
member bc33f1310f30f8fd is healthy: got healthy result from https://dse-k8s-etcd1003.eqiad.wmnet:2379
cluster is healthy
gerritbot added a comment.Aug 10 2022, 5:03 PM

Change 822120 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Remove the bootstrap param from the dse-k8s-etcd cluster

https://gerrit.wikimedia.org/r/822120

gerritbot added a comment.Aug 10 2022, 5:05 PM

Change 822120 merged by Btullis:

[operations/puppet@production] Remove the bootstrap param from the dse-k8s-etcd cluster

https://gerrit.wikimedia.org/r/822120

BTullis moved this task from In progress to Done on the Data-Engineering-Planning (Sprint 02) board.Aug 10 2022, 5:07 PM
BTullis removed a project: Patch-For-Review.
BTullis moved this task from In Progress before Value Stream Kickoff (before Aug 15th) to Done before Values Stream Kickoff (before Aug 15th) on the Shared-Data-Infrastructure board.Aug 10 2022, 5:12 PM
BTullis closed this task as Resolved.Aug 11 2022, 4:39 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits