This is the task to track all the work related to modernize the Toolforge puppet code for etcd in k8s.
It turns out that a proper etcd deployment is required in order to run kubernetes, so we should do this first.
I will be developing the code in toolsbeta before an actual deployment in the tools project.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | • Bstorm | T246122 Upgrade the Toolforge Kubernetes cluster to v1.16 | |||
| Restricted Task | |||||
| Resolved | bd808 | T232536 Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail | |||
| Resolved | • Bstorm | T236565 "tools" Cloud VPS project jessie deprecation | |||
| Resolved | aborrero | T101651 Set up toolsbeta more fully to help make testing easier | |||
| Resolved | • Bstorm | T166949 Homedir/UID info breaks after a while in Tools Kubernetes (can't read replica.my.cnf) | |||
| Resolved | • Bstorm | T246059 Add admin account creation to maintain-kubeusers | |||
| Resolved | • Bstorm | T154504 Make webservice backend default to kubernetes | |||
| Declined | None | T245230 Investigate cpu/ram requests and limits for DaemonSets pods | |||
| Resolved | • Bstorm | T214513 Deploy and migrate tools to a Kubernetes v1.15 or newer cluster | |||
| Resolved | aborrero | T215531 Deploy upgraded Kubernetes to toolsbeta | |||
| Resolved | aborrero | T226098 Toolforge: modernize deployment for etcd in k8s |
Change 517858 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: k8s: etcd: don't wrap profile::etcd, and use base etcd v3 directly
Change 517858 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: k8s: etcd: don't wrap profile::etcd, and use base etcd v3 directly
Change 517896 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: k8s: etcd: specify more certificates
Change 517896 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: k8s: etcd: specify more certificates
Change 517905 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: k8s: etcd: add etcd user to the puppet group
Change 517905 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: k8s: etcd: add etcd user to the puppet group
Change 517906 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: k8s: etcd: use complete fqdn in node name
Change 517906 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: k8s: etcd: use complete fqdn in node name
Change 518020 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: k8s: etcd: restart etcd service when certs change
Using puppet certs for etcd has some other tricky parts, like:
Jun 20 12:06:12 toolsbeta-arturo-k8s-etcd-1 etcd[28005]: health check for peer 5323d67b4ea7da68 could not connect: x509: cannot validate certificate for 172.16.0.243 because it doesn't contain any IP SANs Jun 20 12:06:12 toolsbeta-arturo-k8s-etcd-1 etcd[28005]: health check for peer 7e025ec0fe50d8f2 could not connect: x509: cannot validate certificate for 172.16.0.240 because it doesn't contain any IP SANs
My current config would need some additional tweaks that I'm still investigating.
Change 518075 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: k8s: etcd: use domain names instead of IP addresses
I still don't find why etcd would do that.
I can connect by hand using the same certs:
root@toolsbeta-arturo-k8s-etcd-1:~# openssl s_client -CAfile /var/lib/puppet/ssl/certs/ca.pem -connect 172.16.0.240:2380 \
-cert /var/lib/puppet/ssl/certs/toolsbeta-arturo-k8s-etcd-1.toolsbeta.eqiad.wmflabs.pem \
-key /var/lib/puppet/ssl/private_keys/toolsbeta-arturo-k8s-etcd-1.toolsbeta.eqiad.wmflabs.pem
[... ok ...]I'm using etcd v 3.2.12-1 from our internal repo. It seems newer etcd versions contains an additional config flag to allow arbitrary SANs in the peer certificate: https://etcd.io/docs/v3.3.12/op-guide/configuration/#peer-cert-allowed-cn
Change 518075 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: k8s: etcd: use domain names instead of IP addresses
Change 518235 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: k8s: etcd: enable 2379/tcp for peers as well
Change 518235 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: k8s: etcd: enable 2379/tcp for peers as well
OK, after some help from @Joe and @Vgutierrez I got the cluster working:
root@toolsbeta-arturo-k8s-etcd-1:~# etcdctl --endpoints https://toolsbeta-arturo-k8s-etcd-1.toolsbeta.eqiad.wmflabs:2379 cluster-health member 4c91c386e446da71 is healthy: got healthy result from https://toolsbeta-arturo-k8s-etcd-1.toolsbeta.eqiad.wmflabs:2379 member 591dee308ea139ea is healthy: got healthy result from https://toolsbeta-arturo-k8s-etcd-2.toolsbeta.eqiad.wmflabs:2379 member b0e3c89a1c97e359 is healthy: got healthy result from https://toolsbeta-arturo-k8s-etcd-3.toolsbeta.eqiad.wmflabs:2379
Change 518020 abandoned by Arturo Borrero Gonzalez:
toolforge: k8s: etcd: restart etcd service when certs change
Reason:
Using https://gerrit.wikimedia.org/r/c/operations/puppet/ /518238 instead
Mentioned in SAL (#wikimedia-cloud) [2019-06-21T11:42:49Z] <arturo> re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in T226098
Change 518247 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: k8s: etcd: also create /etc/etcd
Change 518247 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: k8s: etcd: also create /etc/etcd
I'm pretty happy now with how etcd looks in the puppet tree and the resulting state with fresh installed VMs. I will probably leave it as is and move on to kubernetes itself and see if anything else is required once k8s is actually using etcd.
For the record, I only used these hiera keys in the prefix for the basic bootstrap:
profile::etcd::cluster_bootstrap: true profile::ldap::client::labs::client_stack: sssd profile::toolforge::k8s::etcd_hosts: - toolsbeta-arturo-k8s-etcd-1.toolsbeta.eqiad.wmflabs - toolsbeta-arturo-k8s-etcd-2.toolsbeta.eqiad.wmflabs - toolsbeta-arturo-k8s-etcd-3.toolsbeta.eqiad.wmflabs sudo_flavor: sudo
Mentioned in SAL (#wikimedia-cloud) [2019-07-18T12:47:13Z] <arturo> create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code (T226098)