Sister task of T277191
We will be upgrading the kubernetes cluster in eqiad to kubernetes 1.16, calico 3.17 like we did for codfw in T277191
This includes:
* Setting up new master VMs `kubemaster100[12].eqiad.wmnet`, VMs set up in T276204
* Rebooting `kubetcd[1004-1006].eqiad.wmnet` for T273278
* Reimaging worker nodes `kubernetes[1001-1016].eqiad.wmnet`
** Add role to `kubernetes1017.codfw.wmnet` (latest addition to cluster)
** Add `homer/public` change for kubernetes1017
** Pool kubernetes1017 to conftool
** With Kernel 4.19 T262527 (which also fixes issues described in T273279)
The plan is roughly:
== Preparation ==
* Prepare all needed patches
[] Aggregate the IPv4 pools into respective /21: missing
[] Add the role to the new master VMs: missing
[] Enabling the kernel 4.19 profile for nodes: missing
[] Double check `deployment-charts/helmfile.d/admin_ng` has correct values populated and the cluster enabled
[] Private puppet patches for controller manager token
[] The cergen configuration for `kubemaster.svc.eqiad.wmnet`
* Generate a list of all services (service FQDNs) for this DC (from namespaces or service.yaml) as $SERVICE_NAMES
```
apertium.svc.eqiad.wmnet
api-gateway.svc.eqiad.wmnet
blubberoid.svc.eqiad.wmnet
citoid.svc.eqiad.wmnet
cxserver.svc.eqiad.wmnet
echostore.svc.eqiad.wmnet
eventgate-analytics.svc.eqiad.wmnet
eventgate-analytics-external.svc.eqiad.wmnet
eventgate-logging-external.svc.eqiad.wmnet
eventgate-main.svc.eqiad.wmnet
eventstreams.svc.eqiad.wmnet
eventstreams-internal.svc.eqiad.wmnet
linkrecommendation.svc.eqiad.wmnet
mathoid.svc.eqiad.wmnet
mobileapps.svc.eqiad.wmnet
proton.svc.eqiad.wmnet
push-notifications.svc.eqiad.wmnet
recommendation-api.svc.eqiad.wmnet
sessionstore.svc.eqiad.wmnet
similar-users.svc.eqiad.wmnet
termbox.svc.eqiad.wmnet
wikifeeds.svc.eqiad.wmnet
```
== Actions ==
* Downtime masters and nodes
** `sudo cookbook sre.hosts.downtime -r 'Reinitialize eqiad k8s cluster with new etcd' -t T277741 -H 24 'A:eqiad and (A:kubernetes-masters or A:kubernetes-workers)'`
* Downtime all services in the cluster
~~** `for i in api-gateway blubberoid changeprop changeprop-jobqueue citoid cxserver echostore eventgate-analytics eventgate-analytics-external eventgate-logging-external eventgate-main eventstreams eventstreams-internal linkrecommendation mathoid mobileapps proton push-notifications recommendation-api sessionstore similar-users termbox wikifeeds zotero; do sudo cookbook sre.hosts.downtime -r 'Reinitialize eqiad k8s cluster' -t T277741 -H 24 $i.svc.eqiad.wmnet ; done`~~. This doesn't work, had to switch to the icinga UI.
* Cut traffic to all services in the cluster
~~** `sudo cookbook sre.discovery.service-route depool eqiad apertium api-gateway blubberoid citoid cxserver echostore eventgate-analytics eventgate-analytics-external eventgate-logging-external eventgate-main eventstreams eventstreams-internal linkrecommendation mathoid mobileapps proton push-notifications recommendation-api sessionstore similar-users termbox wikifeeds` ~~. This doesn't work, had to use confctl
* Switch restbase-async to eqiad:
** `sudo confctl --object discovery select "name=eqiad,dnsdisc=restbase-async" set/pooled=false && sudo confctl --object discovery select "name=eqiad,dnsdisc=restbase-async" set/pooled=true`
* Disable puppet on masters and nodes
** `sudo cumin 'A:eqiad and (A:kubernetes-masters or A:kubernetes-workers)' 'disable-puppet "Reinitializing cluster - T277741"'`
* Power them off too. ssh to ganeti01.svc.<site>.wmnet and
** `sudo gnt-instance shutdown -f argon.eqiad.wmnet`
** `sudo gnt-instance shutdown -f chroline.eqiad.wmnet`
* Regenerate the eqiad master cert using cergen.
** Revoke and remove the old cert on puppetmaster1001 with `sudo puppet cert clean kubemaster.svc.eqiad.wmnet`
** Run cergen `sudo cergen --base-path /srv/private/modules/secret/secrets/certificates --generate /srv/private/modules/secret/secrets/certificates/certificate.manifests.d`
** Copy certs to files/ssl
* Merge: https://gerrit.wikimedia.org/r/c/operations/puppet/+/671174
* Start reimaging nodes (checkmarks in T273279)
* Empty etcd (`ETCDCTL_API=3 etcdctl --endpoints https://foobar.site.wmnet:2379 del "" --from-key=true`)
* Reboot etcd servers (checkmarks in T273278)
* Merge: https://gerrit.wikimedia.org/r/c/operations/puppet/+/671171
* Set `profile::kubernetes::master::controllermanager_token: ... ` in private puppets `hieradata/role/eqiad/kubernetes/master.yaml`
* Image the new masters
* `helmfile sync` admin_ng
* Deploy all services (use deploy_all.sh script)
* Check all services (service-checker if possible)
* End downtime of services
* Decommission the old masters
== Action items ==
TBD