Blockers:
- T144255 Migrate tools and misc data to labstore1004
- T144633 Setup monitoring for secondary HA cluster
- T127567 revise/fix labstore replicate backup jobs
- T146153 Performance test new secondary labstore HA cluster
### MIGRATION PLAN
#### Prior to the maintenance window
[x] Set up NFS kernel server on labstore1004/5 (DRBD Primary) [DONE]
[x] Define tools as a labstore-secondary mount [DONE]
[] Mount tools from labstore1001 and labstore-secondary simultaneously (need to add tools as a new mount name in nfs-mounts.yaml)
#### During migration:
[] Update lists/irc channel on start of migration
[] Silence shinken alarms
[] Disable puppet across tools
[] K8S Master - Unmount NFS, stop process that depends on NFS
[] Grid Master/Shadow - Stop the grid master process
[] Deploy webservice debian package that sends logs to /dev/null (T149946)
[] Restart all tools-webgrid* nodes to apply new webservice package
[] Make tools share on labstore1001 Read Only
[] Snapshot and latest sync of the tools share
[] Run nfs-exportd on labstore-secondary to make sure the mount is exported to all tools hosts <== comes w/ nfs-manage up
[] Merge gerrit patch to remove the mount from 1001 (Removing defn from nfs-mount.yaml), and symlink mount path - /mnt/nfs/labstore-secondary-tools/project to /data/project and /mnt/nfs/labstore-secondary-tools/home to /home on tools
[] Enable puppet on few tools hosts at a time and roll out above patch to each node by running puppet. Reboot each exec node after running puppet.
#### Post Migration
[] Follow up on lists/irc
### Notes
#### List of nodes with nfs mounted
- Trusty grid exec nodes : tools-exec-1401 - 1420
- Precise grid exec nodes: tools-exec-1201 - 1221
- Trusty grid webgrid-lighttpd nodes: tools-webgrid-lighttpd-1401 - 1418
- Precise grid webgrid-lighttpd nodes: tools-webgrid-lighttpd-1201 - 1210
- Trusty grid webgrid-generic nodes: tools-webgrid-generic-1401 - 1404
- K8S worker nodes: tools-worker-1001 - 1025
- Tools static nodes: tools-static-10 - 11
- Tools bastions: tools-bastion-02, tools-bastion-03, tools-bastion-05
- Other grid submit nodes: tools-cron-01, tools-mail-01, tools-exec-gift, tools-exec-cyberbot, tools-mail
- Grid master/shadow: tools-grid-master, tools-grid-shadow
- K8S master: tools-k8s-master-01
- Tools-checker: tools-checker-01, tools-checker-02
- Tools services: tools-services-01, tools-services-02
- Other: tools-precise-dev
#### NFS not mounted
- tools-puppetmaster-01
- tools-puppetmaster-02
- tools-prometheus-01
- tools-prometheus-02
- tools-logs-02
- tools-docker-builder-01
- tools-docker-builder-02
- tools-docker-registry-01
- tools-flannel-etcd-03
- tools-flannel-etcd-02
- tools-flannel-etcd-01
- tools-k8s-etcd-03
- tools-k8s-etcd-02
- tools-k8s-etcd-01
- tools-redis-1002
- tools-redis-1001
- tools-elastic-03
- tools-elastic-02
- tools-elastic-01
- tools-proxy-02
- tools-proxy-01