The following are plans to make clouddumps reliability better, with priority TBD and the following goals:
1. clouddumps hosts reboots should be a non-event
1. Failover should be simple, quick, reliable and straightforward to do
The current situation is that clouddumps serve these purposes: NFS server (read only) towards Cloud VPS and production (wdqs, stat hosts), plus web serving of `dumps.w.o`. The active host for each is controlled by Puppet via `dumps_dist_active_vps` and `dumps_dist_active_web` respectively; failover is performed according to https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Dumps#Failover .
==== 1. Load balancer (selected solution)
This plan involves fronting clouddumps with a standard lvs load balancer in active/passive fashion. Clients reach the server via a public IP (either shared with existing http address or not, TBD). To make things easier only nfsv4 is going to be supported, meaning only tcp/2049 port is needed. Failover happens via `confctl` like any other load balanced service. Upon failover clients will reconnect to the server and applications may experience `ESTALE` (Stale file handle) errors as nfs sessions are re-established.
==== 2. VIP announced from the hosts
I (Filippo) propose the following: clouddumps service refers to a new single floating IP address (e.g. announced by BGP on the host) that lives on the active server. In this scenario failover happens on the server, as opposed to clients, and is still controlled by Puppet. When failover happens, NFS clients will reconnect automatically once the NFS server moves over, and similarly once the IP moves then clients will follow (i.e. no CNAME flipping). NFS clients will mount a single NFS share pointing to said IP, and ditto for web clients from `dumps.w.o`.
In practice failover would look like the following:
1. We flip the puppet variable to point to the new active host
1. Puppet runs on the now-active host, IP moves and the NFS server is started. Puppet enables the required systemd units to be active now and at boot
1. Puppet runs on the previously-active host, stops NFS server and stops announcing the IP address. Also disables systemd units so they don't come up at boot
There is an obvious race between puppet runs, which we break by making IP announcement + NFS server start depend on being able to acquire an etcd lock. We effectively serialize Puppet runs to wait for the lock to be released on the previously-active host before moving over the IP address and start NFS server. Doing serialization this way also takes care of the situation when the active host is hard-down for long periods of time e.g. hardware failure and we failover while the active host is down:
1. Active host is hard-down
1. We failover in puppet, the lock can be acquired and thus IP + NFS move over on the now-active host
1. At some point the previously-active host comes back before Puppet has a chance to run, tries to acquire the lock and fails
1. Puppet now has a chance to run on the host that just came back and disables the IP + NFS server units for good
We do depend on existing etcd for failover to happen, although I think that is acceptable given etcd's track record in production. We will also devise mechanisms to easily override the lock in case of manual failover.
As far as reboots are concerned, rebooting the passive host does not affect clients by definition. While rebooting the active host should result in a brief downtime for clients, which will resolve itself once the host is back. Assuming we want to "eat" the down time and not perform a failover when we need to do rolling reboots.
For additional context, I took inspiration from the problems outlined in {T391369}
=== Conclusion
For standard/simplicity reasons we'll go for the load balancer plan. This will also enable us to complete {T417028} (i.e. clouddumps hosts won't require public IPs anymore).
=== Implementation
* [x] Pick a public IP address to use for `dumps-nfs.wikimedia.org` and add the DNS record
* [x] Set up a `dumps-nfs` LVS service on tcp port 2049, backed by clouddumps hosts
* [ ] Select a subset of hosts in production (wdqs, stat hosts currently mount dumps) and cloudvps for a trial with the new name. These hosts will have the following hiera set: `dumps_dist_nfs_servers: [clouddumps1001.wikimedia.org, clouddumps1002.wikimedia.org, dumps-nfs.wikimedia.org]`
* [ ] Flip the trial hosts to the new server: `dumps_dist_active_vps: dumps-nfs.wikimedia.org` + `dumps_dist_active_web: dumps-nfs.wikimedia.org` to control cloudvps and prod respectively
* [ ] Carry out failover tests and validate linux nfs client behavior
* [ ] Once validated, add `dumps-nfs.w.o` to `dumps_dist_nfs_servers` everywhere, then flip to it by changing `dumps_dist_active_vps` and `dumps_dist_active_web`
* [ ] Carry out failover tests when all clients are in
cc stakeholders: #netops #traffic #data-platform-sre