We want to evaluate the practical performance implications of using a single, local, cache backend instead of spreading the whole datastet to multiple nodes with c-hashing.
In order to do so, we need to change the current Puppetization to allow taking one host out of c-hash, and use it exclusively as a local backend. This could be done by using a hiera setting, say `cache::single_backend_fqdn`, and excluding such hostname from the pool of backends for a given cache cluster/DC. Additionally, we need to use only localhost as the cache backend for the host `cache::single_backend_fqdn` itself.
Dashboards such as [[ https://grafana.wikimedia.org/d/A__2L7eWz/cache-hosts-comparison?orgId=1 | cache-hosts-comparison ]] can then be used to observe the impact on hitrate/ttfb on various nodes. By having N-1 backend nodes instead of N there may be some backend hitrate change across the board to take into account when interpreting the results.
The procedure for enabling the experiment on one node (`$host`) is as follows:
- Depool the host from all user traffic with `sudo -i depool`
- (Optional but good to evaluate hitrate): Stop `trafficserver.service`, empty ATS backend cache with `traffic_server -C clear_cache`, start `trafficserver.service`
- Set `cache::single_backend_fqdn: $host` in hiera for the DC/cluster the host is part of (eg: for host=cp4027, ulsfo/text)
- Run puppet on all cache nodes in the DC/cluster. Ensure that $host is removed from the list of backends on all varnish instances in the DC/cluster with `sudo -i varnishadm -n frontend backend.list`
- Ensure that varnish on $host points to localhost and that the node behaves well: https://wikitech.wikimedia.org/wiki/Varnish#Force_your_requests_through_a_specific_Varnish_frontend
- Repool the host for user traffic with `sudo -i pool`. Ensure that $host is not listed in `/etc/varnish/directors.frontend.vcl` on any DC/cluster node
Disabling the experiment:
- Depool the host from all user traffic with `sudo -i depool`
- Unset `cache::single_backend_fqdn` in hiera for the DC/cluster the host is part of
- Run puppet on all cache nodes in the DC/cluster. Ensure that $host is added to the list of backends on all varnish instances in the DC/cluster with `sudo -i varnishadm -n frontend backend.list`
- Ensure that varnish on $host points to all nodes in the DC/cluster and that the node behaves well: https://wikitech.wikimedia.org/wiki/Varnish#Force_your_requests_through_a_specific_Varnish_frontend
- Repool the host for user traffic with `sudo -i pool`. Ensure that $host is listed in `/etc/varnish/directors.frontend.vcl` on all DC/cluster nodes