We want to evaluate the practical performance implications of using a single, local, cache backend instead of spreading the whole datastet to multiple nodes with c-hashing.
In order to do so, we need to change the current Puppetization to allow taking one host out of c-hash, and use it exclusively as a local backend. This could be done by using a hiera setting, say cache::single_backend_fqdn, and excluding such hostname from the pool of backends for a given cache cluster/DC. Additionally, we need to use only localhost as the cache backend for the host cache::single_backend_fqdn itself.
Dashboards such as cache-hosts-comparison can then be used to observe the impact on hitrate/ttfb on various nodes. By having N-1 backend nodes instead of N there may be some backend hitrate change across the board to take into account when interpreting the results.
The procedure for enabling the experiment on one node ($host) is as follows:
- Depool the host from all user traffic with sudo -i depool
- (Optional but good to evaluate hitrate): Stop trafficserver.service, empty ATS backend cache with traffic_server -C clear_cache, start trafficserver.service
- Set cache::single_backend_fqdn: $host in hiera for the DC/cluster the host is part of (eg: for host=cp4027, ulsfo/text)
- Run puppet on all cache nodes in the DC/cluster. Ensure that $host is removed from the list of backends on all varnish instances in the DC/cluster with sudo -i varnishadm -n frontend backend.list
- Ensure that varnish on $host points to localhost and that the node behaves well: https://wikitech.wikimedia.org/wiki/Varnish#Force_your_requests_through_a_specific_Varnish_frontend
- Repool the host for user traffic with sudo -i pool. Ensure that $host is not listed in /etc/varnish/directors.frontend.vcl on any DC/cluster node
Disabling the experiment:
- Depool the host from all user traffic with sudo -i depool
- Unset cache::single_backend_fqdn in hiera for the DC/cluster the host is part of
- Run puppet on all cache nodes in the DC/cluster. Ensure that $host is added to the list of backends on all varnish instances in the DC/cluster with sudo -i varnishadm -n frontend backend.list
- Ensure that varnish on $host points to all nodes in the DC/cluster and that the node behaves well: https://wikitech.wikimedia.org/wiki/Varnish#Force_your_requests_through_a_specific_Varnish_frontend
- Repool the host for user traffic with sudo -i pool. Ensure that $host is listed in /etc/varnish/directors.frontend.vcl on all DC/cluster nodes
