**Overview**
Our object caching service is based on mcrouter and memcached. Mcrouter is a memcached protocol router for scaling memcached. Currently, each mediawiki server is running an instance of mcrouter which in turn is configured with the same pool of memcached servers that constitute our pool.
**Current Issues**
* When a shard becomes unavailable, we get TKOs which cause latency problems T203786 T208934 T239983
* All memcached servers are on Debian Jessie, with its LTS support ending in June 2020
To address the above issues, we will initially introduce a secondary pool of memcached servers, called gutter servers, capable of temporarily replacing any unavailable servers. This functionality will be provided by mcrouter. When we have the gutter servers in place and failover works, we can proceed with rolling upgrading all memcached servers to Debian buster. Since there have not been any major changes in the memcached protocol, we do not expect any major issues
**Action Plan**
[] Test gutter pool servers in beta
[] Make relevant puppet changes to support memcached on Debian Buster
[] Make relevant puppet changes to support the gutter pool configuration
[] Image 6 new gutter servers (3 in eqiad, 3 in codfw)
[] Enable and test the gutter pool in secondary datacente (codfw)
[] Enable and test the gutter pool in primary datacentre (eqiad)
[] Roll upgrade to buster in secondary datacente (codfw)
[] Roll upgrade to buster in secondary datacente (eqiad)