In parent task I debugged/investigated auth failures in thanos-swift codfw, turns out the following is happening:
- The failures were prevalent on thanos-fe2001
- This host also runs Thanos "sidecar" processes, i.e. compact and rule
- The additional connections were maxing out the swift-proxy memcache connection limit, resulting in auth failures
Therefore we'll be:
- Bumping max connections in memcached
- Bumping max per-worker memcache limits in swift-proxy