Page MenuHomePhabricator

Bump memcache connections and swift-proxy limits
Closed, ResolvedPublic

Description

In parent task I debugged/investigated auth failures in thanos-swift codfw, turns out the following is happening:

  • The failures were prevalent on thanos-fe2001
  • This host also runs Thanos "sidecar" processes, i.e. compact and rule
  • The additional connections were maxing out the swift-proxy memcache connection limit, resulting in auth failures

Therefore we'll be:

  • Bumping max connections in memcached
  • Bumping max per-worker memcache limits in swift-proxy

Event Timeline

Change 822039 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] memcached: point to active/used configuration options

https://gerrit.wikimedia.org/r/822039

Change 822040 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] swift: bump proxy memcache max connections

https://gerrit.wikimedia.org/r/822040

BCornwall triaged this task as Medium priority.Aug 10 2022, 5:57 PM

Change 822039 merged by Filippo Giunchedi:

[operations/puppet@production] memcached: point to active/used configuration options

https://gerrit.wikimedia.org/r/822039

Change 822040 merged by Filippo Giunchedi:

[operations/puppet@production] swift: bump proxy memcache max connections

https://gerrit.wikimedia.org/r/822040

Mentioned in SAL (#wikimedia-operations) [2022-08-18T07:26:27Z] <godog> roll-restart swift-proxy to apply bumbed memcached limits T314914

fgiunchedi claimed this task.

This has been deployed!