Page MenuHomePhabricator

Reduce swift frontend conntrack usage
Closed, ResolvedPublic

Description

The "conntrack table full" alert has been firing/flapping for ms-fe1005, though other frontends also have a large number of entries in conntrack.

At the very least the internal swift traffic frontend/backend (i.e. port 600[012] tcp) should be NOTRACK. Ditto for stats traffic on localhost port 8125 udp.

Event Timeline

Note that statsd and swift account for the majority of entries in conntrack.

statsd needs to be explicitly excluded. For swift traffic the "direction" in the ferm rules needs to be reversed on the frontends since we assume NO_TRACK is deployed on the machines hosting a service, whereas in the swift frontend -> backend communication we're interested in excluding traffic directed to port 6000, not originating from it.

root@ms-fe1006:/etc/ferm# conntrack -L | grep -c -e port=600. -e port=8125
conntrack v1.4.4 (conntrack-tools): 152804 flow entries have been shown.
143931

Change 373032 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] statsite: don't track statsd udp traffic

https://gerrit.wikimedia.org/r/373032

Change 373032 merged by Filippo Giunchedi:
[operations/puppet@production] statsite: don't track statsd udp traffic

https://gerrit.wikimedia.org/r/373032

Change 373038 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] ferm: introduce ferm::client

https://gerrit.wikimedia.org/r/373038

Change 373039 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] swift: don't track client connections in frontend

https://gerrit.wikimedia.org/r/373039

Change 373253 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] statsite: don't track statsd client traffic

https://gerrit.wikimedia.org/r/373253

Change 373038 merged by Filippo Giunchedi:
[operations/puppet@production] ferm: introduce ferm::client

https://gerrit.wikimedia.org/r/373038

Change 373253 merged by Filippo Giunchedi:
[operations/puppet@production] statsite: don't track statsd client traffic

https://gerrit.wikimedia.org/r/373253

Change 373039 merged by Filippo Giunchedi:
[operations/puppet@production] swift: don't track connections to swift backend services on frontend machines

https://gerrit.wikimedia.org/r/373039

Change 374169 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] ferm: add return traffic for ferm::client notrack

https://gerrit.wikimedia.org/r/374169

Change 374170 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] swift: don't track connections to swift backend services on frontend machines

https://gerrit.wikimedia.org/r/374170

Change 374169 merged by Filippo Giunchedi:
[operations/puppet@production] ferm: add return traffic for ferm::client notrack

https://gerrit.wikimedia.org/r/374169

Change 374170 merged by Filippo Giunchedi:
[operations/puppet@production] swift: don't track connections to swift backend services on frontend machines

https://gerrit.wikimedia.org/r/374170

Mentioned in SAL (#wikimedia-operations) [2017-09-26T08:44:43Z] <godog> run puppet on ms-fe* to reduce conntrack generated by swift internal clients towards backends - T173731

We're now explicitly excluding statsite traffic and swift clients running on the proxy that talk to backend swift:

# conntrack -L | grep -c -e port=600. -e port=8125
conntrack v1.4.4 (conntrack-tools): 9701 flow entries have been shown.
59