Page MenuHomePhabricator

Apply interface::rps to all the mc hosts
Closed, ResolvedPublic


In the parent task, mc1035 was configured with interface::rps as attempt to mitigate the effects of traffic bursts on Nov 5th with

There is another code review open to extend the same setting to the other nodes ( but before applying it some review needs to be done. Two graphs follows with node_network_receive_drop trends over the past weeks:

Screen Shot 2018-11-14 at 2.56.13 PM.png (734×2 px, 105 KB)

Screen Shot 2018-11-14 at 2.55.49 PM.png (931×2 px, 270 KB)

The first one is mc1035 only, and the major peaks corresponds to the increase in traffic that we are trying to solve in the parent task. The second one is about the rest of the nodes.

Event Timeline

elukey triaged this task as Medium priority.Nov 14 2018, 1:58 PM
elukey created this task.

@BBlack do you have any suggestion about things to check / precautions to take before enabling interface::rps to all the memcached shards? I am asking since it is a very delicate piece of infra and I'd like to have a more expert opinion from somebody more expert in the subject :)

Change 478198 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply interface::rps to mc1022

Change 478198 abandoned by Elukey:
Apply interface::rps to mc1022

Done today after merging

More info about timings in the parent task.