Page MenuHomePhabricator

Apply interface::rps to all the mc hosts
Closed, ResolvedPublic

Description

In the parent task, mc1035 was configured with interface::rps as attempt to mitigate the effects of traffic bursts on Nov 5th with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/471234/

There is another code review open to extend the same setting to the other nodes (https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/472099/) but before applying it some review needs to be done. Two graphs follows with node_network_receive_drop trends over the past weeks:

The first one is mc1035 only, and the major peaks corresponds to the increase in traffic that we are trying to solve in the parent task. The second one is about the rest of the nodes.

Details

Related Gerrit Patches:
operations/puppet : productionApply interface::rps to mc1022

Event Timeline

elukey triaged this task as Medium priority.Nov 14 2018, 1:58 PM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 14 2018, 1:58 PM
elukey moved this task from Backlog to In Progress on the User-Elukey board.Nov 20 2018, 10:01 AM
elukey added a subscriber: BBlack.Nov 23 2018, 4:03 PM

@BBlack do you have any suggestion about things to check / precautions to take before enabling interface::rps to all the memcached shards? I am asking since it is a very delicate piece of infra and I'd like to have a more expert opinion from somebody more expert in the subject :)

Change 478198 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply interface::rps to mc1022

https://gerrit.wikimedia.org/r/478198

Change 478198 abandoned by Elukey:
Apply interface::rps to mc1022

https://gerrit.wikimedia.org/r/478198

jijiki added a subscriber: jijiki.Dec 12 2018, 4:56 PM
elukey moved this task from In Progress to Backlog on the User-Elukey board.Mar 4 2019, 9:34 AM
elukey closed this task as Resolved.Mar 27 2019, 6:14 PM

Done today after merging https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/472099/

More info about timings in the parent task.