Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw
Open, NormalPublic

Description

In T203786 we raised the -R memcached parameter from 20 (default) to 200 to remove conn_yields spikes. We should apply the same parameter to all the mc memcached instances but this would require a restart of all the shards, so it needs to be coordinated (a restart implies wiping all the data).

  • mc1019
  • mc1020
  • mc1021
  • mc1022
  • mc1023
  • mc1024
  • mc1025
  • mc1026
  • mc1027
  • mc1028
  • mc1029
  • mc1030
  • mc1031
  • mc1032
  • mc1033
  • mc1034
  • mc1035
  • mc1036
  • mc20XX
elukey created this task.Nov 6 2018, 11:35 AM
elukey triaged this task as Normal priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 6 2018, 11:35 AM

Change 473669 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached on mc1019

https://gerrit.wikimedia.org/r/473669

Change 473669 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached on mc1019

https://gerrit.wikimedia.org/r/473669

Mentioned in SAL (#wikimedia-operations) [2018-11-15T07:08:23Z] <elukey> memcached on mc1019 restarted to apply -R 200 - T208844

elukey moved this task from Backlog to In Progress on the User-Elukey board.Fri, Nov 16, 3:03 PM

mc1019 recovered nicely, and I can confirm from https://grafana.wikimedia.org/dashboard/db/memcache?panelId=38&fullscreen&orgId=1&from=now-2d&to=now that conn_yields dropped to zero.

elukey updated the task description. (Show Details)Fri, Nov 16, 3:06 PM

Change 474670 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached on mc1020

https://gerrit.wikimedia.org/r/474670

Change 474670 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached on mc1020

https://gerrit.wikimedia.org/r/474670

Mentioned in SAL (#wikimedia-operations) [2018-11-19T11:20:14Z] <elukey> restart memcached on mc1020 to apply -R 200 settings (shard wiped) - T208844

elukey updated the task description. (Show Details)Mon, Nov 19, 11:20 AM

Change 475708 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] memcached: add -R 200 to mc1021

https://gerrit.wikimedia.org/r/475708

Change 475708 merged by Elukey:
[operations/puppet@production] memcached: add -R 200 to mc1021

https://gerrit.wikimedia.org/r/475708

Mentioned in SAL (#wikimedia-operations) [2018-11-26T07:36:58Z] <elukey> restart memcached on mc1021 (cache wipe) to add -R 200 - T208844

Change 476210 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached running on mc1022

https://gerrit.wikimedia.org/r/476210

Change 476210 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached running on mc1022

https://gerrit.wikimedia.org/r/476210

Mentioned in SAL (#wikimedia-operations) [2018-11-28T08:12:20Z] <elukey> apply -R 200 to memcached on mc1022 (cache wipe) - T208844

elukey updated the task description. (Show Details)Wed, Nov 28, 8:12 AM