Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw
Open, NormalPublic

Description

In T203786 we raised the -R memcached parameter from 20 (default) to 200 to remove conn_yields spikes. We should apply the same parameter to all the mc memcached instances but this would require a restart of all the shards, so it needs to be coordinated (a restart implies wiping all the data).

eqiad

  • mc1019
  • mc1020
  • mc1021
  • mc1022
  • mc1023
  • mc1024
  • mc1025
  • mc1026
  • mc1027
  • mc1028
  • mc1029
  • mc1030
  • mc1031
  • mc1032
  • mc1033
  • mc1034
  • mc1035
  • mc1036

codfw

  • mc2019
  • mc2020
  • mc2021
  • mc2022
  • mc2023
  • mc2024
  • mc2025
  • mc2026
  • mc2027
  • mc2028
  • mc2029
  • mc2030
  • mc2031
  • mc2032
  • mc2033
  • mc2034
  • mc2035
  • mc2036
elukey created this task.Nov 6 2018, 11:35 AM
elukey triaged this task as Normal priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 6 2018, 11:35 AM

Change 473669 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached on mc1019

https://gerrit.wikimedia.org/r/473669

Change 473669 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached on mc1019

https://gerrit.wikimedia.org/r/473669

Mentioned in SAL (#wikimedia-operations) [2018-11-15T07:08:23Z] <elukey> memcached on mc1019 restarted to apply -R 200 - T208844

elukey moved this task from Backlog to In Progress on the User-Elukey board.Nov 16 2018, 3:03 PM

mc1019 recovered nicely, and I can confirm from https://grafana.wikimedia.org/dashboard/db/memcache?panelId=38&fullscreen&orgId=1&from=now-2d&to=now that conn_yields dropped to zero.

elukey updated the task description. (Show Details)Nov 16 2018, 3:06 PM

Change 474670 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached on mc1020

https://gerrit.wikimedia.org/r/474670

Change 474670 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached on mc1020

https://gerrit.wikimedia.org/r/474670

Mentioned in SAL (#wikimedia-operations) [2018-11-19T11:20:14Z] <elukey> restart memcached on mc1020 to apply -R 200 settings (shard wiped) - T208844

elukey updated the task description. (Show Details)Nov 19 2018, 11:20 AM

Change 475708 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] memcached: add -R 200 to mc1021

https://gerrit.wikimedia.org/r/475708

Change 475708 merged by Elukey:
[operations/puppet@production] memcached: add -R 200 to mc1021

https://gerrit.wikimedia.org/r/475708

Mentioned in SAL (#wikimedia-operations) [2018-11-26T07:36:58Z] <elukey> restart memcached on mc1021 (cache wipe) to add -R 200 - T208844

Change 476210 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached running on mc1022

https://gerrit.wikimedia.org/r/476210

Change 476210 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached running on mc1022

https://gerrit.wikimedia.org/r/476210

Mentioned in SAL (#wikimedia-operations) [2018-11-28T08:12:20Z] <elukey> apply -R 200 to memcached on mc1022 (cache wipe) - T208844

elukey updated the task description. (Show Details)Nov 28 2018, 8:12 AM

Change 481996 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add -R 200 to memcached on mc1023

https://gerrit.wikimedia.org/r/481996

Change 481996 merged by Elukey:
[operations/puppet@production] Add -R 200 to memcached on mc1023

https://gerrit.wikimedia.org/r/481996

Mentioned in SAL (#wikimedia-operations) [2019-01-03T09:51:19Z] <elukey> restart memcached on mc1023 to apply -R 200 - T208844

elukey updated the task description. (Show Details)Jan 3 2019, 9:51 AM
elukey moved this task from In Progress to Stalled on the User-Elukey board.Jan 11 2019, 7:40 AM

Change 484401 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached on mc1024

https://gerrit.wikimedia.org/r/484401

Change 484401 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached on mc1024

https://gerrit.wikimedia.org/r/484401

Mentioned in SAL (#wikimedia-operations) [2019-01-15T13:00:47Z] <elukey> restart memcached on mc1024 to pick up new settings (-R 200) - T208844

elukey updated the task description. (Show Details)Jan 17 2019, 7:23 AM
jijiki added a subscriber: jijiki.Jan 17 2019, 7:43 AM

Change 485172 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Apply -R 200 to memcached on mc1025

https://gerrit.wikimedia.org/r/485172

jijiki updated the task description. (Show Details)Fri, Jan 18, 11:39 AM
jijiki moved this task from Backlog to Doing on the serviceops board.Fri, Jan 18, 12:02 PM
jijiki moved this task from Backlog to In Progress on the User-jijiki board.Mon, Jan 21, 12:33 PM

Change 485172 merged by Effie Mouzeli:
[operations/puppet@production] Apply -R 200 to memcached on mc1025

https://gerrit.wikimedia.org/r/485172

Mentioned in SAL (#wikimedia-operations) [2019-01-21T12:36:05Z] <jijiki> Restarting memcached on mc1025 to apply '-R 200' - T208844

jijiki updated the task description. (Show Details)Mon, Jan 21, 12:37 PM

Change 489175 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Apply -R 200 to memcached on mc1026

https://gerrit.wikimedia.org/r/489175

Mentioned in SAL (#wikimedia-operations) [2019-02-08T10:27:19Z] <jijiki> Restarting memcached on mc1026 to apply '-R 200' - T208844

Change 489175 merged by Effie Mouzeli:
[operations/puppet@production] Apply -R 200 to memcached on mc1026

https://gerrit.wikimedia.org/r/489175

jijiki updated the task description. (Show Details)Fri, Feb 8, 10:31 AM
elukey reassigned this task from elukey to jijiki.Sat, Feb 9, 2:41 PM