Page MenuHomePhabricator

Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw
Closed, ResolvedPublic

Description

In T203786 we raised the -R memcached parameter from 20 (default) to 200 to remove conn_yields spikes. We should apply the same parameter to all the mc memcached instances but this would require a restart of all the shards, so it needs to be coordinated (a restart implies wiping all the data).

We have pushed the -R 200 flag to all memcached hosts, the change will be slowly propagate as we reboot servers.

Event Timeline

elukey triaged this task as Normal priority.Nov 6 2018, 11:35 AM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 6 2018, 11:35 AM

Change 473669 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached on mc1019

https://gerrit.wikimedia.org/r/473669

Change 473669 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached on mc1019

https://gerrit.wikimedia.org/r/473669

Mentioned in SAL (#wikimedia-operations) [2018-11-15T07:08:23Z] <elukey> memcached on mc1019 restarted to apply -R 200 - T208844

elukey moved this task from Backlog to In Progress on the User-Elukey board.Nov 16 2018, 3:03 PM

mc1019 recovered nicely, and I can confirm from https://grafana.wikimedia.org/dashboard/db/memcache?panelId=38&fullscreen&orgId=1&from=now-2d&to=now that conn_yields dropped to zero.

elukey updated the task description. (Show Details)Nov 16 2018, 3:06 PM

Change 474670 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached on mc1020

https://gerrit.wikimedia.org/r/474670

Change 474670 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached on mc1020

https://gerrit.wikimedia.org/r/474670

Mentioned in SAL (#wikimedia-operations) [2018-11-19T11:20:14Z] <elukey> restart memcached on mc1020 to apply -R 200 settings (shard wiped) - T208844

elukey updated the task description. (Show Details)Nov 19 2018, 11:20 AM

Change 475708 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] memcached: add -R 200 to mc1021

https://gerrit.wikimedia.org/r/475708

Change 475708 merged by Elukey:
[operations/puppet@production] memcached: add -R 200 to mc1021

https://gerrit.wikimedia.org/r/475708

Mentioned in SAL (#wikimedia-operations) [2018-11-26T07:36:58Z] <elukey> restart memcached on mc1021 (cache wipe) to add -R 200 - T208844

Change 476210 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached running on mc1022

https://gerrit.wikimedia.org/r/476210

Change 476210 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached running on mc1022

https://gerrit.wikimedia.org/r/476210

Mentioned in SAL (#wikimedia-operations) [2018-11-28T08:12:20Z] <elukey> apply -R 200 to memcached on mc1022 (cache wipe) - T208844

elukey updated the task description. (Show Details)Nov 28 2018, 8:12 AM

Change 481996 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add -R 200 to memcached on mc1023

https://gerrit.wikimedia.org/r/481996

Change 481996 merged by Elukey:
[operations/puppet@production] Add -R 200 to memcached on mc1023

https://gerrit.wikimedia.org/r/481996

Mentioned in SAL (#wikimedia-operations) [2019-01-03T09:51:19Z] <elukey> restart memcached on mc1023 to apply -R 200 - T208844

elukey updated the task description. (Show Details)Jan 3 2019, 9:51 AM
elukey moved this task from In Progress to Stalled on the User-Elukey board.Jan 11 2019, 7:40 AM

Change 484401 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply -R 200 to memcached on mc1024

https://gerrit.wikimedia.org/r/484401

Change 484401 merged by Elukey:
[operations/puppet@production] Apply -R 200 to memcached on mc1024

https://gerrit.wikimedia.org/r/484401

Mentioned in SAL (#wikimedia-operations) [2019-01-15T13:00:47Z] <elukey> restart memcached on mc1024 to pick up new settings (-R 200) - T208844

elukey updated the task description. (Show Details)Jan 17 2019, 7:23 AM
jijiki added a subscriber: jijiki.Jan 17 2019, 7:43 AM

Change 485172 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Apply -R 200 to memcached on mc1025

https://gerrit.wikimedia.org/r/485172

jijiki updated the task description. (Show Details)Jan 18 2019, 11:39 AM
jijiki moved this task from Backlog to Doing on the serviceops board.Jan 18 2019, 12:02 PM

Change 485172 merged by Effie Mouzeli:
[operations/puppet@production] Apply -R 200 to memcached on mc1025

https://gerrit.wikimedia.org/r/485172

Mentioned in SAL (#wikimedia-operations) [2019-01-21T12:36:05Z] <jijiki> Restarting memcached on mc1025 to apply '-R 200' - T208844

jijiki updated the task description. (Show Details)Jan 21 2019, 12:37 PM

Change 489175 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Apply -R 200 to memcached on mc1026

https://gerrit.wikimedia.org/r/489175

Mentioned in SAL (#wikimedia-operations) [2019-02-08T10:27:19Z] <jijiki> Restarting memcached on mc1026 to apply '-R 200' - T208844

Change 489175 merged by Effie Mouzeli:
[operations/puppet@production] Apply -R 200 to memcached on mc1026

https://gerrit.wikimedia.org/r/489175

jijiki updated the task description. (Show Details)Feb 8 2019, 10:31 AM
elukey reassigned this task from elukey to jijiki.Feb 9 2019, 2:41 PM

Change 491541 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Apply -R 200 to memcached on mc1027

https://gerrit.wikimedia.org/r/491541

Change 491541 merged by Effie Mouzeli:
[operations/puppet@production] Apply -R 200 to memcached on mc1027

https://gerrit.wikimedia.org/r/491541

Mentioned in SAL (#wikimedia-operations) [2019-02-19T17:52:43Z] <jijiki> Restarting memcache on mc1027 - T208844

jijiki updated the task description. (Show Details)Feb 19 2019, 5:54 PM

Change 493060 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Apply -R 200 to memcached on mc1028

https://gerrit.wikimedia.org/r/493060

Change 493060 merged by Effie Mouzeli:
[operations/puppet@production] Apply -R 200 to memcached on mc1028

https://gerrit.wikimedia.org/r/493060

Mentioned in SAL (#wikimedia-operations) [2019-02-26T16:24:37Z] <jijiki> Restarting memcached on mc1028 - T208844

elukey moved this task from Stalled to Keep an eye on it on the User-Elukey board.Feb 27 2019, 9:15 AM
jijiki moved this task from In Progress to Pio Kato on the User-jijiki board.Apr 4 2019, 9:22 PM
jijiki moved this task from Pio Kato to In Progress on the User-jijiki board.Apr 23 2019, 5:34 PM

Change 505839 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Apply -R 200 to memcached on mc1029

https://gerrit.wikimedia.org/r/505839

jijiki updated the task description. (Show Details)Apr 23 2019, 5:37 PM

Change 505839 merged by Effie Mouzeli:
[operations/puppet@production] Apply -R 200 to memcached on mc1029

https://gerrit.wikimedia.org/r/505839

Mentioned in SAL (#wikimedia-operations) [2019-04-23T17:43:40Z] <jijiki> Restarting memcached on mc1029 - T208844

Change 511973 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] profile::memcached::instance: Add -R 200 option

https://gerrit.wikimedia.org/r/511973

Change 511973 merged by Effie Mouzeli:
[operations/puppet@production] profile::memcached::instance: Add -R 200 option

https://gerrit.wikimedia.org/r/511973

Change 514243 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hiera::hosts: Remove host configs for memcached

https://gerrit.wikimedia.org/r/514243

Change 514243 merged by Effie Mouzeli:
[operations/puppet@production] hiera::hosts: Remove host configs for memcached

https://gerrit.wikimedia.org/r/514243

jijiki closed this task as Resolved.Jun 4 2019, 11:51 AM
jijiki updated the task description. (Show Details)

This task will be completed after the next round of reboots for the mc* hosts (as FYI for anybody interested).