Page MenuHomePhabricator
Create Task
Maniphest T244852

Upgrade and improve our application object caching service (memcached)
Closed, ResolvedPublic
Actions

Description

Overview
Our object caching service is based on mcrouter and memcached. Mcrouter is a memcached protocol router for scaling memcached. Currently, each mediawiki server is running an instance of mcrouter which in turn is configured with the same pool of memcached servers that constitute our pool.

Current Issues

  • When a shard becomes unavailable, we get TKOs which cause latency problems T203786 T208934 T239983
  • All memcached servers are on Debian Jessie, with its LTS support ending in June 2020
  • Redis is co-located in the same set of servers

To address the above issues, we will initially introduce a secondary pool of memcached servers, called gutter servers, capable of temporarily replacing any unavailable servers. This functionality will be provided by mcrouter. When we have the gutter servers in place and failover works, we can proceed with rolling upgrading all memcached servers to Debian buster. Since there have not been any major changes in the memcached protocol, we do not expect any major issues.

Another thing to take into account is mcrouter proxies. We have 4 mw servers in each DC which are used to replicate specific keys (dictated by mediawiki) from one datacentre to the other. We want to test the gutter pool functionality on the proxy level, i.e. define a secondary set of mcrouter proxy servers on each DC, where mcrouter will failover to in case a primary proxy is unavailable.

Lastly, developers are already working on completely retiring the use of Redis in Mediawiki, thus there will be no need to worry about its upgrade. (TBA links to related tasks)

Action Plan

  • Test gutter pool servers in beta
  • Test new memcached settings in beta
  • Image 6 new gutter servers (3 in eqiad, 3 in codfw)
  • Make relevant puppet changes to get gutter pool metrics
  • Make relevant puppet changes to support memcached on Debian Buster
  • Test gutter pool in production (mwdebug*)
  • Test proxy gutter pool in eqiad and/or codfw
  • Make relevant puppet changes to support the gutter pool configuration
  • Enable and test the gutter pool in canaries
  • Test memcached 1.5.x (buster) in canaries
  • Enable and test the gutter pool in production
  • Test onhost memcached
  • Deploy onhost memcached
  • Roll upgrade to buster in secondary datacenter (codfw)
  • Roll upgrade to buster in secondary datacenter (eqiad)
  • Upgrade memcached to version 1.6.x
  • Enable TLS

Related tasks: T203786

Reads:

Details

SubjectRepoBranchLines +/-
mcrouter_wancache: Clean up $use_gutter now that it's true everywhere.operations/puppetproduction+11 -25
mcrouter: enable the gutter pool everywhere.operations/puppetproduction+18 -0
mcrouter: enable failover route for on all canariesoperations/puppetproduction+6 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT244852 Upgrade and improve our application object caching service (memcached)
 ResolvedelukeyT208934 mcrouter does not remove a memcached shard from consistent hashing when timeouts happen
Unknown Object (Task)
 InvalidJclark-ctrT244886 (2020-01-15) rack/setup/install mc-gp100[123].eqiad.wmnet
 ResolvedelukeyT241795 (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet
Restricted Task
 ResolvedaaronT252564 Let WANObjectCache store "sister keys" on the same backend as the main value key
ResolvedelukeyT239249 (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet
 ResolvedjijikiT244340 Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached)
 ResolvedaaronT257003 Make memcached BagOStuff classes use 'gets' only if a CAS token is needed
 ResolvedNoneT263958 Test onhost memcached performance and functionality
 DeclinedaaronT264604 Enable "/*/mw-with-onhost-tier/" route for MediaWiki where safe
 ResolvedjijikiT273115 Migrate onhost memcached to use a unix socket
 DeclinedaaronT286531 Make WANObjectCache handle "lowTTL"/checkKeys/touchedCallback with "onHostRoutingPrefix" properly
 DeclinedNoneT244476 Test and deploy mcrouter 0.41
ResolvedKrinkleT240684 Test gutter pool failover in production and memcached 1.5.x
 ResolvedJoeT251378 Chaos Engineering - Stop for x hours one or more mc10xx memcached shards
 ResolvedjijikiT258779 Roll out remote-DC gutter pool for /*/mw-wan/
 DeclinedNoneT253384 add monitoring of sustained memcached TKO rates
 DeclinedNoneT227265 mcrouter codfw proxies sometimes lead to TKOs
 ResolvedJoeT255511 mcrouter memcached flapping in gutter pool
 ResolvedelukeyT225642 Allow async foreign set/delete WAN cache operations in mcrouter
 ResolvedaaronT254608 Monitor bytes written and read via WANObjectCache by key group
 ResolvedjijikiT258679 Recurrent TX bw saturation for mediawiki memcached shards
 ResolvedjijikiT213089 Upgrade memcached cluster to Debian Buster
 ResolvedelukeyT217020 Test different growth factors for memcached (prep step for upgrade to newer versions)
 ResolvedelukeyT223863 memkeys segfaults on Debian Stretch
 ResolvedjbondT251574 Package mcrouter 0.41 for Debian Buster
ResolvedjijikiT265643 Upgrade MediaWiki's Redis cluster to Debian Buster
 ResolvedjijikiT252391 Reimage one memcached shard per DC to Buster
 ResolvedhasharT267388 Puppet failure on deployment-memc08.deployment-prep.eqiad.wmflabs
ResolvedjijikiT271967 Enable TLS on memcached for cross-dc replication
 ResolvedjijikiT270315 Upgrade memcached to version 1.6.x

Event Timeline

jijiki created this task.Feb 11 2020, 12:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 11 2020, 12:51 PM
jijiki added a subtask: T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen.Feb 11 2020, 12:52 PM
jijiki added a subtask: T241795: (Need By: Jan 10) rack/setup/install mc-gp100[123].eqiad.wmnet.
jijiki added subtasks: Unknown Object (Task), Restricted Task, T213089: Upgrade memcached cluster to Debian Buster, T217020: Test different growth factors for memcached (prep step for upgrade to newer versions), T223863: memkeys segfaults on Debian Stretch, T239249: (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet, T244340: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached), T244476: Test and deploy mcrouter 0.41 .Feb 11 2020, 12:54 PM
jijiki triaged this task as Medium priority.Feb 11 2020, 12:57 PM
jijiki added projects: serviceops, SRE.
jijiki added subscribers: Ladsgroup, MoritzMuehlenhoff, Paladox and 4 others.
jijiki renamed this task from Upgrade and improve our application object caching service (memcachedd) to Upgrade and improve our application object caching service (memcached).Feb 11 2020, 1:51 PM
jijiki added a subtask: T240684: Test gutter pool failover in production and memcached 1.5.x.
RobH closed subtask Unknown Object (Task) as Resolved.Feb 18 2020, 10:35 PM
jijiki updated the task description. (Show Details)Feb 19 2020, 11:56 AM
jijiki changed the status of subtask T213089: Upgrade memcached cluster to Debian Buster from Open to Stalled.Feb 28 2020, 4:39 PM
Joe updated the task description. (Show Details)Apr 27 2020, 6:31 AM
gerritbot added a comment.Apr 27 2020, 6:36 AM

Change 592519 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mcrouter: enable failover route for on all canaries

https://gerrit.wikimedia.org/r/592519

gerritbot added a project: Patch-For-Review.Apr 27 2020, 6:36 AM

Change 592520 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mcrouter: enable the gutter pool everywhere.

https://gerrit.wikimedia.org/r/592520

gerritbot added a comment.Apr 27 2020, 7:10 AM

Change 592519 merged by Giuseppe Lavagetto:
[operations/puppet@production] mcrouter: enable failover route for on all canaries

https://gerrit.wikimedia.org/r/592519

JMeybohm subscribed.Apr 27 2020, 7:19 AM
gerritbot added a comment.Apr 28 2020, 2:43 PM

Change 592520 merged by Giuseppe Lavagetto:
[operations/puppet@production] mcrouter: enable the gutter pool everywhere.

https://gerrit.wikimedia.org/r/592520

Joe updated the task description. (Show Details)Apr 28 2020, 3:04 PM
gerritbot added a comment.May 4 2020, 5:06 PM

Change 594239 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] mediawiki: Clean up $use_gutter now that it's true everywhere.

https://gerrit.wikimedia.org/r/594239

gerritbot added a comment.May 5 2020, 5:46 PM

Change 594239 merged by RLazarus:
[operations/puppet@production] mcrouter_wancache: Clean up $use_gutter now that it's true everywhere.

https://gerrit.wikimedia.org/r/594239

Joe closed subtask T251378: Chaos Engineering - Stop for x hours one or more mc10xx memcached shards as Resolved.May 11 2020, 8:39 AM
elukey changed the status of subtask T252391: Reimage one memcached shard per DC to Buster from Open to Stalled.May 13 2020, 9:56 AM
elukey closed subtask T208934: mcrouter does not remove a memcached shard from consistent hashing when timeouts happen as Resolved.May 14 2020, 6:09 AM
elukey mentioned this in T240684: Test gutter pool failover in production and memcached 1.5.x.Jun 4 2020, 5:09 PM
Krinkle updated the task description. (Show Details)Jun 30 2020, 11:42 PM
jijiki added a subtask: T258779: Roll out remote-DC gutter pool for /*/mw-wan/.Jul 24 2020, 9:23 AM
aaron closed subtask T240684: Test gutter pool failover in production and memcached 1.5.x as Resolved.Jul 29 2020, 1:38 AM
jijiki moved this task from Incoming 🐫 to 🔦Unused2 on the serviceops board.Aug 17 2020, 11:46 PM
jijiki added a subtask: T253384: add monitoring of sustained memcached TKO rates.Sep 17 2020, 7:04 PM
jijiki added a subtask: T227265: mcrouter codfw proxies sometimes lead to TKOs.
jijiki added a subtask: T255511: mcrouter memcached flapping in gutter pool.Sep 17 2020, 7:08 PM
jijiki added a subtask: T225642: Allow async foreign set/delete WAN cache operations in mcrouter.
jijiki added a subtask: T254608: Monitor bytes written and read via WANObjectCache by key group.
jijiki added a subtask: T258679: Recurrent TX bw saturation for mediawiki memcached shards.Sep 17 2020, 7:13 PM
jijiki changed the status of subtask T252391: Reimage one memcached shard per DC to Buster from Stalled to Open.Oct 15 2020, 5:41 PM
jijiki removed a subtask: T252391: Reimage one memcached shard per DC to Buster.
jijiki removed a subtask: T213089: Upgrade memcached cluster to Debian Buster.Oct 15 2020, 5:44 PM
jijiki added a parent task: T213089: Upgrade memcached cluster to Debian Buster.
jijiki removed a parent task: T213089: Upgrade memcached cluster to Debian Buster.Oct 15 2020, 5:47 PM
jijiki added a subtask: T213089: Upgrade memcached cluster to Debian Buster.
jijiki updated the task description. (Show Details)
jijiki changed the status of subtask T213089: Upgrade memcached cluster to Debian Buster from Stalled to Open.Oct 20 2020, 8:44 PM
jijiki closed subtask T244340: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached) as Resolved.Dec 8 2020, 10:27 AM
jijiki added a subtask: T270315: Upgrade memcached to version 1.6.x.Dec 16 2020, 5:15 PM
jijiki updated the task description. (Show Details)
jijiki closed subtask T244476: Test and deploy mcrouter 0.41 as Declined.Dec 16 2020, 5:29 PM
jijiki closed subtask T258679: Recurrent TX bw saturation for mediawiki memcached shards as Resolved.
jijiki closed subtask T213089: Upgrade memcached cluster to Debian Buster as Resolved.Jan 13 2021, 3:34 PM
jijiki updated the task description. (Show Details)Jan 13 2021, 3:40 PM
jijiki added a subtask: T271967: Enable TLS on memcached for cross-dc replication.Jan 13 2021, 6:19 PM
Krinkle closed subtask Restricted Task as Resolved.Feb 5 2021, 10:24 PM
aaron closed subtask T254608: Monitor bytes written and read via WANObjectCache by key group as Resolved.Feb 22 2021, 7:14 PM
jijiki closed subtask T270315: Upgrade memcached to version 1.6.x as Resolved.Mar 24 2021, 5:33 PM
jijiki reopened subtask T270315: Upgrade memcached to version 1.6.x as Open.Apr 23 2021, 3:21 PM
jijiki closed subtask T271967: Enable TLS on memcached for cross-dc replication as Resolved.Jul 21 2021, 7:08 PM
jijiki closed subtask T227265: mcrouter codfw proxies sometimes lead to TKOs as Declined.Oct 15 2021, 2:41 PM
Krinkle added projects: Performance-Team (Radar), Sustainability (Incident Followup).Aug 23 2022, 1:54 PM
Krinkle moved this task from Limbo to Perf recommendation on the Performance-Team (Radar) board.
Krinkle mentioned this in T106066: Don't show "Nonce already used" error on memcache failure.Aug 23 2022, 1:56 PM
Krinkle moved this task from Perf recommendation to Watching on the Performance-Team (Radar) board.Oct 3 2022, 8:13 PM
jijiki moved this task from 🔦Unused2 to 💾 Datastores on the serviceops board.Nov 6 2022, 3:12 PM
jijiki moved this task from 💾 Datastores to 🗄 Projects on the serviceops board.Nov 9 2022, 12:54 PM
jijiki updated the task description. (Show Details)Nov 9 2022, 12:55 PM
jijiki closed subtask T258779: Roll out remote-DC gutter pool for /*/mw-wan/ as Resolved.Jan 9 2023, 6:02 PM
jijiki closed this task as Resolved.Jan 10 2023, 8:51 PM
jijiki closed subtask T270315: Upgrade memcached to version 1.6.x as Resolved.
jijiki updated the task description. (Show Details)
Joe closed subtask T253384: add monitoring of sustained memcached TKO rates as Declined.Mar 21 2023, 9:08 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 21 2023, 9:10 AM
Joe closed subtask T255511: mcrouter memcached flapping in gutter pool as Resolved.Mar 21 2023, 9:11 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL