Page MenuHomePhabricator

Deploy mcrouter to production as a wancache backend
Closed, ResolvedPublic

Description

Having cross-datacenter consistency between WANObjectCache objects is one of the prerequisites for being able to serve MediaWiki from multiple datacenters.

This means basically substituting the current twemproxy solution for aggregating multiple memcached instances in a single cluster with a software that allows to broadcast/replicate writes

To this end, we want to use Facebook's mcrouter T156938

We have debian packages built, and a basic module that supports a single-datacenter installation in deployment-prep. The production installation, though, will need some work on our end.

Specifically, we need to figure out a suitable architecture for cross-dc to work, add proper monitoring of the software, and execute the transition from twemproxy to mcrouter in a secure way.

Event Timeline

Joe created this task.Apr 17 2018, 3:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 17 2018, 3:15 PM
Imarlier moved this task from Inbox to Radar on the Performance-Team board.Apr 23 2018, 8:22 PM
Imarlier edited projects, added Performance-Team (Radar); removed Performance-Team.

Change 431736 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mcrouter: add support for listening on the ssl port

https://gerrit.wikimedia.org/r/431736

Change 431737 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::mcrouter_wancache: add ssl, proxy support

https://gerrit.wikimedia.org/r/431737

Change 431738 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] puppet_ecdsacert: allow IP-based SANs

https://gerrit.wikimedia.org/r/431738

Change 431736 merged by Giuseppe Lavagetto:
[operations/puppet@production] mcrouter: add support for listening on the ssl port

https://gerrit.wikimedia.org/r/431736

Change 431737 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::mcrouter_wancache: add ssl, proxy support

https://gerrit.wikimedia.org/r/431737

Change 431738 merged by Giuseppe Lavagetto:
[operations/puppet@production] puppet_ecdsacert: allow IP-based SANs

https://gerrit.wikimedia.org/r/431738

Note that memcached-pecl (which uses Nutcracker) is still used in wmf-config in two places:

  1. On all wikis, for parser cache. [wmf-config/CommonSettings.php#mysql-multiwrite]
  2. On labswiki and labtestwiki, as the wiki's main cache type (instead of mcrouter). [wmf-config/mc.php]
Joe added a comment.Sep 7 2018, 6:43 AM

Note that memcached-pecl (which uses Nutcracker) is still used in wmf-config in two places:

  1. On all wikis, for parser cache. [wmf-config/CommonSettings.php#mysql-multiwrite]

Should we just move that as well to mcrouter? it can also be used without replication.

Joe closed this task as Resolved.Feb 6 2019, 12:21 PM