Page MenuHomePhabricator

Deploy mcrouter to production as a wancache backend
Closed, ResolvedPublic

Description

Having cross-datacenter consistency between WANObjectCache objects is one of the prerequisites for being able to serve MediaWiki from multiple datacenters.

This means basically substituting the current twemproxy solution for aggregating multiple memcached instances in a single cluster with a software that allows to broadcast/replicate writes

To this end, we want to use Facebook's mcrouter T156938

We have debian packages built, and a basic module that supports a single-datacenter installation in deployment-prep. The production installation, though, will need some work on our end.

Specifically, we need to figure out a suitable architecture for cross-dc to work, add proper monitoring of the software, and execute the transition from twemproxy to mcrouter in a secure way.

Event Timeline

Change 431736 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mcrouter: add support for listening on the ssl port

https://gerrit.wikimedia.org/r/431736

Change 431737 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::mcrouter_wancache: add ssl, proxy support

https://gerrit.wikimedia.org/r/431737

Change 431738 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] puppet_ecdsacert: allow IP-based SANs

https://gerrit.wikimedia.org/r/431738

Change 431736 merged by Giuseppe Lavagetto:
[operations/puppet@production] mcrouter: add support for listening on the ssl port

https://gerrit.wikimedia.org/r/431736

Change 431737 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::mcrouter_wancache: add ssl, proxy support

https://gerrit.wikimedia.org/r/431737

Change 431738 merged by Giuseppe Lavagetto:
[operations/puppet@production] puppet_ecdsacert: allow IP-based SANs

https://gerrit.wikimedia.org/r/431738

Note that memcached-pecl (which uses Nutcracker) is still used in wmf-config in two places:

  1. On all wikis, for parser cache. [wmf-config/CommonSettings.php#mysql-multiwrite]
  2. On labswiki and labtestwiki, as the wiki's main cache type (instead of mcrouter). [wmf-config/mc.php]

Note that memcached-pecl (which uses Nutcracker) is still used in wmf-config in two places:

  1. On all wikis, for parser cache. [wmf-config/CommonSettings.php#mysql-multiwrite]

Should we just move that as well to mcrouter? it can also be used without replication.