Page MenuHomePhabricator
Create Task
Maniphest T192370

Deploy mcrouter to production as a wancache backend
Closed, ResolvedPublic
Actions

Description

Having cross-datacenter consistency between WANObjectCache objects is one of the prerequisites for being able to serve MediaWiki from multiple datacenters.

This means basically substituting the current twemproxy solution for aggregating multiple memcached instances in a single cluster with a software that allows to broadcast/replicate writes

To this end, we want to use Facebook's mcrouter T156938

We have debian packages built, and a basic module that supports a single-datacenter installation in deployment-prep. The production installation, though, will need some work on our end.

Specifically, we need to figure out a suitable architecture for cross-dc to work, add proper monitoring of the software, and execute the transition from twemproxy to mcrouter in a secure way.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+113 -15profile::mediawiki::mcrouter_wancache: add ssl, proxy support
operations/puppetproduction+21 -6puppet_ecdsacert: allow IP-based SANs
operations/puppetproduction+28 -23mcrouter: add support for listening on the ssl port
Customize query in gerrit

Revisions and Commits

rOPUP Wikimedia Puppet
rOPUP0ed6295ad3c3 profile::dumps::generation::worker::common: add mcrouter

Related Objects
Search...

Event Timeline

Joe created this task.Apr 17 2018, 3:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 17 2018, 3:15 PM
Joe moved this task from Later to Current: Performance Team on the Sustainability (MediaWiki-MultiDC) board.Apr 17 2018, 3:16 PM
• Imarlier moved this task from Inbox to Radar on the Performance-Team board.Apr 23 2018, 8:22 PM
• Imarlier edited projects, added Performance-Team (Radar); removed Performance-Team.
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.Apr 26 2018, 12:12 AM
fgiunchedi closed subtask T192763: Create a prometheus exporter for mcrouter as Resolved.Apr 30 2018, 8:15 AM
gerritbot added a subscriber: gerritbot.May 8 2018, 11:59 AM

Change 431736 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mcrouter: add support for listening on the ssl port

https://gerrit.wikimedia.org/r/431736

gerritbot added a project: Patch-For-Review.May 8 2018, 11:59 AM

Change 431737 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::mcrouter_wancache: add ssl, proxy support

https://gerrit.wikimedia.org/r/431737

gerritbot added a comment.May 8 2018, 11:59 AM

Change 431738 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] puppet_ecdsacert: allow IP-based SANs

https://gerrit.wikimedia.org/r/431738

gerritbot added a comment.May 22 2018, 10:52 AM

Change 431736 merged by Giuseppe Lavagetto:
[operations/puppet@production] mcrouter: add support for listening on the ssl port

https://gerrit.wikimedia.org/r/431736

gerritbot added a comment.May 24 2018, 9:08 AM

Change 431737 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::mcrouter_wancache: add ssl, proxy support

https://gerrit.wikimedia.org/r/431737

gerritbot added a comment.May 24 2018, 9:31 AM

Change 431738 merged by Giuseppe Lavagetto:
[operations/puppet@production] puppet_ecdsacert: allow IP-based SANs

https://gerrit.wikimedia.org/r/431738

Joe closed subtask T192771: mcrouter production architecture as Resolved.Jun 1 2018, 12:23 PM
jcrespo removed a subtask: T196378: Investigate solutions for MySQL connection pooling.Jun 4 2018, 3:24 PM
Krinkle added a subtask: T198239: Rollout use of mcrouter for MediaWiki in production.Jun 27 2018, 1:54 AM
Krinkle added subtasks: T198279: Exception "Key contains invalid characters" from MemcachedBagOStuff.php, T198280: Beta Cluster: Unable to obtain lock via objectcache (memcached add() fails).Jun 28 2018, 9:33 PM
Krinkle removed a subtask: T198279: Exception "Key contains invalid characters" from MemcachedBagOStuff.php.
Krinkle removed a subtask: T198280: Beta Cluster: Unable to obtain lock via objectcache (memcached add() fails).
Krinkle added a commit: rOPUP0ed6295ad3c3: profile::dumps::generation::worker::common: add mcrouter.Aug 1 2018, 5:47 PM
aaron closed subtask T198239: Rollout use of mcrouter for MediaWiki in production as Resolved.Aug 21 2018, 8:02 AM
Krinkle added a subtask: T203626: deploy1001 can't talk to memcached, breaking invalidation of RL localization cache.Sep 7 2018, 12:56 AM
Krinkle added a subtask: T202431: Move labweb hosts from nutcracker to mcrouter?.
Krinkle added a subscriber: Krinkle.Sep 7 2018, 12:59 AM

Note that memcached-pecl (which uses Nutcracker) is still used in wmf-config in two places:

  1. On all wikis, for parser cache. [wmf-config/CommonSettings.php#mysql-multiwrite]
  2. On labswiki and labtestwiki, as the wiki's main cache type (instead of mcrouter). [wmf-config/mc.php]
Joe added a comment.Sep 7 2018, 6:43 AM
In T192370#4564888, @Krinkle wrote:

Note that memcached-pecl (which uses Nutcracker) is still used in wmf-config in two places:

  1. On all wikis, for parser cache. [wmf-config/CommonSettings.php#mysql-multiwrite]

Should we just move that as well to mcrouter? it can also be used without replication.

Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:56 PM
Joe closed this task as Resolved.Feb 6 2019, 12:21 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL