Maniphest T192370

Deploy mcrouter to production as a wancache backend
Closed, ResolvedPublic
Actions

Description

Having cross-datacenter consistency between WANObjectCache objects is one of the prerequisites for being able to serve MediaWiki from multiple datacenters.

This means basically substituting the current twemproxy solution for aggregating multiple memcached instances in a single cluster with a software that allows to broadcast/replicate writes

To this end, we want to use Facebook's mcrouter T156938

We have debian packages built, and a basic module that supports a single-datacenter installation in deployment-prep. The production installation, though, will need some work on our end.

Specifically, we need to figure out a suitable architecture for cross-dc to work, add proper monitoring of the software, and execute the transition from twemproxy to mcrouter in a secure way.

Details

Related Gerrit Patches:

operations/puppet : production	profile::mediawiki::mcrouter_wancache: add ssl, proxy support
operations/puppet : production	puppet_ecdsacert: allow IP-based SANs
operations/puppet : production	mcrouter: add support for listening on the ssl port

Revisions and Commits

rOPUP Wikimedia Puppet
	rOPUP0ed6295ad3c3 profile::dumps::generation::worker::common: add mcrouter

Related Objects
Search...

Status	Assigned	Task
Resolved	Joe	T192370 Deploy mcrouter to production as a wancache backend
Resolved	Joe	T192763 Create a prometheus exporter for mcrouter
Resolved	Joe	T192771 mcrouter production architecture
Open	jbond	T194031 Setup a new PKI software as an alternative to the puppet CA for managing services certificates
Resolved	aaron	T198239 Rollout use of mcrouter for MediaWiki in production
Resolved	aaron	T198279 Exception "Key contains invalid characters" from MemcachedBagOStuff.php
Resolved	aaron	T198280 Beta Cluster: Unable to obtain lock via objectcache (memcached add() fails)
Resolved	Legoktm	T197450 test.wp is using test2.wp's message cache
Resolved	Krinkle	T199237 Unable to submit edit due to Exception "Key contains invalid characters" from MemcachedBagOStuff
Resolved	Joe	T203626 deploy1001 can't talk to memcached, breaking invalidation of RL localization cache
Open	None	T202431 Move labweb hosts from nutcracker to mcrouter?

Event Timeline

Joe created this task.Apr 17 2018, 3:15 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 17 2018, 3:15 PM

Joe moved this task from Backlog to Doing on the Availability (MediaWiki-MultiDC) board.Apr 17 2018, 3:16 PM

• Imarlier moved this task from Inbox to Radar on the Performance-Team board.Apr 23 2018, 8:22 PM

• Imarlier edited projects, added Performance-Team (Radar); removed Performance-Team.

Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.Apr 26 2018, 12:12 AM

fgiunchedi closed subtask T192763: Create a prometheus exporter for mcrouter as Resolved.Apr 30 2018, 8:15 AM

Change 431736 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mcrouter: add support for listening on the ssl port

https://gerrit.wikimedia.org/r/431736

Change 431737 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::mcrouter_wancache: add ssl, proxy support

https://gerrit.wikimedia.org/r/431737

Change 431738 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] puppet_ecdsacert: allow IP-based SANs

https://gerrit.wikimedia.org/r/431738

Change 431736 merged by Giuseppe Lavagetto:
[operations/puppet@production] mcrouter: add support for listening on the ssl port

https://gerrit.wikimedia.org/r/431736

Change 431737 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::mcrouter_wancache: add ssl, proxy support

https://gerrit.wikimedia.org/r/431737

Change 431738 merged by Giuseppe Lavagetto:
[operations/puppet@production] puppet_ecdsacert: allow IP-based SANs

https://gerrit.wikimedia.org/r/431738

Joe closed subtask T192771: mcrouter production architecture as Resolved.Jun 1 2018, 12:23 PM

jcrespo removed a subtask: T196378: Investigate solutions for MySQL connection pooling.Jun 4 2018, 3:24 PM

Krinkle added a subtask: T198239: Rollout use of mcrouter for MediaWiki in production.Jun 27 2018, 1:54 AM

Krinkle added subtasks: T198279: Exception "Key contains invalid characters" from MemcachedBagOStuff.php, T198280: Beta Cluster: Unable to obtain lock via objectcache (memcached add() fails).Jun 28 2018, 9:33 PM

Krinkle removed a subtask: T198279: Exception "Key contains invalid characters" from MemcachedBagOStuff.php.

Krinkle removed a subtask: T198280: Beta Cluster: Unable to obtain lock via objectcache (memcached add() fails).

Krinkle added a commit: rOPUP0ed6295ad3c3: profile::dumps::generation::worker::common: add mcrouter.Aug 1 2018, 5:47 PM

aaron closed subtask T198239: Rollout use of mcrouter for MediaWiki in production as Resolved.Aug 21 2018, 8:02 AM

Krinkle added a subtask: T203626: deploy1001 can't talk to memcached, breaking invalidation of RL localization cache.Sep 7 2018, 12:56 AM

Krinkle added a subtask: T202431: Move labweb hosts from nutcracker to mcrouter?.

Note that memcached-pecl (which uses Nutcracker) is still used in wmf-config in two places:

On all wikis, for parser cache. [wmf-config/CommonSettings.php#mysql-multiwrite]
On labswiki and labtestwiki, as the wiki's main cache type (instead of mcrouter). [wmf-config/mc.php]

In T192370#4564888, @Krinkle wrote:

Note that memcached-pecl (which uses Nutcracker) is still used in wmf-config in two places:

On all wikis, for parser cache. [wmf-config/CommonSettings.php#mysql-multiwrite]

Should we just move that as well to mcrouter? it can also be used without replication.

Phabricator_maintenance moved this task from Backlog to Acknowledged on the Operations board.Jan 26 2019, 9:56 PM

Joe closed this task as Resolved.Feb 6 2019, 12:21 PM

Deploy mcrouter to production as a wancache backendClosed, ResolvedPublicActions