Page MenuHomePhabricator

Flow: Use WAN cache delete() and slave populations to avoid merge()
Closed, ResolvedPublic

Description

Instead of using merge() to update the existing cache value, delete() it. Populate local Memcached from a slave on GET requests rather than populating on master requests. Tombstoning prevents slave lagged data from getting into long-lived cache.

See https://etherpad.wikimedia.org/p/Flow-performance-2015-12-01

Event Timeline

Change 307951 had a related patch set uploaded (by Sbisson):
[WIP] Update Flow caching strategy for multi DC

https://gerrit.wikimedia.org/r/307951

Change 307951 merged by jenkins-bot:
Update Flow caching strategy for multi DC

https://gerrit.wikimedia.org/r/307951

Can this be closed now?

Let's wait until the patch has survived deployment to production next week. We'll be on the lookout for errors and warnings in the logs.

@Catrope: there are ERRORs in https://logstash-beta.wmflabs.org/app/kibana#/dashboard/memcached...

2016-11-18T18:57:23	ERROR	enwiki	deployment-mediawiki04	Memcached error for key "WANCache:v:enwiki:filerepo:wikimediacommons:ForeignAPIRepo:Metadata:755faa509b6477c48...
normalized_message	  	Memcached error for key "{memcached-key}" on server "{memcached-server}": SERVER ERROR

@Catrope: there are ERRORs in https://logstash-beta.wmflabs.org/app/kibana#/dashboard/memcached...

2016-11-18T18:57:23	ERROR	enwiki	deployment-mediawiki04	Memcached error for key "WANCache:v:enwiki:filerepo:wikimediacommons:ForeignAPIRepo:Metadata:755faa509b6477c48...
normalized_message	  	Memcached error for key "{memcached-key}" on server "{memcached-server}": SERVER ERROR

The good news is: this cache key has nothing to do with Flow...

Can this be closed now?

Let's wait until the patch has survived deployment to production next week. We'll be on the lookout for errors and warnings in the logs.

Being bold (tm) and closing this.