Page MenuHomePhabricator

Flow: Use WAN cache delete() and slave populations to avoid merge()
Closed, ResolvedPublic

Description

Instead of using merge() to update the existing cache value, delete() it. Populate local Memcached from a slave on GET requests rather than populating on master requests. Tombstoning prevents slave lagged data from getting into long-lived cache.

See https://etherpad.wikimedia.org/p/Flow-performance-2015-12-01

Event Timeline

Krinkle removed a subscriber: Krinkle.Apr 7 2016, 11:39 PM
Mattflaschen-WMF renamed this task from Use WAN cache delete() and slave populations to avoid merge() to Flow: Use WAN cache delete() and slave populations to avoid merge().Apr 12 2016, 9:06 PM

Change 307951 had a related patch set uploaded (by Sbisson):
[WIP] Update Flow caching strategy for multi DC

https://gerrit.wikimedia.org/r/307951

Change 307951 merged by jenkins-bot:
Update Flow caching strategy for multi DC

https://gerrit.wikimedia.org/r/307951

aaron added a comment.Nov 10 2016, 8:54 PM

Can this be closed now?

Can this be closed now?

Let's wait until the patch has survived deployment to production next week. We'll be on the lookout for errors and warnings in the logs.

Etonkovidova added a comment.EditedNov 18 2016, 7:10 PM

@Catrope: there are ERRORs in https://logstash-beta.wmflabs.org/app/kibana#/dashboard/memcached...

2016-11-18T18:57:23	ERROR	enwiki	deployment-mediawiki04	Memcached error for key "WANCache:v:enwiki:filerepo:wikimediacommons:ForeignAPIRepo:Metadata:755faa509b6477c48...
normalized_message	  	Memcached error for key "{memcached-key}" on server "{memcached-server}": SERVER ERROR

@Catrope: there are ERRORs in https://logstash-beta.wmflabs.org/app/kibana#/dashboard/memcached...

2016-11-18T18:57:23	ERROR	enwiki	deployment-mediawiki04	Memcached error for key "WANCache:v:enwiki:filerepo:wikimediacommons:ForeignAPIRepo:Metadata:755faa509b6477c48...
normalized_message	  	Memcached error for key "{memcached-key}" on server "{memcached-server}": SERVER ERROR

The good news is: this cache key has nothing to do with Flow...

aaron closed this task as Resolved.Dec 14 2016, 5:29 PM

Can this be closed now?

Let's wait until the patch has survived deployment to production next week. We'll be on the lookout for errors and warnings in the logs.

Being bold (tm) and closing this.