Instead of using merge() to update the existing cache value, delete() it. Populate local Memcached from a slave on GET requests rather than populating on master requests. Tombstoning prevents slave lagged data from getting into long-lived cache.
See https://etherpad.wikimedia.org/p/Flow-performance-2015-12-01
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Update Flow caching strategy for multi DC | mediawiki/extensions/Flow | master | +234 -1 K |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T128228 Improve how transactions work for board moves | |||
| Resolved | aaron | T88445 MediaWiki active/active datacenter investigation and work (tracking) | |||
| Resolved | SBisson | T120009 Flow: Use WAN cache delete() and replica-based filling to avoid merge() | |||
| Resolved | matthiasmullie | T94029 Spike: Avoid use of merge() in Flow caches | |||
| Resolved | SBisson | T120008 Remove in-process Memcached wrapper or find an appropriate solution for cross-DC world | |||
| Open | None | T120007 Add debug/warn for cache reads from in-process Memcached cache wrapper |
Change 307951 had a related patch set uploaded (by Sbisson):
[WIP] Update Flow caching strategy for multi DC
Let's wait until the patch has survived deployment to production next week. We'll be on the lookout for errors and warnings in the logs.
@Catrope: there are ERRORs in https://logstash-beta.wmflabs.org/app/kibana#/dashboard/memcached...
2016-11-18T18:57:23 ERROR enwiki deployment-mediawiki04 Memcached error for key "WANCache:v:enwiki:filerepo:wikimediacommons:ForeignAPIRepo:Metadata:755faa509b6477c48...
normalized_message Memcached error for key "{memcached-key}" on server "{memcached-server}": SERVER ERROR