Page MenuHomePhabricator

Evaluate need for a "local server stash"
Open, Needs TriagePublic

Description

Status quo

See also: https://www.mediawiki.org/wiki/Manual:Caching#Interfaces

Our caching primitives are:

  • local server cache (e.g. typically APCu, which has a set amount of RAM to use). Falls back to no-cache.
  • local cluster cache (e.g. typically Memcached, which uses RAM over the network from one more dedicated hosts). Can fall back to a DB table.
  • main WAN cache (local cluster cache + broadcasted purges for cross-dc).

Our stashing primitives (stash is a store with higher persistence, and little to no LRU):

  • main stash (typically replicated redis). Falls back to db-replicated.
Idea

There seem to be a number of use cases we've seen emerge over the past year for things that are expensive to compute but also fairly large in size. So large that we were uncomfortable storing them in APCu by default for MW core. Doing so might cause typical third-party configurations of APCu to stop being effective due to the limited megabytes being filled up by less commonly needed stuff. The fallback to Memcached isnt' always practical either as that leads to high network saturation which we've seen at the heart of mulitple outages at WMF. Storing in Memcached might be fine to allow re-use between servers but we'd still want something local as well.

That local component for large and higher-persisted stuff coud very well still be APCu at WMF. But from the MW perspective we might want to abstract that.

@aaron has suggested we use an sqlite3 file as the default in core for such "local server stash".

Event Timeline

Krinkle created this task.Mar 18 2020, 5:07 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 18 2020, 5:07 PM

Change 524657 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] objectcache: add local sqlite3 file as a getLocalServerObjectCache() fallback

https://gerrit.wikimedia.org/r/524657

Change 570191 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: create a Database class that uses the Sqlite3 extension

https://gerrit.wikimedia.org/r/570191