Whilst debugging a recent experiment deployment to testwiki, I noticed that the experiment configs bundle (sent to the client as part of the ext.testKitchen RL module) varied based on which DC the app server that served the request was in. I dropped into a PHP shell on the deployment server and observed the following:

1. The stashed experiment configs in both the eqiad and codfw DCs were up to date
2. The cached experiment configs in the eqiad DC were up to date
3. The cached experiment configs in the codfw DC were out of date

This is caused by the following:

• We use a DC-local cache to cache experiment configs for up to 1 week on cache miss
• We use the multi-DC stash to store experiment configs for up to 1 week
• The periodic job that updates the experiment configs only runs in the primary DC

e.g. consider the following situtation:

1. [DC 1] The periodic job runs, stashes new experiment configs and invalidates the DC-local cache
2. [DC 2] The stashed new experiment configs are replicated
3. [DC 1] Experiment configs are requested, the cache misses, the new experiment configs are fetched from the stash, and then cached for 1 week
4. [DC 2] Experiment configs are requested, the old experiment configs are fetched from the cache

(4) continues until the old experiment configs are evicted from the cache, which takes up to 1 week.

Proposed Solution

1. Use the WAN cache to cache experiment configs
2. Update the periodic job to not invalidate the DC-local cache
3. Reduce the cache TTL to 10 minutes. This:
   • Is significantly larger than the replication lag between the primary and secondary DCs
   • Is similar to the TTL of the ResourceLoader startup module, which controls how quickly new JS, CSS, and message assets are delivered to browsers
   • Keeps the stash read rate to a minimum (~6/hr)