To really have a plan, I would like to know the trajectory of our hot set. Unfortunately, I'm not really sure how to measure the hot set. The only data points i can think of at the moment are times when we ran out of IO, and estimating from there. Also these "total cache" numbers aren't exactly the cache size, rather the estimate of referenced mmap pages (vs loaded but unreferenced) at the point when our servers started to have issues.

july 2017 cache estimate:

• No direct history of memory.
• We started running the extra clusters after this, so estimate as current + 8G to account for second elastic process
• That gives 88G buff/cache
• page-types reported 30G of pages loaded but not referenced
• 58G/server of useful cache

sept 2020 cache estimate:

• top reports 50G used, 80G buff/cache
• page-types reports 11G of pages loaded but not referenced
• 70G/server of useful cache

oct 2020 cache estimate (post-mitigation)

• top reports 50G used, 80G buff/cache
• page-types reports 3G of pages loaded but not referenced
• 77G/server of useful cache

If these estimates are right, then simply getting to the point where we replace the current 128G instances with 256G instances will be sufficient. At that point our available cache baloons to 7,175 GB. I'm not sure if we can make it to that point though. The recent readahead mitigation should have increased available cache from 2,450G to 2,695G. A previous increase of 21% bought us 3 years of growth. With the same timeline we might get 12-18 months of runway from the current adjustments (but i still think we likely need hardware next FY, what exactly tbd).

Perhaps a silly thought, but one problem we have is that it's hard to directly measure how much cache memory we need. Since we have swap turned off on these machines we can simulate having less memory by having some other application simply allocate the memory (have python create a 10GB string). Can start at a low value and increase a couple GB at a time until IO starts climbing. We would then have a direct measure of how much memory was required (at that moment in time).

Or maybe there is a much simpler and more direct method?