Logs will eat all the disk we can give them. That's just the nature of logging. With ElasticSearch as a backing store we can spread things out over multiple physical machines such that each machine only has a portion of the total log volume locally such that each host only needs to hold 2M/N of the total where M is the desired capacity and N is the number of nodes in the cluster.

ElasticSearch is not going to be happy with 30K (1K tools * 30 days) distinct indices. If we lump all of the tools into a single index per-day (basically what we do with ELK in production) then we can keep a lid on that, but we would lose the ability to easily drop logs for a tool that was out of control.

@bd808 can we instead have one index per tool and then just limit the size of that index? maybe a max of 1G per tool and then things start getting auto-dropped...

In T127368#2041872, @yuvipanda wrote:

@bd808 can we instead have one index per tool and then just limit the size of that index? maybe a max of 1G per tool and then things start getting auto-dropped...

Deleting old logs without doing index rotation is not the greatest thing for ElasticSearch performance. I wrote a lot about this on another task but I can't find it now. The TL;DR is that the Lucene data is append only and "deleting" means flagging a record as deleted and then at some point rebuilding the shard with the deleted records removed. ElasticSearch manages the mechanics of that for you but it requires disk space and iops to accomplish.

Having something like ElasticSearch backing the log storage gives a lot of neat benefits, but ultimately we just need to separate log shipping from log storage (so we don't use NFS for both) and create a way for tool maintainers to get at their log streams. Having a horizontally sharded pool of servers that act like fluorine does in production with a nice shipping solution may really be all we need.

I imagine most of these will be relatively low volume, as in <5GB per tool per month. In this case we might be best off using a single index per tool. In this case 1k indices with 1 primary and 1 replica each is probably something elasticsearch can support fairly reasonably.

1 index per day per tool would be a non-starter without having multiple independant elasticsearch clusters

related to bd808's comments, i'm not as worried about deleting 3% of an index per day. In the prod search cluster we run a deleted documents rate of ~30% without major issues. It might require some testing but my intuition is 3%/day is minimal.

In T127368#2041976, @EBernhardson wrote:

related to bd808's comments, i'm not as worried about deleting 3% of an index per day. In the prod search cluster we run a deleted documents rate of ~30% without major issues. It might require some testing but my intuition is 3%/day is minimal.

A lot of this depends on how the fragments in the shards are tuned too, so I may be over reacting (indices are composed of shards, shards are composed of fragments, fragments are the append-only bit). We can have fairly large shards and still manage to not eat up too much space during compaction if the fragment size is reasonable (say <2G).

I've got battle scars from $DAYJOB-1 and an initial design of our ElasticSearch cluster that melted at 40% disk utilization, so I'm probably erring on the side of extreme caution.