This task is to do some estimation for specs of elasticsearch servers needed for logging on toolforge.
If we estimate:
That requires 1.8T of total space on an ongoing basis in the elasticsearch cluster.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Bstorm | T126083 overhaul labstore setup [tracking] | |||
| Resolved | • GTirloni | T216988 labstore1004 - DISK CRITICAL - free space: /srv/tools 115904 MB (1% inode=79%): | |||
| Resolved | Bstorm | T217993 2019-03-10: tools and NFS share cleanup (high usage) | |||
| Resolved | Bstorm | T122508 Prevent overly-large log files | |||
| Declined | None | T286847 Add webservice flag to mount project directory read-only | |||
| Open | None | T127367 [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge webservices and bots | |||
| Declined | None | T127368 Estimate hardware requirements for Toolforge logging elastic cluster |
Logs will eat all the disk we can give them. That's just the nature of logging. With ElasticSearch as a backing store we can spread things out over multiple physical machines such that each machine only has a portion of the total log volume locally such that each host only needs to hold 2M/N of the total where M is the desired capacity and N is the number of nodes in the cluster.
ElasticSearch is not going to be happy with 30K (1K tools * 30 days) distinct indices. If we lump all of the tools into a single index per-day (basically what we do with ELK in production) then we can keep a lid on that, but we would lose the ability to easily drop logs for a tool that was out of control.
@bd808 can we instead have one index per tool and then just limit the size of that index? maybe a max of 1G per tool and then things start getting auto-dropped...
Deleting old logs without doing index rotation is not the greatest thing for ElasticSearch performance. I wrote a lot about this on another task but I can't find it now. The TL;DR is that the Lucene data is append only and "deleting" means flagging a record as deleted and then at some point rebuilding the shard with the deleted records removed. ElasticSearch manages the mechanics of that for you but it requires disk space and iops to accomplish.
Having something like ElasticSearch backing the log storage gives a lot of neat benefits, but ultimately we just need to separate log shipping from log storage (so we don't use NFS for both) and create a way for tool maintainers to get at their log streams. Having a horizontally sharded pool of servers that act like fluorine does in production with a nice shipping solution may really be all we need.
I imagine most of these will be relatively low volume, as in <5GB per tool per month. In this case we might be best off using a single index per tool. In this case 1k indices with 1 primary and 1 replica each is probably something elasticsearch can support fairly reasonably.
1 index per day per tool would be a non-starter without having multiple independant elasticsearch clusters
related to bd808's comments, i'm not as worried about deleting 3% of an index per day. In the prod search cluster we run a deleted documents rate of ~30% without major issues. It might require some testing but my intuition is 3%/day is minimal.
A lot of this depends on how the fragments in the shards are tuned too, so I may be over reacting (indices are composed of shards, shards are composed of fragments, fragments are the append-only bit). We can have fairly large shards and still manage to not eat up too much space during compaction if the fragment size is reasonable (say <2G).
I've got battle scars from $DAYJOB-1 and an initial design of our ElasticSearch cluster that melted at 40% disk utilization, so I'm probably erring on the side of extreme caution.
This solutioning task is old enough that the contents are not accurate anymore. I'll leave it alive to be edited later perhaps for history, but for now, it's stalled and going in the graveyard I think.
We should have closed this years ago when we figured out that the FOSS ELK stack was not suitably multi-tenant (open core paywall blocking the needed authz features).