We have a documented rule that shards on the cirrus cluster should be 30GB max. When shards start to grow over this limit, relocation of shards becomes complicated and we should increase the number of shards for this index. We have been surprised a few times by shards growing up to 70GB.
An icinga check, running at low frequency (once per day is enough) would help identify those shards in a timely fashion. This check should have warning and critical threshold. It should report the indices that have shards over limit.