Our current external storage clusters just reached 90% disk usage and icinga sent warnings. We still have some months of time to spare; the current clusters has been filling up for at least 2.5 years.
We need to:
- Review the ES server spec, mostly only to maximize disk space.
- Order and provision 6 to 8 nodes for EQIAD (and same for CODFW; so 12 or 16 all up)
MediaWiki writes to multiple active ES clusters in order to avoid SPOF. A cluster is a minimum of 3 nodes (1 master, 2 slaves) but ideally for capacity 4 nodes (1 master, 3 slaves).