The conversion to multi-instance is now complete in the eqiad datacenter, and is on track for completion in codfw RSN. Our current baseline is an instance count of 2 per host, with the exception of restbase200[1-2].codfw.wmnet, which are already running 3 instances each.
Back-of-napkin: If each instance in eqiad is currently ~1T in size, bumping instance count to 3 should reduce node density to ~682G (based on present storage levels). My expectation is that this will improve read latency by reducing the SSTables/read, put us in a more favorable position to begin incremental repairs, and give the aggressive memory configurations that have been proposed in T125906, a better chance of succeeding.
Based on the outcome of T130540, we can move forward in eqiad without the need to serialize with the on-going expansions in codfw.
See:
- {T130540}
- {T95253}
----
### Instances to bootstrap
- [x] 1007-c
- [x] 1008-c
- [ ] 1009-c
- [x] 1010-c
- [x] 1011-c
- [x] 1012-c
- [x] 1013-c
- [ ] 1014-c
- [ ] 1015-c
- [x] 2003-b
- [x] 2003-c
- [x] 2004-b
- [x] 2004-c
- [x] 2005-b
- [x] 2005-c
- [x] 2006-b
- [ ] 2006-c
- [x] 2007-c
- [x] 2008-c
- [ ] 2009-c
NOTE: 2016-05-25T16:06:58-05:00: While the bootstraps can run concurrently across data-centers, codfw has more instances to bootstrap, with less initial concurrency, and so it represents the upper bound on completion. Taking into account the evolving per-rack concurrencies and data set sizes, I calculate ~115 hours of total bootstrapping time (or ~4.79 days).