The conversion to multi-instance is now complete in the eqiad datacenter, and is on track for completion in codfw RSN. Our current baseline is an instance count of 2 per host, with the exception of restbase200[1-2].codfw.wmnet, which are already running 3 instances each.
Back-of-napkin: If each instance in eqiad is currently ~1T in size, bumping instance count to 3 should reduce node density to ~682G (based on present storage levels). My expectation is that this will improve read latency by reducing the SSTables/read, put us in a more favorable position to begin incremental repairs, and give the aggressive memory configurations that have been proposed in T125906, a better chance of succeeding.
Based on the outcome of T130540, we can move forward in eqiad without the need to serialize with the on-going expansions in codfw.
See:
- {T130540}
- {T95253}
----
### Instances to bootstrap
- [x] 1007-c
- [x] 1008-c
- [x] 1009-c
- [x] 1010-c
- [x] 1011-c
- [x] 1012-c
- [x] 1013-c
- [x] 1014-c
- [x] 1015-c
- [x] 2003-b
- [x] 2003-c
- [x] 2004-b
- [x] 2004-c
- [x] 2005-b
- [x] 2005-c
- [x] 2006-b
- [x] 2006-c
- [x] 2007-c
- [x] 2008-c
- [x] 2009-c
NOTE: 2016-05-25T16:06:58-05:00: While the bootstraps can run concurrently across data-centers, codfw has more instances to bootstrap, with less initial concurrency, and so it represents the upper bound on completion. Taking into account the evolving per-rack concurrencies and data set sizes, I calculate ~115 hours of total bootstrapping time (or ~4.79 days).
----
Some cleanup activity has occurred as the expansion has progressed, but one final sweep will be needed on each rack, once all range movements have completed.
### Instances to cleanup
- Eqiad
- Rack A
- [x] 1007-a
- [x] 1007-b
- [x] 1007-c
- [x] 1010-a
- [x] 1010-b
- [x] 1010-c
- [x] 1011-a
- [x] 1011-b
- [x] 1011-c
- Rack B
- [x] 1008-a
- [x] 1008-b
- [ ] 1008-c (in-progress)
- [x] 1012-a
- [x] 1012-b
- [ ] 1012-c (in-progress)
- [x] 1013-a
- [x] 1013-b
- [ ] 1013-c (in-progress)
- Rack D
- [x] 1009-a
- [ ] 1009-b (in-progress)
- [ ] 1009-c
- [x] 1014-a
- [ ] 1014-b (in-progress)
- [ ] 1014-c
- [x] 1015-a
- [x] 1015-b
- [ ] 1015-c (in-progress)
- Codfw
- Rack B
- [x] 2001-a
- [x] 2001-b
- [x] 2001-c
- [x] 2002-a
- [x] 2002-b
- [x] 2002-c
- [x] 2007-a
- [x] 2007-b
- [x] 2007-c
- Rack C
- [x] 2003-a
- [x] 2003-b
- [x] 2003-c
- [x] 2004-a
- [x] 2004-b
- [x] 2004-c
- [x] 2008-a
- [x] 2008-b
- [x] 2008-c
- Rack D
- [x] 2005-a
- [x] 2005-b
- [x] 2005-c
- [x] 2006-a
- [x] 2006-b
- [x] 2006-c
- [x] 2009-a
- [x] 2009-b
- [x] 2009-c