T209620 shows that dbstore100[3-5] are ready to be used in production, so we should start planning the move from dbstore1002 to the new multi instance setup as soon as possible. Two important things needs to be kept in mind:
* the new set of hosts will probably be used a lot by the Analytics team so a shared knowledge of procedures/setup/etc.. would be really great, so we'll try to bother the Data Persistence team as few as possible :)
* Ubuntu Trusty 14.04.5 is going EOL In April 2019 (https://wiki.ubuntu.com/Releases) so we have a hard deadline to decommission dbstore1002.
I would use this task to make a high level plan (and possibly subtasks) about the steps needed to do this work.
A very coarse grained list of things to do:
[x] review data in dbstore1002 and come up with a multi-instance setup scheme for dbstore100[3-5] T212487
Distribution:
|host | sections | size |
|dbstore1003| s1, s5, s7 | 2.1T |
|dbstore1004| s2, s3, s4 | 2,5T |
|dbstore1005| s8, s6,x1 | 2,3T |
Where to place the existing `staging` database: dbstore1005:3350 (decided as of 11th Feb)
[x] review special settings that are not puppetized, and then come up with a basic setup for dbstore100[3-5] (this is a goal for Analytics in this quarter)
** There is nothing apart from grants not being puppetized - grants for the new hosts are now being tracked at: https://github.com/wikimedia/puppet/blob/production/modules/role/templates/mariadb/grants/analytics-dbstore.sql
[] review strategies to allow multi-user set up on dbstore100[3-5]. The Analytics team would really love to decommission the `research` user account and introduce a per-user account scheme, in order to be able to track down query patterns and in the future (hopefully) migrate all of them to Hadoop. This is absolutely a stretch goal and can be done afterwards if too complicated/long. T214469
[x] When the new instances are ready, start the data replication.
[] Fully migrate users to the new hosts {T215589}
[] Create a decommission task for dbstore1002
Instances to be set up:
[x] dbstore1003
** [x] s1
** [x] s5
** [x] s7
[x] dbstore1004
** [x] s2
** [x] s3
** [x] s4
[] dbstore1005
** [x] s6
** [x] s8
** [x] x1
** [] staging (port 3350)