db1001 needs maintenance, or even decommissioning like db100[2-7]. To do that we need to perform an m1-master rotation to db1016.
In the past this was a painful prospect due to the sheer number of services connecting directly to db1001, but m1-master CNAME and dbproxy1001 were added and services migrated to use them, so it should now be easier.
However, big stuff is affected:
Needs planning or input from other Opsen.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dzahn | T123525 reduce amount of remaining Ubuntu 12.04 (precise) systems in production | |||
| Resolved | • jcrespo | T125028 reimage or decom db servers on precise | |||
| Resolved | • jcrespo | T135973 Upgrade m1 db servers | |||
| Resolved | • jcrespo | T106312 m1-master switch from db1001 to db1016 |
We already recently tried that for etherpad with @jcrespo. It failed due to db1016 not having the same rights as db1001. On the next effort everything went smoothly. I suppose that is one area we need to check before everything else. Also how about we pool db1016 in db1001 proxy ?
Just a side note: the issues we experience probably were caused by actually overpassing the proxy, and doing the failover "manually", so that it only affected etherpad, and not the rest of the services.
Thanks for commenting, but I believe that there are services here that you do not "own", and we still need feedback from other ops to prepare this, is that right?
After the clean up, these are the databases that are still there (aside from the mysql ones mysql, information_schema and performance schema):
bacula etherpadlite heartbeat librenms puppet racktables reviewdb rt
The rest have been temporarily archived on es2001 /srv/backup/m1 and will be either permanently deleted or archived properly on bacula.
@akosiaris @MoritzMuehlenhoff @fgiunchedi @demon @Krenair @Dzahn I intend to perform the failover on Wednesday 22, 16:00 UTC.
I do not really need you to do anything, and this should be a trivial process that needs no attention, but it would be great if you could check every service listed up here continues working after that with no issues.
I can check on RT ('rt') and racktables (the other rt) on Wednesday, maybe around 18UTC but i dont worry about it since these services are just used by ops themselves.
That said, please note that "reviewdb" is Gerrit, affects quite a few users and the releng team owns that one. Etherpad/Bacula/Librenms i believe are covered by Alex (?). I dont know about heartbeat and puppet.
I 'd be happy to check the various services I 've signed up for. But can we delay this for an hour or so ? That is 17:00 UTC, just to make sure I 'll be around.
17.00 UTC works for me, if that's finalized please grab the slot at https://wikitech.wikimedia.org/wiki/Deployments
These are the notes from the migration, to be documented on wiki:
Change 295562 had a related patch set uploaded (by Jcrespo):
Set temporarilly m1 haproxy to failover to itself (db1016)
Change 295562 merged by Jcrespo:
Set temporarily m1 haproxy to failover to itself (db1016)
Change 295563 had a related patch set uploaded (by Jcrespo):
Promote db1016 as the m1 shard master, set db1001 as a m1 slave
Change 295563 merged by Jcrespo:
Promote db1016 as the m1 shard master, set db1001 as a m1 slave