This task is to track all the work related to the keystone merge in the main/eqiad1 openstack deployments.
An email has been sent to the cloud-announce mailing list, operations scheduled for next Mon 13th Aug at 14:00 UTC (the D day).
Before D day:
- announcements
- prepare network connectivity: ACL, firewalling, etc
- prepare new endpoints
- prepare DB grants and any other SQL commands for the D day
- prepare puppet patch to introduce the new keystone server/daemon @ cloudcontrol1003.wikimedia.org (https://gerrit.wikimedia.org/r/451314)
- re-check nova_controller hiera keys (https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/452345/)
- other DB operations (T201674)
- check/review the schedule/planning for concrete operations/steps on D day. Also, include some rollback instructions.
On D day:
- icinga downtimes
- put Horizon in downtime
- stop nodepool
- stop puppet on cloudcontrol1003,1004 and labcontrol1001 and 1002
- stop keystone on all hosts
- DB updates (endpoints)
- merge puppet patches (https://gerrit.wikimedia.org/r/451314 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/452345/)
- make/commit private puppet patches to update eqiad1 database username/password
- apply puppet patches
- start keystone on cloudcontrol1003, confirm it can r/w the database on m5
- tests, etc
The outcome of the operations is:
- keystone daemon is running on cloudcontrol1003.wikimedia.org
- keystone database is on m5
- keystone has support for 2 regions: eqiad (the old main deployment region) and eqiad1-r (the new eqiad1 deployment region)
- keystone database contains endpoints for both deployments
- no local keystone database (mysql) exists in cloudcontrol1003.wikimedia.org or cloudcontrol1004.wikimedia.org
I will be using this etherpad to paste things and track ongoing work: https://etherpad.wikimedia.org/p/cloudvps-keystone-merge
Related docs:
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/keystone_notes
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron_Notes/phases
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployments