Objectives:
- Primary frdb swap after utf8mb4 conversion of tables (T270218)
- Switch primary payments db back to eqiad
- Kernel upgrades for all buster hosts
- Upgrade frdb1002 from stretch to buster
- Replace primary frqueue server frqueue1001 with frqueue1003. (T274681)
Other possible objectives/tasks to be evaluated by FR-Tech
- delete civicrm.civicrm_activity 'details' column data older than 1 year (T261353)
- delete log_civicrm.civicrm_activity 'details' column data (T261353)
- clean up after old contacts deletes (T247487)
- delete old log_ table data (T257232)
Tasks (times are UTC)
- prep: confirm that all campaigns are disabled
- 14:00 put payments/listener in maint mode via puppet $down_for_maintenance
- 14:00 stop process-control on fran1001
- disable silverpop export job on civi1001 (whoops)
- 14:15 swap payments-mariadb origin server from payments2001 to payments1001
- 14:20 update payments-wiki mariadb config from payments2001 to payments1001
- 14:30 put civicrm in maint mode via puppet $down_for_maintenance
- 14:45 shut down redis on frqueue1001
- 14:50 promote frqueue1003 to frqueue-redis origin server
- 15:00 update various service queue config from frqueue1001 to frqueue1003
- localsettings/payments-wiki/LocalSettings.php
- localsettings/payments-wiki-staging/LocalSettings.php
- puppet/hieradata/common.yaml (queue_redis_master_ip, queue_redis_master, unattended_upgrades)
- 16:00 swap fundraisingdb origin server from frdb1002 to frdb1004, enabling GTID
- 17:00 fundraisingdb mariadb and kernel updates
- fix frdev's outbound access to rackspace.com mariadb mirror
- frdev1001 kernel -> 4.19.0-14
- frdev1001 mariadb -> 10.4.18
- frdb1003 mariadb -> 10.4.18
- civi1001 reboot to activate kernel ->4.19.0.-14
- test payment transactions
- test manual queue consumer runs
- 18:00 pull payments/civi/listener out of maintenance mode (leave frdev/fran)
- 17:30 start utf8mb4 conversion of frdb1003 including analytics dbs (~30H)
- 17:30 start utf8mb4 conversion of frdev1001 (~28H)
- 17:30 upgrade frdb1002 to buster (~40m)
- 18:30 start utf8mb4 conversion of frdb1002 (~26H)
- add GTID into my.cnf template, set it for payments role and frdb roles - Nothing needed in my.cnf unless we want to set a domain or other global config item
- make necessary corrections to monitoring configuration
- switch database dump collection to archive from frdb1004 (missing actually) to frdb1002
- create a decom task for frqueue1001 (T277171)
- restart replication on frdb1002 after utf8mb4 conversion
- restart replication on frdb1003 after utf8mb4 conversion
- restart replication on frdev1001 after utf8mb4 conversion
- reenable process-control jobs for fran1001
- test/confirm that silverpop exporter works properly after utf8mb4 conversion
- reenable silverpop exporter job on civi1001
- enable GTID for payments-wiki mariadb - Shifting out of the maint window to T277710
- refresh/improve mariadb replication swap documentation (T236849)