db1107:

The hosts acts as master in m4 cluster. The maintenance itself can be done as:

• Agree in a usable maintenance window
• Stop eventlog1002 to push data
• Disable eventlogging_cleaner on both hosts
• Make sure eventlogging_sync.sh was run on db1108 (to have uptodate data there in case of something goes wrong)
• Do the maintenance itself as https://phabricator.wikimedia.org/P7496
• Restart recieving data from eventlog1002
• Enabe eventlogging_cleaner on both hosts

db1108

• Agree in a usable maintenance window
• Disable eventogging_sync.sh on db1108
• Disable eventlogging_cleaner
• Do the maintenance itself as https://phabricator.wikimedia.org/P7496
• Enable eventlogging_sync.sh
• Enable eventlogging_cleaner

Notes about the performable operations can be found here: https://docs.google.com/document/d/1JVwK2TvaLJ7cs5-qnx_536LYM_3G-aAum66FtoFjhFE/

refined the maintenance operation with @elukey;
Tomorrow 10:00am CEST we do the maintenance of db1107.

The steps will be:

• @elukey stops the eventlogging service on eventlog1002
• @elukey disables the syncer and the cleaner jobs
• @elukey syncs the data manually between db1107 and db1108
• @Banyek does the maintenance as P7496
• @elukey enables the syncer and the cleaner jobs
• @elukey starts the eventlogging service on eventlog1002

a walk in a park

Is there any specific reason to start with the master first?

Yes, the replica is the one which is actively used

Got it! Thanks

Mentioned in SAL (#wikimedia-operations) [2018-09-26T08:17:04Z] <banyek> db1107 upgrade finished (T205288)

db1107 upgrade done, the operaton is resumed

Great! Make sure to update: https://phabricator.wikimedia.org/P7510 and reload haproxy!

upgrade of db1108 is planned on 2018-09-27 10:00 CEST

Update for 1108:
db1108

Agree in a usable maintenance window
Downtime dbproxy1004 and dbproxy1009 services in icinga
Disable eventogging_sync.sh on db1108
Disable eventlogging_cleaner
Do the maintenance itself as https://phabricator.wikimedia.org/P7496
Enable eventlogging_sync.sh
Enable eventlogging_cleaner
Reload HAProxy on  dbproxy1004 and dbproxy1009

Mentioned in SAL (#wikimedia-operations) [2018-09-27T07:58:10Z] <banyek> rebooting db1108 for kernel & mariadb upgrade (T205288)