2020-11-18 13:54:38 INFO Connecting to backend :3306: maxConnections: 128, maxIdleConns: 32
2020-11-18 13:54:38 DEBUG Cache hostname resolve zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud as zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud
2020-11-18 13:54:38 INFO verifyCert requested, client certificates will be verified
2020-11-18 13:54:38 INFO Read in CA file: /etc/ssl/certs/Puppet_Internal_CA.pem
2020-11-18 13:54:38 INFO GracefulMasterTakeover: Will demote zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306 and promote zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306 instead
2020-11-18 13:54:38 ERROR Error 1142: SELECT command denied to user 'orchestrator'@'172.16.1.250' for table 'slave_master_info'
2020-11-18 13:54:38 DEBUG Cache hostname resolve slave1.mariadb104-test.eqiad1.wikimedia.cloud as slave1.mariadb104-test.eqiad1.wikimedia.cloud
2020-11-18 13:54:38 INFO topology_recovery: No PreGracefulTakeoverProcesses hooks to run
2020-11-18 13:54:38 INFO GracefulMasterTakeover: Will set zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306 as read_only
2020-11-18 13:54:38 INFO instance zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306 read_only: true
2020-11-18 13:54:38 INFO GracefulMasterTakeover: Will wait for zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306 to reach master coordinates zarcillo1-bin.000013:959
2020-11-18 13:54:38 DEBUG Cache hostname resolve zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud as zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud
2020-11-18 13:54:38 INFO GracefulMasterTakeover: attempting recovery
2020-11-18 13:54:38 INFO executeCheckAndRecoverFunction: proceeding with DeadMaster detection on zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306; isActionable?: true; skipProcesses: false
2020-11-18 13:54:38 INFO topology_recovery: detected DeadMaster failure on zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306
2020-11-18 13:54:38 INFO topology_recovery: No OnFailureDetectionProcesses hooks to run
2020-11-18 13:54:38 INFO executeCheckAndRecoverFunction: proceeding with DeadMaster recovery on zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306; isRecoverable?: true; skipProcesses: false
2020-11-18 13:54:38 INFO topology_recovery: will handle DeadMaster event on zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306
2020-11-18 13:54:38 INFO topology_recovery: No PreFailoverProcesses hooks to run
2020-11-18 13:54:38 INFO topology_recovery: RecoverDeadMaster: will recover zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306
2020-11-18 13:54:38 INFO topology_recovery: RecoverDeadMaster: masterRecoveryType=MasterRecoveryGTID
2020-11-18 13:54:38 INFO topology_recovery: RecoverDeadMaster: regrouping replicas via GTID
2020-11-18 13:54:38 INFO topology_recovery: RecoverDeadMaster: 1 postponed functions
2020-11-18 13:54:38 INFO topology_recovery: checking if should replace promoted replica with a better candidate
2020-11-18 13:54:38 INFO topology_recovery: + sanity check: found our very own server to promote; doing nothing
2020-11-18 13:54:38 INFO topology_recovery: replace-promoted-replica-with-candidate: promoted instance zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306 requires no further action
2020-11-18 13:54:38 INFO topology_recovery: promoted replica: zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306
2020-11-18 13:54:38 INFO topology_recovery: RecoverDeadMaster: successfully promoted zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306
2020-11-18 13:54:38 INFO topology_recovery: - RecoverDeadMaster: promoted server coordinates: zarcillo0-bin.000022:577772
2020-11-18 13:54:38 INFO topology_recovery: - RecoverDeadMaster: will apply MySQL changes to promoted master
2020-11-18 13:54:38 INFO Will reset replica on zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306
2020-11-18 13:54:38 INFO Stopped replication on zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306, Self:zarcillo0-bin.000022:577772, Exec:zarcillo1-bin.000013:959
2020-11-18 13:54:39 INFO Reset replication zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306
2020-11-18 13:54:39 INFO topology_recovery: - RecoverDeadMaster: applying RESET SLAVE ALL on promoted master: success=true
2020-11-18 13:54:39 INFO instance zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306 read_only: false
2020-11-18 13:54:39 INFO topology_recovery: - RecoverDeadMaster: applying read-only=0 on promoted master: success=true
2020-11-18 13:54:39 DEBUG ChangeMasterTo: will attempt changing master on zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306 to zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306, zarcillo0-bin.000022:577772
2020-11-18 13:54:39 INFO ChangeMasterTo: Changed master on zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306 to: zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306, zarcillo0-bin.000022:577772. GTID: true
2020-11-18 13:54:39 DEBUG EnableMasterSSL: Will attempt enabling SSL replication on zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306
2020-11-18 13:54:39 INFO EnableMasterSSL: Enabled SSL replication on zarcillo1.mariadb104-test.eqiad1.wikimedia.cloud:3306
2020-11-18 13:54:39 INFO topology_recovery: No PostGracefulTakeoverProcesses hooks to run
2020-11-18 13:54:39 DEBUG Promoted zarcillo0.mariadb104-test.eqiad1.wikimedia.cloud:3306 as new master. Binlog coordinates at time of promotion: zarcillo0-bin.000022:577772