Page MenuHomePhabricator
Create Task
Maniphest T266483

Enable report_host for mariadb
Closed, ResolvedPublic
Actions

Description

By default mariadb does not report a hostname when it starts replicating from another server. This means the output of show slave status only contains very opaque information:

root@pc2007.codfw.wmnet[(none)]> show slave hosts;
+-----------+------+------+-----------+
| Server_id | Host | Port | Master_id |
+-----------+------+------+-----------+
| 171966644 |      | 3306 | 180355176 |
| 180367374 |      | 3306 | 180355176 |
+-----------+------+------+-----------+

If we use puppet to set report_host to the fqdn, then it will show up in the show slave hosts table, as well as be available for orchestrator to query.

NOTE: This requires MySQL daemon to be restarted
NOTE: No need to enable this on labsdb* hosts, those are not stable and they are being ignored on the discovery: https://gerrit.wikimedia.org/r/c/operations/puppet/+/660839/

https://mariadb.com/kb/en/replication-and-binary-log-system-variables/#report_host

Scripts used for rolling this out:

Restarting hosts progress

Details

SubjectRepoBranchLines +/-
orchestrator.conf: Do not discover labsdb* hostsoperations/puppetproduction+3 -0
db-eqiad.php: Depool es5 from writesoperations/mediawiki-configmaster+1 -1
db-eqiad.php: Depool pc1009operations/mediawiki-configmaster+2 -2
db-eqiad.php: Depool pc1008operations/mediawiki-configmaster+2 -2
db-eqiad.php: Pool pc1010 instead of pc1007.operations/mediawiki-configmaster+2 -2
mariadb: (Ab)use wsrep_cluster_name for DC nameoperations/puppetproduction+143 -122
mariadb: Enable report_hostoperations/puppetproduction+86 -80
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 ResolvedLegoktmT281515 June 2021 Datacenter switchover
 ResolvedMarosteguiT285139 DB maintenance work to do while eqiad is passive (June 2021)
 ResolvedMarosteguiT266483 Enable report_host for mariadb
 Resolved• KormatT271106 Enable report_host on candidate masters
 ResolvedMarosteguiT271427 Switchover s4 (commonswiki) from db1081 to db1138
 ResolvedTrizek-WMFT271791 Database primary master failover on s4 (commonswiki)
 Resolved• KormatT272954 Fix db-switchover update zarcillo part
 ResolvedMarosteguiT272568 Add m* and es4/es5 sections to Orchestrator
 ResolvedMarosteguiT271540 Upgrade and restart m1 master (db1080)
 ResolvedMarosteguiT272596 Restart m3 (phabricator) database master db1132
 ResolvedMarosteguiT272388 Restart m5 master (db1128)
 ResolvedMarosteguiT272964 Restart m2 database master (db1107)
 ResolvedMarosteguiT273758 Restart x1 database master (db1103)
 ResolvedsgrabarczukT273762 Read only time for extension 1 (x1) primary database on 2021-02-17
 ResolvedMarosteguiT274336 Switchover s7 from db1086 to db1136
 ResolvedsgrabarczukT276899 Read only time window needed for s7
 Resolved• CmjohnsonT277007 Upgrade firmware on db1136
 ResolvedMarosteguiT278214 Switchover s1 from db1083 to db1163
 ResolvedMarosteguiT276150 Schema change to make rc_id unsigned and rc_timestamp BINARY
 ResolvedsgrabarczukT279505 Read only time window needed for s1 (enwiki)
 ResolvedRequest• CmjohnsonT281445 decommission db1083.eqiad.wmnet
 ResolvedMarosteguiT280492 Upgrade all sanitarium masters to 10.4 and Buster
 ResolvedRequestwiki_willyT281794 decommission db1082.eqiad.wmnet
 ResolvedRequest• CmjohnsonT281959 decommission db1074.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282079 decommission db1079.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282093 decommission db1087.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282096 decommission db1085.eqiad.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Marostegui updated the task description. (Show Details)Jan 21 2021, 9:11 AM
Marostegui mentioned this in T272568: Add m* and es4/es5 sections to Orchestrator.Jan 21 2021, 9:32 AM
Marostegui added a subtask: T272568: Add m* and es4/es5 sections to Orchestrator.Jan 21 2021, 1:26 PM
• Kormat closed subtask T271106: Enable report_host on candidate masters as Resolved.Jan 22 2021, 1:27 PM
Marostegui added a comment.EditedJan 25 2021, 7:31 AM

s4 progress:

labsdb1012.eqiad.wmnet:3306
labsdb1011.eqiad.wmnet:3306
labsdb1010.eqiad.wmnet:3306
labsdb1009.eqiad.wmnet:3306

  • dbstore1004.eqiad.wmnet:3314
  • db1155.eqiad.wmnet:3314
  • db1150.eqiad.wmnet:3314
  • db1149.eqiad.wmnet:3306
  • db1148.eqiad.wmnet:3306
  • db1147.eqiad.wmnet:3306
  • db1146.eqiad.wmnet:3314
  • db1145.eqiad.wmnet:3314
  • db1144.eqiad.wmnet:3314
  • db1143.eqiad.wmnet:3306
  • db1142.eqiad.wmnet:3306
  • db1141.eqiad.wmnet:3306
  • db1138.eqiad.wmnet:3306
  • db1125.eqiad.wmnet:3314
  • db1121.eqiad.wmnet:3306
  • db1081.eqiad.wmnet:3306
  • clouddb1019.eqiad.wmnet:3314
  • clouddb1015.eqiad.wmnet:3314
Marostegui updated the task description. (Show Details)Jan 25 2021, 7:31 AM
Marostegui closed subtask T271427: Switchover s4 (commonswiki) from db1081 to db1138 as Resolved.Jan 26 2021, 7:44 AM
Marostegui closed subtask T272596: Restart m3 (phabricator) database master db1132 as Resolved.Jan 27 2021, 6:06 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Jan 28 2021, 7:38 AM

Change 659216 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool es5 from writes

https://gerrit.wikimedia.org/r/659216

gerritbot added a project: Patch-For-Review.Jan 28 2021, 7:38 AM
Marostegui closed subtask T272388: Restart m5 master (db1128) as Resolved.Jan 28 2021, 9:04 AM
gerritbot added a comment.Jan 28 2021, 10:44 AM

Change 659216 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool es5 from writes

https://gerrit.wikimedia.org/r/659216

Stashbot added a comment.Jan 28 2021, 10:46 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-28T10:46:16Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool es5 from writes T266483 (duration: 01m 09s)

Stashbot added a comment.Jan 28 2021, 10:46 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-28T10:46:53Z] <marostegui> Restart mysql on es1024 T266483

Stashbot added a comment.Jan 28 2021, 10:52 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-28T10:52:14Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool es5 on writes T266483 (duration: 01m 05s)

Stashbot added a comment.Jan 28 2021, 11:03 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-28T11:03:54Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1025 T266483', diff saved to https://phabricator.wikimedia.org/P14018 and previous config saved to /var/cache/conftool/dbconfig/20210128-110353-marostegui.json

Stashbot added a comment.Jan 28 2021, 11:04 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-28T11:04:25Z] <marostegui> Restart mysql on es1025 T266483

Marostegui updated the task description. (Show Details)Jan 28 2021, 11:09 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 28 2021, 11:10 AM
Marostegui updated the task description. (Show Details)Jan 28 2021, 1:56 PM
Stashbot added a comment.Feb 1 2021, 9:45 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-01T09:45:49Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool es4 from writes T266483 (duration: 01m 04s)

Stashbot added a comment.Feb 1 2021, 9:46 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-01T09:46:04Z] <marostegui> Restart mysql on es1021 T266483

Stashbot added a comment.Feb 1 2021, 9:52 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-01T09:52:52Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool es4 into writes T266483 (duration: 00m 56s)

gerritbot added a comment.Feb 1 2021, 2:29 PM

Change 660839 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] orchestrator.conf: Do not discover labsdb* hosts

https://gerrit.wikimedia.org/r/660839

gerritbot added a project: Patch-For-Review.Feb 1 2021, 2:29 PM
gerritbot added a comment.Feb 1 2021, 2:35 PM

Change 660839 merged by Marostegui:
[operations/puppet@production] orchestrator.conf: Do not discover labsdb* hosts

https://gerrit.wikimedia.org/r/660839

Marostegui updated the task description. (Show Details)Feb 1 2021, 2:38 PM
Stashbot added a comment.Feb 1 2021, 2:39 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-01T14:39:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1147 T266483', diff saved to https://phabricator.wikimedia.org/P14104 and previous config saved to /var/cache/conftool/dbconfig/20210201-143925-marostegui.json

Stashbot added a comment.Feb 1 2021, 2:40 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-01T14:40:03Z] <marostegui> Restart mysql on db1147 T266483

Marostegui updated the task description. (Show Details)Feb 1 2021, 2:43 PM
Stashbot added a comment.Feb 2 2021, 6:23 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-02T06:23:04Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1022 T266483', diff saved to https://phabricator.wikimedia.org/P14113 and previous config saved to /var/cache/conftool/dbconfig/20210202-062303-marostegui.json

Marostegui updated the task description. (Show Details)Feb 2 2021, 6:31 AM
Marostegui closed subtask T272964: Restart m2 database master (db1107) as Resolved.Feb 3 2021, 10:03 AM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T272568: Add m* and es4/es5 sections to Orchestrator as Resolved.Feb 3 2021, 10:06 AM
Marostegui added a comment.EditedFeb 3 2021, 1:28 PM

x1:

  • dbstore1005
  • db1137
  • db1120
  • db1103
  • db1102
Marostegui updated the task description. (Show Details)Feb 3 2021, 1:28 PM
Stashbot added a comment.Feb 3 2021, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-03T13:29:38Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1120 T266483', diff saved to https://phabricator.wikimedia.org/P14164 and previous config saved to /var/cache/conftool/dbconfig/20210203-132938-marostegui.json

Stashbot added a comment.Feb 3 2021, 1:30 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-03T13:30:26Z] <marostegui> Stop mysql on db1120 to enable report_host T266483

Marostegui mentioned this in T273758: Restart x1 database master (db1103).Feb 3 2021, 1:45 PM
Stashbot added a comment.Feb 4 2021, 6:42 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-04T06:41:58Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1137 T266483', diff saved to https://phabricator.wikimedia.org/P14181 and previous config saved to /var/cache/conftool/dbconfig/20210204-064157-marostegui.json

Marostegui added a comment.EditedFeb 10 2021, 8:01 AM

s7 eqiad

  • labsdb1012 - not needed
  • labsdb1011 - not needed
  • labsdb1010 - not needed
  • labsdb1009 - not needed
  • dbstore1003
  • db1174
  • db1170
  • db1155
  • db1136
  • db1127
  • db1125
  • db1116
  • db1101
  • db1098
  • db1090
  • db1086
  • db1079
  • clouddb1018
  • clouddb1014
Stashbot added a comment.Feb 10 2021, 8:05 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-10T08:05:12Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1127 T266483', diff saved to https://phabricator.wikimedia.org/P14283 and previous config saved to /var/cache/conftool/dbconfig/20210210-080512-marostegui.json

Marostegui updated the task description. (Show Details)Feb 10 2021, 8:12 AM
Marostegui added a subtask: T274336: Switchover s7 from db1086 to db1136.Feb 10 2021, 8:44 AM
Marostegui closed subtask T273758: Restart x1 database master (db1103) as Resolved.Feb 17 2021, 7:13 AM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T274336: Switchover s7 from db1086 to db1136 as Resolved.Mar 23 2021, 6:16 AM
Stashbot added a comment.Mar 23 2021, 6:58 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-23T06:58:36Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3317 to enable report_host T266483', diff saved to https://phabricator.wikimedia.org/P15002 and previous config saved to /var/cache/conftool/dbconfig/20210323-065836-marostegui.json

Stashbot added a comment.Mar 23 2021, 6:59 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-23T06:59:48Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3318 to enable report_host T266483', diff saved to https://phabricator.wikimedia.org/P15003 and previous config saved to /var/cache/conftool/dbconfig/20210323-065947-marostegui.json

Stashbot added a comment.Mar 23 2021, 7:52 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-23T07:52:53Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1098:3317 to enable report_host T266483', diff saved to https://phabricator.wikimedia.org/P15016 and previous config saved to /var/cache/conftool/dbconfig/20210323-075253-marostegui.json

Marostegui added a comment.Mar 23 2021, 8:07 AM

With the crash that happened on labsdb1009 a couple of weeks ago, report_host is now enabled there. It doesn't really matter as we've excluded labsdb* hosts from orchestrator.

Marostegui updated the task description. (Show Details)Mar 23 2021, 8:08 AM
Marostegui added a comment.EditedMar 23 2021, 9:06 AM
  • labsdb1011 not needed
  • labsdb1010 not needed
  • labsdb1009 not needed
  • dbstore1003
  • db1184
  • db1169
  • db1164
  • db1163
  • db1154
  • db1140
  • db1139
  • db1135
  • db1134
  • db1133
  • db1119
  • db1118
  • db1106
  • db1105
  • db1099
  • db1083
  • clouddb1021
  • clouddb1017
  • clouddb1013
Marostegui updated the task description. (Show Details)Mar 23 2021, 9:07 AM
Marostegui added a subtask: T278214: Switchover s1 from db1083 to db1163.Mar 23 2021, 9:13 AM
Marostegui added a comment.Apr 26 2021, 12:45 PM

s1 is fully done only pending the master (T278214)

Marostegui added a comment.Apr 28 2021, 5:14 AM
In T266483#7033854, @Marostegui wrote:

s1 is fully done only pending the master (T278214)

done

Marostegui updated the task description. (Show Details)Apr 28 2021, 5:14 AM
Marostegui added a subtask: T280492: Upgrade all sanitarium masters to 10.4 and Buster.Apr 28 2021, 1:00 PM

s6 is fully done apart from the master (db1131)

Marostegui updated the task description. (Show Details)Apr 28 2021, 1:02 PM
Marostegui closed subtask T278214: Switchover s1 from db1083 to db1163 as Resolved.Apr 29 2021, 6:55 AM
Marostegui added a comment.Apr 29 2021, 11:41 AM

s2 is fully done apart from the master (db1122)

Marostegui updated the task description. (Show Details)Apr 29 2021, 11:41 AM
Marostegui added a comment.Apr 29 2021, 11:58 AM

s5 is fully done apart from the master (db1100)

Marostegui updated the task description. (Show Details)Apr 29 2021, 11:59 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 29 2021, 12:11 PM
Marostegui added a comment.Apr 29 2021, 2:00 PM

s3 is fully done apart from the master (db1123)

Marostegui updated the task description. (Show Details)Apr 29 2021, 2:01 PM
Stashbot added a comment.Apr 30 2021, 5:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-30T05:15:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1114 to enable report_host T266483', diff saved to https://phabricator.wikimedia.org/P15663 and previous config saved to /var/cache/conftool/dbconfig/20210430-051558-marostegui.json

Marostegui added a comment.EditedApr 30 2021, 5:18 AM

s8 eqiad:

  • labsdb1011 (not needed)
  • labsdb1010 (not needed)
  • labsdb1009 (not needed)
  • dbstore1005
  • db1177
  • db1172
  • db1167
  • db1154
  • db1126
  • db1116
  • db1114
  • db1111
  • db1109
  • db1104
  • db1101
  • db1099
  • db1087
  • clouddb1021
  • clouddb1020
  • clouddb1016
Marostegui updated the task description. (Show Details)Apr 30 2021, 5:18 AM
LSobanski moved this task from In progress to Blocked on the DBA board.May 4 2021, 10:51 AM
Marostegui added a comment.May 5 2021, 2:18 PM

s8 is fully done apart from the master (db1104)

Stashbot added a comment.May 17 2021, 6:01 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-17T06:01:32Z] <kormat> restarting mariadb on db1131 to pick up report_host T266483

Marostegui updated the task description. (Show Details)May 17 2021, 6:03 AM
Marostegui closed subtask T280492: Upgrade all sanitarium masters to 10.4 and Buster as Resolved.May 19 2021, 6:44 AM
Marostegui added a parent task: T285139: DB maintenance work to do while eqiad is passive (June 2021).Jun 18 2021, 11:14 AM
• Kormat updated the task description. (Show Details)Jun 21 2021, 6:03 AM
Marostegui updated the task description. (Show Details)Jun 22 2021, 7:40 AM
Marostegui added a comment.EditedJun 24 2021, 1:19 PM

Pending eqiad hosts:
[x] db1129 (slave)
[x] db1122
[x] db1104

Marostegui closed this task as Resolved.Jul 1 2021, 5:42 AM
Marostegui claimed this task.

This is all done and all hosts are now in Orchestrator

Marostegui updated the task description. (Show Details)Jul 1 2021, 5:42 AM
• Kormat awarded a token.Jul 1 2021, 8:36 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL