This is the current list of candidate masters (taken from hiera) that need a mysql restart to pick up report_host flag enablement:
db[1075-1076,1093,1109,1130,1134,1136,1138,1141,1149].eqiad.wmnet
db[2080,2103-2104,2110,2113-2114,2121,2127].codfw.wmnet:
es[2020,2024].codfw.wmnet,es[1020,1023].eqiad.wmnet:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | Legoktm | T281515 June 2021 Datacenter switchover | |||
| Resolved | Marostegui | T285139 DB maintenance work to do while eqiad is passive (June 2021) | |||
| Resolved | Marostegui | T266483 Enable report_host for mariadb | |||
| Resolved | • Kormat | T271106 Enable report_host on candidate masters | |||
| Resolved | Marostegui | T271427 Switchover s4 (commonswiki) from db1081 to db1138 | |||
| Resolved | Trizek-WMF | T271791 Database primary master failover on s4 (commonswiki) | |||
| Resolved | • Kormat | T272954 Fix db-switchover update zarcillo part |
codfw ones should be easy, the rest would need depooling and we should probably take the opportunity to upgrade kernels if there're pending updates.
Mentioned in SAL (#wikimedia-operations) [2021-01-04T14:34:42Z] <marostegui> Upgrade and restart mysql on es2020 and es2024 - T271106
Mentioned in SAL (#wikimedia-operations) [2021-01-05T08:30:26Z] <marostegui> Restart db2127 T271106
# for i in es2020 es2024 db2127; do mysql.py -h$i -e "select @@report_host";done +--------------------+ | @@report_host | +--------------------+ | es2020.codfw.wmnet | +--------------------+ +--------------------+ | @@report_host | +--------------------+ | es2024.codfw.wmnet | +--------------------+ +--------------------+ | @@report_host | +--------------------+ | db2127.codfw.wmnet | +--------------------+
Mentioned in SAL (#wikimedia-operations) [2021-01-05T12:01:40Z] <marostegui> Restart db2121 T271106
Mentioned in SAL (#wikimedia-operations) [2021-01-12T10:13:19Z] <marostegui> Restart mysql on db1138 to pick up new config T271427 T271106
root@db1138:~# mysql -e "select @@report_host" +--------------------+ | @@report_host | +--------------------+ | db1138.eqiad.wmnet | +--------------------+
Mentioned in SAL (#wikimedia-operations) [2021-01-13T09:49:43Z] <marostegui> Enable report_host on all codfw sby masters - T271106
enabled on all candidate masters at codfw
db2080 +--------------------+ | @@report_host | +--------------------+ | db2080.codfw.wmnet | +--------------------+ db2103 +--------------------+ | @@report_host | +--------------------+ | db2103.codfw.wmnet | +--------------------+ db2104 +--------------------+ | @@report_host | +--------------------+ | db2104.codfw.wmnet | +--------------------+ db2110 +--------------------+ | @@report_host | +--------------------+ | db2110.codfw.wmnet | +--------------------+ db2113 +--------------------+ | @@report_host | +--------------------+ | db2113.codfw.wmnet | +--------------------+ db2114 +--------------------+ | @@report_host | +--------------------+ | db2114.codfw.wmnet | +--------------------+ db2121 +--------------------+ | @@report_host | +--------------------+ | db2121.codfw.wmnet | +--------------------+ db2127 +--------------------+ | @@report_host | +--------------------+ | db2127.codfw.wmnet | +--------------------+
Mentioned in SAL (#wikimedia-operations) [2021-01-13T09:59:05Z] <marostegui> Enable report_host on es1020 T271106
root@es1020:~# mysql -e "select @@report_host" +--------------------+ | @@report_host | +--------------------+ | es1020.eqiad.wmnet | +--------------------+
Mentioned in SAL (#wikimedia-operations) [2021-01-22T09:43:37Z] <kormat@cumin1001> dbctl commit (dc=all): 'Temporarily add db1088 to api group T271106', diff saved to https://phabricator.wikimedia.org/P13887 and previous config saved to /var/cache/conftool/dbconfig/20210122-094337-kormat.json
Script to check:
NODESET="db[1075-1076,1093,1109,1130,1134,1136,1138,1141,1149].eqiad.wmnet"
for inst in $(nodeset -e "$NODESET"); do
if [ "$(sudo -H mysql.py -h $inst -BAN -e "select @@report_host")" = "$inst" ]; then
echo "[x] ${inst%%.*}"
else
echo "[] ${inst%%.*}"
fi
doneMentioned in SAL (#wikimedia-operations) [2021-01-22T10:53:45Z] <kormat@cumin1001> dbctl commit (dc=all): 'Remove db1088 from api group T271106', diff saved to https://phabricator.wikimedia.org/P13900 and previous config saved to /var/cache/conftool/dbconfig/20210122-105345-kormat.json
Mentioned in SAL (#wikimedia-operations) [2021-01-22T11:24:25Z] <kormat@cumin1001> dbctl commit (dc=all): 'es1023 depooling: enable report_host T271106', diff saved to https://phabricator.wikimedia.org/P13911 and previous config saved to /var/cache/conftool/dbconfig/20210122-112424-kormat.json