Let's proactively rebuild s5, for the following tables: linter recentchanges pagelinks
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| db1230,db2157: Disable notifications | operations/puppet | production | +3 -2 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T382842 Upgrade to 10.6.20 and rebuild recentchanges and pagelinks tables | |||
| Resolved | Marostegui | T384994 Upgrade and rebuild s5 | |||
| Resolved | Marostegui | T385147 Switchover s5 master (db1183 -> db1230) | |||
| Resolved | Marostegui | T385148 Switchover s5 master (db2213 -> db2192) |
Mentioned in SAL (#wikimedia-operations) [2025-01-29T06:45:46Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1230 db2157 T384994', diff saved to https://phabricator.wikimedia.org/P72693 and previous config saved to /var/cache/conftool/dbconfig/20250129-064545-marostegui.json
Change #1114904 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1230,db2157: Disable notifications
Change #1114904 merged by Marostegui:
[operations/puppet@production] db1230,db2157: Disable notifications
Icinga downtime and Alertmanager silence (ID=86ebeba6-2d2c-4803-8858-91cbce85548a) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db1230.eqiad.wmnet
Icinga downtime and Alertmanager silence (ID=b82e417c-2f16-402f-9961-ca2be6e9a7bf) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db2157.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2025-01-29T07:40:45Z] <root@cumin1002> START - Cookbook sre.mysql.pool db1230 gradually with 4 steps - Repooling after rebuild index T384994
Mentioned in SAL (#wikimedia-operations) [2025-01-29T07:40:49Z] <root@cumin1002> END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) db1230 gradually with 4 steps - Repooling after rebuild index T384994
Mentioned in SAL (#wikimedia-operations) [2025-01-29T07:58:40Z] <root@cumin1002> START - Cookbook sre.mysql.pool db2157 gradually with 4 steps - Repooling after rebuild index T384994
Mentioned in SAL (#wikimedia-operations) [2025-01-29T08:44:01Z] <root@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2157 gradually with 4 steps - Repooling after rebuild index T384994
Mentioned in SAL (#wikimedia-operations) [2025-01-29T08:46:11Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1210 db2223 T384994', diff saved to https://phabricator.wikimedia.org/P72707 and previous config saved to /var/cache/conftool/dbconfig/20250129-084611-marostegui.json
Icinga downtime and Alertmanager silence (ID=dec5e6b3-6b49-49d4-8e89-fa63710c1267) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db1210.eqiad.wmnet
Icinga downtime and Alertmanager silence (ID=f9930502-7464-400f-a540-8914ac98a0e3) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db2223.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2025-01-29T12:02:14Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2211 T384994', diff saved to https://phabricator.wikimedia.org/P72738 and previous config saved to /var/cache/conftool/dbconfig/20250129-120213-marostegui.json
Icinga downtime and Alertmanager silence (ID=d5c30bc5-21e3-44b5-992e-5f9440acb386) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db1200.eqiad.wmnet
Icinga downtime and Alertmanager silence (ID=6ba0f8a7-2442-41ca-bf9a-fe2bdc14b01a) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db2211.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2025-01-29T14:33:17Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1185 db2178 T384994', diff saved to https://phabricator.wikimedia.org/P72756 and previous config saved to /var/cache/conftool/dbconfig/20250129-143317-marostegui.json
Icinga downtime and Alertmanager silence (ID=6cab907d-0410-4ee8-9872-07efea8a1af9) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db1185.eqiad.wmnet
Icinga downtime and Alertmanager silence (ID=55a65570-1527-41c1-acc2-db48d1a634cd) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db2178.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2025-01-29T16:59:52Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1159 T384994', diff saved to https://phabricator.wikimedia.org/P72787 and previous config saved to /var/cache/conftool/dbconfig/20250129-165951-marostegui.json
Icinga downtime and Alertmanager silence (ID=f50cd93a-2f3f-49fb-86ab-01a6eb754d28) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db1159.eqiad.wmnet
Mentioned in SAL (#wikimedia-operations) [2025-01-30T07:42:28Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1161 T384994', diff saved to https://phabricator.wikimedia.org/P72833 and previous config saved to /var/cache/conftool/dbconfig/20250130-074228-marostegui.json
Icinga downtime and Alertmanager silence (ID=c4581d24-e675-446f-9964-7ca1f43b1b90) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db1161.eqiad.wmnet
Icinga downtime and Alertmanager silence (ID=0b7c9e8a-8699-4caf-810a-7e1bcdecc95d) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db1183.eqiad.wmnet
Icinga downtime and Alertmanager silence (ID=e02af35a-4d65-4168-8cdf-d64df75eace2) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild
db2213.codfw.wmnet