Page MenuHomePhabricator
Create Task
Maniphest T384807

Upgrade and rebuild s3
Closed, ResolvedPublic
Actions

Description

Let's proactively rebuild s3, for the following tables:linter recentchanges pagelinks

  • db2239
  • db2227
  • db2209
  • db2205
  • db2194
  • db2190
  • db2186
  • db2177
  • db2156
  • db2149
  • db2139 decommissioned
  • db1240
  • db1223
  • db1212
  • db1198
  • db1189
  • db1175
  • db1166
  • db1157
  • db1154
  • db1150
  • an-redacteddb-1001
  • cloudb*
    • clouddb1013
    • clouddb1017
    • an-redacteddb-1001

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Jan 27 2025, 12:22 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-27T12:22:50Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1223 (re)pooling @ 50%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72471 and previous config saved to /var/cache/conftool/dbconfig/20250127-122249-root.json

Marostegui updated the task description. (Show Details)Jan 27 2025, 12:30 PM
Stashbot added a comment.Jan 27 2025, 12:37 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-27T12:37:56Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1223 (re)pooling @ 75%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72473 and previous config saved to /var/cache/conftool/dbconfig/20250127-123754-root.json

Stashbot added a comment.Jan 27 2025, 12:53 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-27T12:53:02Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1223 (re)pooling @ 100%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72476 and previous config saved to /var/cache/conftool/dbconfig/20250127-125301-root.json

Stashbot added a comment.Jan 27 2025, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-27T13:27:10Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1157 T384807', diff saved to https://phabricator.wikimedia.org/P72480 and previous config saved to /var/cache/conftool/dbconfig/20250127-132710-marostegui.json

Marostegui updated the task description. (Show Details)Jan 27 2025, 1:28 PM
Marostegui updated the task description. (Show Details)Jan 27 2025, 3:49 PM
Stashbot added a comment.Jan 27 2025, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-27T15:56:13Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2194 T384807', diff saved to https://phabricator.wikimedia.org/P72495 and previous config saved to /var/cache/conftool/dbconfig/20250127-155613-marostegui.json

Marostegui updated the task description. (Show Details)Jan 27 2025, 4:02 PM
ops-monitoring-bot added a comment.Jan 27 2025, 4:02 PM

Icinga downtime and Alertmanager silence (ID=537f82b4-1ee3-4614-90b3-f799f6e91387) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2194.codfw.wmnet
Stashbot added a comment.Jan 27 2025, 4:07 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-27T16:07:45Z] <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2194.codfw.wmnet with reason: Index rebuild T384807

Marostegui updated the task description. (Show Details)Jan 27 2025, 4:19 PM
Marostegui updated the task description. (Show Details)Jan 27 2025, 6:53 PM
Stashbot added a comment.Jan 28 2025, 6:12 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T06:12:31Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2205 T384807', diff saved to https://phabricator.wikimedia.org/P72555 and previous config saved to /var/cache/conftool/dbconfig/20250128-061230-marostegui.json

ops-monitoring-bot added a comment.Jan 28 2025, 6:16 AM

Icinga downtime and Alertmanager silence (ID=50efbd9c-6b49-4071-80f5-ea05c52051f6) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2205.codfw.wmnet
ops-monitoring-bot added a comment.Jan 28 2025, 6:25 AM

Icinga downtime and Alertmanager silence (ID=128b7e67-c62e-41e7-914f-3ee9606a4d66) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db1175.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 28 2025, 6:25 AM

Icinga downtime and Alertmanager silence (ID=5bc22b5d-85bb-4be9-b5f3-57dc3401ad26) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2205.codfw.wmnet
Marostegui updated the task description. (Show Details)Jan 28 2025, 6:25 AM
Stashbot added a comment.Jan 28 2025, 6:28 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T06:28:47Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1175 T384807', diff saved to https://phabricator.wikimedia.org/P72556 and previous config saved to /var/cache/conftool/dbconfig/20250128-062846-marostegui.json

Marostegui updated the task description. (Show Details)Jan 28 2025, 8:49 AM
Stashbot added a comment.Jan 28 2025, 9:18 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T09:18:47Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1175 (re)pooling @ 10%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72574 and previous config saved to /var/cache/conftool/dbconfig/20250128-091846-root.json

Stashbot added a comment.Jan 28 2025, 9:33 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T09:33:52Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72576 and previous config saved to /var/cache/conftool/dbconfig/20250128-093352-root.json

Marostegui updated the task description. (Show Details)Jan 28 2025, 9:40 AM
ops-monitoring-bot added a comment.Jan 28 2025, 9:41 AM

Icinga downtime and Alertmanager silence (ID=14b1d18a-1c1b-46da-8f16-91c6a015cf8c) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2190.codfw.wmnet
Stashbot added a comment.Jan 28 2025, 9:48 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T09:48:58Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72581 and previous config saved to /var/cache/conftool/dbconfig/20250128-094857-root.json

Stashbot added a comment.Jan 28 2025, 10:04 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T10:04:03Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72585 and previous config saved to /var/cache/conftool/dbconfig/20250128-100402-root.json

Stashbot added a comment.Jan 28 2025, 10:12 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T10:12:25Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2155 T384807', diff saved to https://phabricator.wikimedia.org/P72589 and previous config saved to /var/cache/conftool/dbconfig/20250128-101224-marostegui.json

Stashbot added a comment.Jan 28 2025, 10:19 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T10:19:09Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72591 and previous config saved to /var/cache/conftool/dbconfig/20250128-101908-root.json

Stashbot added a comment.Jan 28 2025, 10:44 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T10:44:37Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es2030 (re)pooling @ 10%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72597 and previous config saved to /var/cache/conftool/dbconfig/20250128-104436-root.json

Stashbot added a comment.Jan 28 2025, 10:59 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T10:59:42Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es2030 (re)pooling @ 25%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72599 and previous config saved to /var/cache/conftool/dbconfig/20250128-105942-root.json

Stashbot added a comment.Jan 28 2025, 11:14 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T11:14:48Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es2030 (re)pooling @ 50%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72601 and previous config saved to /var/cache/conftool/dbconfig/20250128-111447-root.json

Stashbot added a comment.Jan 28 2025, 11:29 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T11:29:53Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es2030 (re)pooling @ 75%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72604 and previous config saved to /var/cache/conftool/dbconfig/20250128-112952-root.json

Stashbot added a comment.Jan 28 2025, 11:44 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T11:44:58Z] <marostegui@cumin1002> dbctl commit (dc=all): 'es2030 (re)pooling @ 100%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72607 and previous config saved to /var/cache/conftool/dbconfig/20250128-114458-root.json

Marostegui updated the task description. (Show Details)Jan 28 2025, 12:25 PM
ops-monitoring-bot added a comment.Jan 28 2025, 12:27 PM

Icinga downtime and Alertmanager silence (ID=b459121d-b50b-407b-834f-42f1087285ab) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2230.codfw.wmnet
ops-monitoring-bot added a comment.Jan 28 2025, 12:27 PM

Icinga downtime and Alertmanager silence (ID=4de4d9eb-b61e-46c3-8b68-deaa846e9549) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2230.codfw.wmnet
ops-monitoring-bot added a comment.Jan 28 2025, 12:39 PM

Icinga downtime and Alertmanager silence (ID=c02e47a3-10c9-4304-9f41-99bc14749c48) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db1166.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 28 2025, 1:27 PM

Icinga downtime and Alertmanager silence (ID=b6f4ad5a-e548-4238-9697-54e17f7c6af7) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db1166.eqiad.wmnet
Stashbot added a comment.Jan 28 2025, 4:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T16:03:02Z] <root@cumin1002> START - Cookbook sre.mysql.pool db1166 gradually with 4 steps - Repooling after rebuild index T384807

Stashbot added a comment.Jan 28 2025, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T16:04:56Z] <root@cumin1002> END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) db1166 gradually with 4 steps - Repooling after rebuild index T384807

Marostegui updated the task description. (Show Details)Jan 28 2025, 4:17 PM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Jan 28 2025, 4:19 PM
ops-monitoring-bot added a comment.Jan 28 2025, 4:26 PM

Icinga downtime and Alertmanager silence (ID=e1cf339b-ef67-4db2-b5bf-2ef550644b49) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2177.codfw.wmnet
Marostegui updated the task description. (Show Details)Jan 28 2025, 7:47 PM
ops-monitoring-bot added a comment.Jan 29 2025, 6:20 AM

Icinga downtime and Alertmanager silence (ID=06ccd3aa-ef8e-4305-bdcb-f3694f57fd35) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2156.codfw.wmnet
Marostegui updated the task description. (Show Details)Jan 29 2025, 6:41 AM
Stashbot added a comment.Jan 29 2025, 9:31 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-29T09:31:29Z] <root@cumin1002> START - Cookbook sre.mysql.pool db2156 gradually with 4 steps - Repooling after rebuild index T384807

Marostegui updated the task description. (Show Details)Jan 29 2025, 9:47 AM
Marostegui updated the task description. (Show Details)Jan 29 2025, 10:02 AM
Stashbot added a comment.Jan 29 2025, 10:16 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-29T10:16:51Z] <root@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2156 gradually with 4 steps - Repooling after rebuild index T384807

Marostegui updated the task description. (Show Details)Jan 29 2025, 10:50 AM
Marostegui updated the task description. (Show Details)Jan 29 2025, 12:01 PM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Jan 29 2025, 12:48 PM
Stashbot added a comment.Jan 29 2025, 12:50 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-29T12:50:16Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1212 T384807', diff saved to https://phabricator.wikimedia.org/P72741 and previous config saved to /var/cache/conftool/dbconfig/20250129-125015-marostegui.json

Marostegui updated the task description. (Show Details)Jan 29 2025, 12:50 PM
ops-monitoring-bot added a comment.Jan 29 2025, 12:56 PM

Icinga downtime and Alertmanager silence (ID=26ec8a07-56ae-4da5-8858-6a714de99985) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db1212.eqiad.wmnet
Marostegui updated the task description. (Show Details)Jan 29 2025, 3:28 PM
Marostegui added a subtask: T385051: Switchover s3 master (db1189 -> db1223).
Marostegui added a subtask: T385160: Switchover s3 master (db1189 -> db1223).Jan 30 2025, 11:20 AM
Marostegui closed subtask T385160: Switchover s3 master (db1189 -> db1223) as Invalid.
Marostegui closed subtask T385051: Switchover s3 master (db1189 -> db1223) as Resolved.Jan 31 2025, 11:31 AM
Marostegui updated the task description. (Show Details)
ops-monitoring-bot added a comment.Jan 31 2025, 11:40 AM

Icinga downtime and Alertmanager silence (ID=bad66e06-06de-4ed5-9453-9c1040a7b178) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db1189.eqiad.wmnet
Marostegui updated the task description. (Show Details)Jan 31 2025, 12:02 PM
Marostegui updated the task description. (Show Details)Jan 31 2025, 2:34 PM
Marostegui added a subtask: T385457: Switchover s3 master (db2209 -> db2205).Feb 3 2025, 1:22 PM
Marostegui closed subtask T385457: Switchover s3 master (db2209 -> db2205) as Resolved.
ops-monitoring-bot added a comment.Feb 3 2025, 1:28 PM

Icinga downtime and Alertmanager silence (ID=808bdf65-caf8-46e1-9256-a7ba84c6f6cf) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2209.codfw.wmnet
Marostegui updated the task description. (Show Details)Feb 3 2025, 1:29 PM
Marostegui updated the task description. (Show Details)Feb 4 2025, 7:46 AM
Marostegui updated the task description. (Show Details)Feb 4 2025, 7:49 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Feb 4 2025, 9:52 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Feb 5 2025, 11:55 AM
Marostegui updated the task description. (Show Details)Feb 6 2025, 6:44 AM
Marostegui closed this task as Resolved.Feb 6 2025, 6:46 AM
Marostegui mentioned this in T382842: Upgrade to 10.6.20 and rebuild recentchanges and pagelinks tables.
Maintenance_bot moved this task from In progress to Done on the DBA board.Feb 6 2025, 7:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits