Page MenuHomePhabricator
Create Task
Maniphest T382842

Upgrade to 10.6.20 and rebuild recentchanges and pagelinks tables
Closed, ResolvedPublic
Actions

Description

Let's try to upgrade to 10.6.20 quickly and once upgraded, rebuild the pagelinks and recentchanges tables with:

set session sql_log_bin=0;alter table pagelinks engine=innodb,force; alter table recentchanges engine=innodb,force;stop slave; start slave;

Upgrades

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
rebuild_tables.sh: Add sleepoperations/softwaremaster+1 -0
rebuild_tables.sh Add automatic repoolingoperations/softwaremaster+20 -3
db2155: Disable notificationsoperations/puppetproduction+1 -0
rebuild_tables.sh: Add downtimeoperations/softwaremaster+6 -0
rebuild_tables.sh: Add linteroperations/softwaremaster+1 -1
db1222: Disable notificationsoperations/puppetproduction+2 -1
rebuild_tables.sh: STOP and START SLAVEoperations/softwaremaster+1 -1
rebuild_tables.sh: Add start and finish timeoperations/softwaremaster+8 -0
rebuild_tables.sh: Quick script to rebuild tablesoperations/softwaremaster+39 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedMarosteguiT382842 Upgrade to 10.6.20 and rebuild recentchanges and pagelinks tables
 ResolvedMarosteguiT383401 db1170 replication broken
 ResolvedMarosteguiT384272 Upgrade and rebuild db2207 and db2148
 ResolvedMarosteguiT384801 db2182 depooled (Errmsg: Error Index for table recentchanges is corrupt)
 ResolvedMarosteguiT384799 db1241 corrupt index
 ResolvedMarosteguiT384807 Upgrade and rebuild s3
 ResolvedMarosteguiT385051 Switchover s3 master (db1189 -> db1223)
 InvalidNoneT385160 Switchover s3 master (db1189 -> db1223)
 ResolvedMarosteguiT385457 Switchover s3 master (db2209 -> db2205)
 ResolvedMarosteguiT384818 Upgrade dbstore* hosts to 10.6.20 and rebuild tables
 ResolvedMarosteguiT384994 Upgrade and rebuild s5
 ResolvedMarosteguiT385147 Switchover s5 master (db1183 -> db1230)
 ResolvedMarosteguiT385148 Switchover s5 master (db2213 -> db2192)
 ResolvedMarosteguiT385084 Upgrade and rebuild s2
 DeclinedMarosteguiT385576 Switchover s2 master (db1162 -> db1222)
 DeclinedMarosteguiT386213 Switchover s2 master (db2204 -> db2207)
 ResolvedMarosteguiT389376 Switchover s2 master (db2204 -> db2207)
 ResolvedMarosteguiT385550 Upgrade and rebuild s7
 ResolvedMarosteguiT387216 Switchover s7 master (db1181 -> db1236)
 ResolvedMarosteguiT387270 Switchover s7 master (db2218 -> db2220)
 ResolvedMarosteguiT385561 Upgrade and rebuild s1
 ResolvedMarosteguiT387552 Switchover s1 master (db1184 -> db1163)
 ResolvedMarosteguiT386242 Upgrade and rebuild s6
 ResolvedMarosteguiT387433 Switchover s6 master (db1201 -> db1173)
 ResolvedMarosteguiT389382 Switchover s6 master (db2214 -> db2229)
 ResolvedMarosteguiT389754 Reclone db2214, s6 candidate master
 ResolvedMarosteguiT387439 Upgrade and rebuild s4
 ResolvedMarosteguiT387816 Switchover s4 master (db1160 -> db1244)
 ResolvedMarosteguiT387441 Upgrade and rebuild s8
 ResolvedMarosteguiT387661 db2163 index corrutption
 ResolvedMarosteguiT388093 Switchover s8 master (db1209 -> db1193)
 ResolvedMarosteguiT389367 Switchover s8 master (db2165 -> db2161)
 ResolvedFCeratto-WMFT389373 Switchover s1 master (db2212 -> db2203)
 ResolvedFCeratto-WMFT389377 Switchover s3 master (db2205 -> db2209)
 ResolvedMarosteguiT389378 Switchover s4 master (db2179 -> db2240)
 ResolvedFCeratto-WMFT389381 Switchover s5 master (db2192 -> db2213)
 ResolvedFCeratto-WMFT389383 Switchover s7 master (db2220 -> db2218)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Maintenance_bot removed a project: Patch-For-Review.Jan 15 2025, 4:30 PM
Marostegui added a comment.Jan 20 2025, 9:16 AM

Rebuilding db2212, s1 candidate master which will be promoted to master tomorrow per T383690

Marostegui added a subtask: T384272: Upgrade and rebuild db2207 and db2148.Jan 21 2025, 6:29 AM
Marostegui closed subtask T384272: Upgrade and rebuild db2207 and db2148 as Resolved.Jan 21 2025, 9:46 AM
Marostegui added a comment.Jan 23 2025, 6:40 AM

Rebuilding db2166, which was powered off for on-site maintenance T383709

Marostegui added a comment.Jan 23 2025, 1:16 PM

db1162 s2 candidate master upgraded and being rebuilt to be able to become a master at T384597

Marostegui added a comment.Jan 23 2025, 4:25 PM

Rebuilding db2189, which was powered off for on-site maintenance T383709

gerritbot added a comment.Jan 23 2025, 4:47 PM

Change #1113844 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1222: Disable notifications

https://gerrit.wikimedia.org/r/1113844

gerritbot added a project: Patch-For-Review.Jan 23 2025, 4:47 PM

Change #1113844 merged by Marostegui:

[operations/puppet@production] db1222: Disable notifications

https://gerrit.wikimedia.org/r/1113844

gerritbot added a comment.Jan 23 2025, 4:51 PM

Change #1113847 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] rebuild_tables.sh: STOP and START SLAVE

https://gerrit.wikimedia.org/r/1113847

gerritbot added a comment.Jan 23 2025, 4:54 PM

Change #1113847 merged by jenkins-bot:

[operations/software@master] rebuild_tables.sh: STOP and START SLAVE

https://gerrit.wikimedia.org/r/1113847

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2025, 5:31 PM
Marostegui added subtasks: T384801: db2182 depooled (Errmsg: Error Index for table recentchanges is corrupt), T384799: db1241 corrupt index.Jan 27 2025, 8:41 AM
Marostegui closed subtask T384799: db1241 corrupt index as Resolved.
gerritbot added a comment.Jan 27 2025, 8:48 AM

Change #1114332 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] rebuild_tables.sh: Add linter

https://gerrit.wikimedia.org/r/1114332

gerritbot added a project: Patch-For-Review.Jan 27 2025, 8:48 AM
gerritbot added a comment.Jan 27 2025, 8:52 AM

Change #1114332 merged by jenkins-bot:

[operations/software@master] rebuild_tables.sh: Add linter

https://gerrit.wikimedia.org/r/1114332

gerritbot added a comment.Jan 27 2025, 8:57 AM

Change #1114333 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] rebuild_tables.sh: Add downtime

https://gerrit.wikimedia.org/r/1114333

gerritbot added a comment.Jan 27 2025, 8:59 AM

Change #1114333 merged by jenkins-bot:

[operations/software@master] rebuild_tables.sh: Add downtime

https://gerrit.wikimedia.org/r/1114333

Marostegui closed subtask T384801: db2182 depooled (Errmsg: Error Index for table recentchanges is corrupt) as Resolved.Jan 27 2025, 1:16 PM
Stashbot added a comment.Jan 28 2025, 7:58 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T07:58:58Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2155 T382842', diff saved to https://phabricator.wikimedia.org/P72559 and previous config saved to /var/cache/conftool/dbconfig/20250128-075857-marostegui.json

gerritbot added a comment.Jan 28 2025, 8:00 AM

Change #1114646 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2155: Disable notifications

https://gerrit.wikimedia.org/r/1114646

gerritbot added a comment.Jan 28 2025, 8:01 AM

Change #1114646 merged by Marostegui:

[operations/puppet@production] db2155: Disable notifications

https://gerrit.wikimedia.org/r/1114646

Marostegui added a comment.Jan 28 2025, 8:02 AM

Rebuilding db2155 as it needed a kernel reboot anyway, so I upgraded and starting to rebuild.

ops-monitoring-bot added a comment.Jan 28 2025, 8:07 AM

Icinga downtime and Alertmanager silence (ID=69b1841b-0478-45fc-a044-c5bf059f441f) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2155.codfw.wmnet
gerritbot added a comment.Jan 28 2025, 12:34 PM

Change #1114709 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] rebuild_tables.sh Add automatic repooling

https://gerrit.wikimedia.org/r/1114709

gerritbot added a comment.Jan 28 2025, 12:36 PM

Change #1114709 merged by jenkins-bot:

[operations/software@master] rebuild_tables.sh Add automatic repooling

https://gerrit.wikimedia.org/r/1114709

Stashbot added a comment.Jan 28 2025, 12:37 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T12:37:14Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1166 T382842', diff saved to https://phabricator.wikimedia.org/P72615 and previous config saved to /var/cache/conftool/dbconfig/20250128-123713-marostegui.json

Stashbot added a comment.Jan 28 2025, 4:18 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T16:18:58Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2177 T382842', diff saved to https://phabricator.wikimedia.org/P72640 and previous config saved to /var/cache/conftool/dbconfig/20250128-161857-marostegui.json

gerritbot added a comment.Jan 29 2025, 6:14 AM

Change #1114832 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] rebuild_tables.sh: Add sleep

https://gerrit.wikimedia.org/r/1114832

gerritbot added a comment.Jan 29 2025, 6:18 AM

Change #1114832 merged by jenkins-bot:

[operations/software@master] rebuild_tables.sh: Add sleep

https://gerrit.wikimedia.org/r/1114832

ops-monitoring-bot added a comment.Jan 30 2025, 11:22 AM

Icinga downtime and Alertmanager silence (ID=44325b56-6fa0-409b-9146-9232bd3dab5b) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2172.codfw.wmnet
ops-monitoring-bot added a comment.Jan 30 2025, 11:35 AM

Icinga downtime and Alertmanager silence (ID=f5355eda-8bda-46bf-a241-28d7fb1ffa11) set by root@cumin1002 for 12:00:00 on 1 host(s) and their services with reason: Index rebuild

db2224.codfw.wmnet
Marostegui closed subtask T384818: Upgrade dbstore* hosts to 10.6.20 and rebuild tables as Resolved.Jan 31 2025, 2:36 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 31 2025, 3:31 PM
Marostegui added a comment.Feb 3 2025, 11:40 AM

db1155 is having the tables rebuilt in all the sections.

Marostegui added a comment.Feb 3 2025, 11:41 AM

db2187 is having the tables rebuilt in all the sections.

Marostegui added a subtask: T385457: Switchover s3 master (db2209 -> db2205).Feb 3 2025, 1:03 PM
Marostegui closed subtask T385457: Switchover s3 master (db2209 -> db2205) as Resolved.Feb 3 2025, 1:23 PM
Marostegui closed subtask T384807: Upgrade and rebuild s3 as Resolved.Feb 6 2025, 6:46 AM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T384994: Upgrade and rebuild s5 as Resolved.Feb 7 2025, 5:29 PM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Feb 12 2025, 3:24 PM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)
ops-monitoring-bot added a comment.Feb 24 2025, 9:12 AM

Icinga downtime and Alertmanager silence (ID=585b8d47-6304-4235-a605-ec7d4d09ca3b) set by jynus@cumin1002 for 2 days, 0:00:00 on 1 host(s) and their services with reason: Upgrade and rebuild tables

db2199.codfw.wmnet
ops-monitoring-bot added a comment.Feb 24 2025, 9:37 AM

Icinga downtime and Alertmanager silence (ID=6b93bcb1-3388-433e-b2a1-bda5a75d87e5) set by jynus@cumin1002 for 2 days, 0:00:00 on 1 host(s) and their services with reason: Upgrade and rebuild tables

db2183.codfw.wmnet
ops-monitoring-bot added a comment.Feb 24 2025, 9:38 AM

Icinga downtime and Alertmanager silence (ID=2b92accc-a89e-4664-82b9-26fc22822d0a) set by jynus@cumin1002 for 2 days, 0:00:00 on 1 host(s) and their services with reason: Upgrade and rebuild tables

db2184.codfw.wmnet
Marostegui updated the task description. (Show Details)Feb 26 2025, 6:23 AM
Marostegui closed subtask T385550: Upgrade and rebuild s7 as Resolved.Feb 26 2025, 10:30 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Feb 27 2025, 10:00 AM
Marostegui updated the task description. (Show Details)Feb 27 2025, 10:39 AM
Marostegui added a comment.Mar 6 2025, 11:10 AM

All hosts are done. Pending only s4 codfw masters, which will be done once codfw is no longer primary DC T385155: 🧭 Northward Datacentre Switchover (March 2025) (scheduled for: MediaWiki: Wednesday, 19 March 2025 @ 14:00 UTC)

Marostegui changed the task status from Open to Stalled.Mar 6 2025, 11:10 AM
Marostegui claimed this task.
Marostegui changed the task status from Stalled to Open.Mar 19 2025, 2:46 PM
FCeratto-WMF added subtasks: T389373: Switchover s1 master (db2212 -> db2203), T389376: Switchover s2 master (db2204 -> db2207), T389377: Switchover s3 master (db2205 -> db2209), T389378: Switchover s4 master (db2179 -> db2240), T389381: Switchover s5 master (db2192 -> db2213), T389382: Switchover s6 master (db2214 -> db2229).Mar 19 2025, 3:52 PM
FCeratto-WMF added a subtask: T389383: Switchover s7 master (db2220 -> db2218).Mar 19 2025, 3:59 PM
Marostegui closed subtask T387441: Upgrade and rebuild s8 as Resolved.Mar 20 2025, 12:01 PM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T389376: Switchover s2 master (db2204 -> db2207) as Resolved.Mar 24 2025, 6:26 AM
Marostegui closed subtask T389382: Switchover s6 master (db2214 -> db2229) as Resolved.Mar 24 2025, 7:04 AM
FCeratto-WMF changed the status of subtask T389373: Switchover s1 master (db2212 -> db2203) from Open to In Progress.Mar 24 2025, 10:24 AM
FCeratto-WMF closed subtask T389373: Switchover s1 master (db2212 -> db2203) as Resolved.Mar 24 2025, 11:18 AM
Marostegui closed subtask T386242: Upgrade and rebuild s6 as Resolved.Mar 24 2025, 11:47 AM
Marostegui updated the task description. (Show Details)
FCeratto-WMF changed the status of subtask T389377: Switchover s3 master (db2205 -> db2209) from Open to In Progress.Mar 24 2025, 11:51 AM
FCeratto-WMF closed subtask T389377: Switchover s3 master (db2205 -> db2209) as Resolved.Mar 24 2025, 12:13 PM
Marostegui closed subtask T389378: Switchover s4 master (db2179 -> db2240) as Resolved.Mar 24 2025, 2:52 PM
Marostegui closed subtask T385084: Upgrade and rebuild s2 as Resolved.Mar 24 2025, 3:30 PM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T385561: Upgrade and rebuild s1 as Resolved.Mar 24 2025, 4:49 PM
Marostegui updated the task description. (Show Details)
Marostegui closed this task as Resolved.Mar 25 2025, 6:12 AM
Marostegui closed subtask T387439: Upgrade and rebuild s4 as Resolved.
Marostegui updated the task description. (Show Details)

All done!

Maintenance_bot moved this task from In progress to Done on the DBA board.Mar 25 2025, 6:29 AM
FCeratto-WMF changed the status of subtask T389381: Switchover s5 master (db2192 -> db2213) from Open to In Progress.Mar 25 2025, 9:55 AM
FCeratto-WMF closed subtask T389381: Switchover s5 master (db2192 -> db2213) as Resolved.Mar 25 2025, 10:13 AM
FCeratto-WMF changed the status of subtask T389383: Switchover s7 master (db2220 -> db2218) from Open to In Progress.Mar 25 2025, 10:43 AM
FCeratto-WMF closed subtask T389383: Switchover s7 master (db2220 -> db2218) as Resolved.Mar 25 2025, 11:22 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits