Page MenuHomePhabricator
Create Task
Maniphest T301848

Check for compressed templatelinks tables
Closed, ResolvedPublic
Actions

Description

Following the events at T301313 we should check and rebuild templatelinks tables on s3 (and the rest of pending sections) before they are attempted to get altered.

The following hosts need to be checked:

s1:

  • all clean

s2:

  • db1129.eqiad.wmnet:3306 (needs rebuild) (cleaned)
  • db1162.eqiad.wmnet:3306 (needs rebuild) (cleaned)
  • db2088.codfw.wmnet:3306 (needs rebuild) (cleaned)
  • db2104.codfw.wmnet:3306 (needs rebuild) (cleaned)
  • db2126.codfw.wmnet:3306 (needs rebuild) (cleaned)

s3:

  • dbstore1007:3313 (clean)
  • db1112.eqiad.wmnet:3306 (needs rebuild) (cleaned) if time allows, follow up: T301848#7716449
  • db1102.eqiad.wmnet:3313 (needs rebuild) (cleaned)
  • clouddb1021.eqiad.wmnet:3313 (clean)
  • clouddb1017.eqiad.wmnet:3313 (clean)
  • clouddb1013.eqiad.wmnet:3313 (clean)
  • db1179.eqiad.wmnet:3306 (needs rebuild) (cleaned)
  • db1175.eqiad.wmnet:3306 (needs rebuild) (cleaned)
  • db1166.eqiad.wmnet:3306 (needs rebuild) (cleaned)
  • db1157.eqiad.wmnet:3306 (master) (cleaned)
  • db1154.eqiad.wmnet:3313 (clean)

s4:

  • all clean

s7:

  • db1136.eqiad.wmnet:3306 ( master - needs rebuild) (cleaned)
  • db1158.eqiad.wmnet:3306 (needs rebuild) (cleaned)
  • db1174.eqiad.wmnet:3306 (needs rebuild) (cleaned)
  • db1181.eqiad.wmnet:3306 (needs rebuild) (cleaned)
  • db2077.codfw.wmnet:3306 (needs rebuild) (cleaned)

s8:

  • all clean

s5 and s6 were done already.

db2074, db2094 and db1123 are done

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
db2126,db2095: Disable notificationsoperations/puppetproduction+2 -0
db2088: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Feb 16 2022, 6:42 AM
Marostegui triaged this task as High priority.
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)Feb 16 2022, 7:47 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)
Marostegui added a comment.Feb 16 2022, 7:54 AM

db1112 needs to get the schema change with table rebuild, db1154 (sanitarium - db1112's slave is clean and so are clouddb* replicas)

Marostegui updated the task description. (Show Details)Feb 16 2022, 8:11 AM
Marostegui added a comment.Feb 16 2022, 8:18 AM

db1102 is running the schema change with ALGORITHM=COPY and it is progressing nicely and converting the table as they get altered.

Marostegui added a comment.Feb 16 2022, 8:23 AM

The master needs to be switched over, as it has lots of tables with Compact format and of course we cannot rebuild them while the host is acting as master.

Marostegui updated the task description. (Show Details)Feb 16 2022, 8:26 AM
Marostegui updated the task description. (Show Details)Feb 16 2022, 8:50 AM
Marostegui updated the task description. (Show Details)Feb 16 2022, 8:57 AM
gerritbot added a comment.Feb 16 2022, 11:30 AM

Change 763204 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2088: Disable notifications

https://gerrit.wikimedia.org/r/763204

gerritbot added a project: Patch-For-Review.Feb 16 2022, 11:30 AM

Change 763204 merged by Marostegui:

[operations/puppet@production] db2088: Disable notifications

https://gerrit.wikimedia.org/r/763204

Maintenance_bot removed a project: Patch-For-Review.Feb 16 2022, 12:10 PM
Marostegui added a comment.EditedFeb 16 2022, 1:23 PM

I am deploying T300775 on db1179, db1175, db1166, db1112 with table rebuilt so that will get cleaned up (on s3)

Marostegui updated the task description. (Show Details)Feb 16 2022, 5:04 PM
gerritbot added a comment.Feb 16 2022, 5:12 PM

Change 763280 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2126,db2095: Disable notifications

https://gerrit.wikimedia.org/r/763280

gerritbot added a project: Patch-For-Review.Feb 16 2022, 5:12 PM
Marostegui added a comment.Feb 16 2022, 5:13 PM

Going to clean the other s2 codfw hosts, so we don't have to run the rebuild on codfw.

gerritbot added a comment.Feb 16 2022, 5:13 PM

Change 763280 merged by Marostegui:

[operations/puppet@production] db2126,db2095: Disable notifications

https://gerrit.wikimedia.org/r/763280

Maintenance_bot removed a project: Patch-For-Review.Feb 16 2022, 6:10 PM
Marostegui updated the task description. (Show Details)Feb 16 2022, 9:27 PM

db2126 has been cleaned it shouldn't crash anymore when deploying the schema change. Pending s2 master, so we can deploy there with replication and no crashes.

Marostegui updated the task description. (Show Details)Feb 16 2022, 9:34 PM

db1112 is fully clean too, only a few templatelinks pages from closed wikis are pending (which won't be altered anyways). I might clean those up too once all this has gotten under control

Marostegui updated the task description. (Show Details)Feb 16 2022, 9:36 PM
Marostegui added a comment.Feb 21 2022, 6:08 AM

I am now rebuilding the last three s3 slaves: db1179, db1175, db1166.

LSobanski subscribed.Feb 21 2022, 9:06 AM
Marostegui updated the task description. (Show Details)Feb 21 2022, 9:29 AM
Marostegui added a comment.Feb 21 2022, 10:00 AM

Rebuilding db2104 (s2 codfw master), there'll be lag on s2 codfw while this happens

Stashbot added a comment.Feb 21 2022, 10:01 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-21T10:01:01Z] <marostegui> Rebuild templatelinks table on s2 codfw master (db2104), lag to be expected on codfw T301848

Stashbot added a comment.Feb 21 2022, 11:57 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-21T11:57:51Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1129 T301848', diff saved to https://phabricator.wikimedia.org/P21133 and previous config saved to /var/cache/conftool/dbconfig/20220221-115750-marostegui.json

Stashbot added a comment.Feb 21 2022, 11:58 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-21T11:58:11Z] <marostegui> Rebuild templatelinks table on db1129 (s2) T301848

Marostegui added a comment.Feb 21 2022, 11:59 AM

Fixing db1129 (s2)

Marostegui updated the task description. (Show Details)Feb 21 2022, 12:22 PM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Feb 21 2022, 12:36 PM
Stashbot added a comment.Feb 21 2022, 12:36 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-21T12:36:52Z] <marostegui> Rebuild templatelinks table on db2077 (s7) T301848

Marostegui added a comment.EditedFeb 21 2022, 12:37 PM

Fixing db2077 (its slave, db2095 is fine)

Marostegui mentioned this in T302222: Check and fix compressed mismatched tables.Feb 21 2022, 1:57 PM
Marostegui mentioned this in T301313: db2076, db2074 and db1123 crashed while altering templatelinks table.
Marostegui updated the task description. (Show Details)Feb 21 2022, 2:18 PM

db1129 fixed

Marostegui updated the task description. (Show Details)Feb 22 2022, 1:26 PM
Marostegui updated the task description. (Show Details)Feb 22 2022, 2:07 PM
Marostegui updated the task description. (Show Details)Feb 22 2022, 2:09 PM
Marostegui updated the task description. (Show Details)Mar 7 2022, 10:17 AM
Marostegui updated the task description. (Show Details)Mar 7 2022, 10:19 AM
Marostegui mentioned this in T300775: Add tl_target_id column to templatelinks.Mar 7 2022, 12:10 PM
Marostegui added a comment.Mar 8 2022, 6:18 AM

Fixing s7 replicas

Marostegui updated the task description. (Show Details)Mar 8 2022, 8:57 AM
Marostegui updated the task description. (Show Details)Mar 8 2022, 12:28 PM
Marostegui updated the task description. (Show Details)Mar 8 2022, 3:13 PM

All replicas are now fixed. Only pending s3 and s7 masters which will be done once we've switched them as part of the upgrades.

Marostegui mentioned this in T301850: Switchover s3 master (db1157 -> db1123).Mar 16 2022, 7:24 AM
Stashbot added a comment.Mar 29 2022, 6:11 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-29T06:11:29Z] <marostegui> Maintenance on db1157 (old s3 master) T301848

Marostegui added a comment.Mar 29 2022, 6:11 AM

Running this on old s3 master (db1157)

Marostegui closed subtask T301850: Switchover s3 master (db1157 -> db1123) as Resolved.Mar 29 2022, 6:12 AM
Marostegui updated the task description. (Show Details)Mar 29 2022, 6:19 AM
Marostegui updated the task description. (Show Details)Apr 19 2022, 7:17 AM
Stashbot added a comment.Apr 19 2022, 7:19 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-19T07:18:59Z] <marostegui> dbmaint s7@eqiad T301848

Marostegui closed this task as Resolved.Apr 19 2022, 11:18 AM
Marostegui updated the task description. (Show Details)

db1136, old s7 master, done. So everything is clean.

Maintenance_bot moved this task from In progress to Done on the DBA board.Apr 19 2022, 11:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits