Page MenuHomePhabricator
Create Task
Maniphest T309303

Move db1128 from m1 (misc) to s1 (mediawiki)
Closed, ResolvedPublic
Actions

Description

After a just-in-case backup of db1128 finishes and hw issues have ben solved/mitigated, place db1128 into s1, in substitution of db1164.

  • Backup has been made
  • Make puppet changes to place db1128 in s1
  • Clone db1128 out of some s1 host (cloned from db1184)
  • Pool db1128 into s1

Details

SubjectRepoBranchLines +/-
db1128: Enanble notificationsoperations/puppetproduction+0 -2
db1128: Add it to dbctloperations/puppetproduction+1 -0
db1184: Disable notificationsoperations/puppetproduction+1 -0
mariadb: Move db1128 from m1 to s1operations/puppetproduction+3 -10
site.pp: db1128 current situationoperations/puppetproduction+2 -1
mariabackup: Make the vendor detection account for known variationsoperations/software/wmfbackupsmaster+63 -7
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo created this task.May 26 2022, 11:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 26 2022, 11:59 AM
Ladsgroup subscribed.May 26 2022, 12:01 PM

Didn't we take the db from s4? Just double checking.

jcrespo added a comment.May 26 2022, 12:01 PM

This is a useless comment done just because phabricator notification doesn't work for direct subscribers on first task creation, so writing it so Manuel is aware of the ticket creation (this was what happened on a previous ticket for this incident).

jcrespo added a comment.May 26 2022, 12:01 PM
In T309303#7960237, @Ladsgroup wrote:

Didn't we take the db from s4? Just double checking.

No idea, manuel to say.

jcrespo renamed this task from Move db1128 from m1 (misc) to s1 (mediawiki) to Move db1128 from m1 (misc) to sX (mediawiki).May 26 2022, 12:01 PM
jcrespo updated the task description. (Show Details)
jcrespo mentioned this in T309296: Failover m1 primary db from db1128 to db1164.May 26 2022, 12:06 PM
jcrespo updated the task description. (Show Details)May 26 2022, 12:18 PM
• Marostegui renamed this task from Move db1128 from m1 (misc) to sX (mediawiki) to Move db1128 from m1 (misc) to s1 (mediawiki).May 26 2022, 1:15 PM
• Marostegui claimed this task.
• Marostegui triaged this task as High priority.
• Marostegui updated the task description. (Show Details)
• Marostegui moved this task from Triage to Blocked on the DBA board.
In T309303#7960237, @Ladsgroup wrote:

Didn't we take the db from s4? Just double checking.

No, s1

• Marostegui mentioned this in T309286: db1128 host (containing m1 databases) crashed.May 27 2022, 5:19 AM
jcrespo updated the task description. (Show Details)May 27 2022, 6:28 AM
jcrespo added a comment.May 27 2022, 6:33 AM

Backup completed, stored on dbprov1001.

RhinosF1 subscribed.May 27 2022, 6:38 AM
gerritbot added a comment.May 27 2022, 11:15 AM

Change 800708 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/wmfbackups@master] mariabackup: Make the vendor detection account for known variations

https://gerrit.wikimedia.org/r/800708

gerritbot added a project: Patch-For-Review.May 27 2022, 11:15 AM
gerritbot added a comment.May 27 2022, 12:22 PM

Change 800708 merged by Jcrespo:

[operations/software/wmfbackups@master] mariabackup: Make the vendor detection account for known variations

https://gerrit.wikimedia.org/r/800708

• Marostegui moved this task from Blocked to In progress on the DBA board.May 27 2022, 12:27 PM
• Marostegui added a subtask: T309291: db1128 faulty memory.
Maintenance_bot removed a project: Patch-For-Review.May 27 2022, 12:30 PM
gerritbot added a comment.May 27 2022, 12:59 PM

Change 800720 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: db1128 current situation

https://gerrit.wikimedia.org/r/800720

gerritbot added a project: Patch-For-Review.May 27 2022, 12:59 PM

Change 800720 merged by Marostegui:

[operations/puppet@production] site.pp: db1128 current situation

https://gerrit.wikimedia.org/r/800720

Maintenance_bot removed a project: Patch-For-Review.May 27 2022, 1:30 PM
• Marostegui added a comment.May 30 2022, 5:15 AM

db1184 will be used to clone db1128

gerritbot added a comment.May 30 2022, 5:15 AM

Change 801019 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1184: Disable notifications

https://gerrit.wikimedia.org/r/801019

gerritbot added a project: Patch-For-Review.May 30 2022, 5:15 AM

Change 801019 merged by Marostegui:

[operations/puppet@production] db1184: Disable notifications

https://gerrit.wikimedia.org/r/801019

gerritbot added a comment.May 30 2022, 5:22 AM

Change 801086 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move db1128 from m1 to s1

https://gerrit.wikimedia.org/r/801086

gerritbot added a comment.May 30 2022, 5:24 AM

Change 801086 merged by Marostegui:

[operations/puppet@production] mariadb: Move db1128 from m1 to s1

https://gerrit.wikimedia.org/r/801086

Maintenance_bot removed a project: Patch-For-Review.May 30 2022, 5:30 AM
• Marostegui updated the task description. (Show Details)May 30 2022, 5:43 AM
• Marostegui updated the task description. (Show Details)May 30 2022, 9:51 AM

db1128 has been recloned. I have installed 10.4.25 there for what is worth. I won't repool it until it has gotten the new memory (or replaced the bad DIMM) as it could fail anytime.

• Marostegui added a comment.May 30 2022, 11:09 AM

db1128 caught up - GTID enabled. Leaving the host depooled until the memory is replaced

• Marostegui added a comment.May 30 2022, 11:09 AM

Starting to repool db1184

gerritbot added a comment.Jun 3 2022, 5:21 AM

Change 802689 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1128: Add it to dbctl

https://gerrit.wikimedia.org/r/802689

gerritbot added a project: Patch-For-Review.Jun 3 2022, 5:21 AM
gerritbot added a comment.Jun 3 2022, 5:25 AM

Change 802689 merged by Marostegui:

[operations/puppet@production] db1128: Add it to dbctl

https://gerrit.wikimedia.org/r/802689

• Cmjohnson closed subtask T309291: db1128 faulty memory as Resolved.Jun 3 2022, 1:21 PM
• Marostegui added a comment.Jun 4 2022, 9:39 AM

Chris replaced the DIMM on Friday - I have just started mysql and its replication

gerritbot added a comment.Jun 6 2022, 5:00 AM

Change 802945 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1128: Enanble notifications

https://gerrit.wikimedia.org/r/802945

gerritbot added a comment.Jun 6 2022, 5:02 AM

Change 802945 merged by Marostegui:

[operations/puppet@production] db1128: Enanble notifications

https://gerrit.wikimedia.org/r/802945

Stashbot added a comment.Jun 6 2022, 5:06 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-06T05:06:16Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1128 to dbctl T309303', diff saved to https://phabricator.wikimedia.org/P29418 and previous config saved to /var/cache/conftool/dbconfig/20220606-050616-marostegui.json

Stashbot added a comment.Jun 6 2022, 5:07 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-06T05:07:07Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1128 on s1 with small weight after DIMM replacement T309303', diff saved to https://phabricator.wikimedia.org/P29419 and previous config saved to /var/cache/conftool/dbconfig/20220606-050707-root.json

• Marostegui added a comment.Jun 6 2022, 5:07 AM

Starting to pool db1128 into s1.

• Marostegui added a comment.Jun 6 2022, 6:01 AM

Started automatic repooling after checking that all seems to be working fine

• Marostegui closed this task as Resolved.Jun 6 2022, 6:26 AM
• Marostegui updated the task description. (Show Details)

It will get pooled automatically throughout the day, so closing this as fixed.

Maintenance_bot moved this task from In progress to Done on the DBA board.Jun 6 2022, 6:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL