Page MenuHomePhabricator
Create Task
Maniphest T214264

BBU issues on codfw
Closed, ResolvedPublic
Actions

Description

This is a tracking task for all the BBU issues we have on codfw.
All those hosts are out of warranty so we probably just have to replace them with new hardware instead of buying a new BBU

Handed over for decommission T220002: dbstore2002 temporary backup host T205257
Handed over for decommission T225090: db2042 m3 master T209261
Handed over for decommission T224079: db2040 s7 old master (replaced after BBU failed: T214264#4897163)
Handed over for decommission T220070: db2033 x1 slave T184888

If needed:

  • Force write back
hpssacli controller all show detail | grep "Drive Write Cache"
hpssacli ctrl slot=0 modify dwc=enable
hpssacli controller all show detail | grep "Drive Write Cache"
  • Go back to default policy:
hpssacli ctrl slot=0 modify dwc=disable

Details

ProjectBranchLines +/-Subject
operations/mediawiki-configmaster+1 -1mariadb: Repool db2040 after maintenance
operations/puppetproduction+2 -6mariadb: Demote db2040 from being an s7 master to just a replica
operations/mediawiki-configmaster+3 -3mariadb: Depool db2040, promote db2047 to master of s7 section
operations/puppetproduction+6 -2mariadb: Promote db2047 to master on configuration management
Customize query in gerrit

Related Objects

Event Timeline

Marostegui created this task.Jan 20 2019, 2:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 20 2019, 2:56 PM
Marostegui triaged this task as Medium priority.Jan 20 2019, 2:57 PM
Marostegui merged a task: T205257: BBU problems dbstore2002.
Marostegui merged a task: T209261: Failover m3 codfw master.
Marostegui moved this task from Triage to Meta/Epic on the DBA board.
Marostegui added subscribers: • Banyek, Stashbot, Papaul.
Marostegui added a subscriber: Volans.
Marostegui removed subscribers: Volans, Papaul, • Banyek, Marostegui.
Marostegui updated the task description. (Show Details)Jan 20 2019, 3:08 PM
Stashbot added a comment.Jan 20 2019, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-20T15:13:17Z] <marostegui> Force WriteBack on db2040 - T214264

Marostegui added a comment.Jan 20 2019, 3:13 PM

db2040 was really delayed, and I checked that the actor migration script finished yesterday, so it is probably not because of that but because of the failed BBU and the policy being back to WriteThru.
I have forced WriteBack there to let it catch up.

Marostegui added a comment.Jan 20 2019, 8:09 PM

db2040 caught up, and I switched back to WriteThru and it wasn't able to keep up with replication.
So I have forced WB again, let's promote db2047 to master and make db2040 a replica.

jcrespo claimed this task.Jan 21 2019, 9:10 AM
jcrespo moved this task from Meta/Epic to In progress on the DBA board.
gerritbot added a subscriber: gerritbot.Jan 21 2019, 10:23 AM

Change 485617 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Promote db2047 to master on configuration management

https://gerrit.wikimedia.org/r/485617

gerritbot added a project: Patch-For-Review.Jan 21 2019, 10:23 AM
Stashbot added a comment.Jan 21 2019, 10:33 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-21T10:33:24Z] <jynus> upgrade and restart db2047 T214264

Stashbot added a comment.Jan 21 2019, 5:44 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-21T17:44:33Z] <jynus> stop replication on db2040 for master switch T214264

gerritbot added a comment.Jan 21 2019, 5:50 PM

Change 485617 merged by Jcrespo:
[operations/puppet@production] mariadb: Promote db2047 to master on configuration management

https://gerrit.wikimedia.org/r/485617

Stashbot added a comment.Jan 21 2019, 5:51 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-21T17:51:33Z] <jynus> stop and apply puppet changes to db2047 T214264

gerritbot added a comment.Jan 21 2019, 6:17 PM

Change 485696 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db2040, promote db2047 to master of s7 section

https://gerrit.wikimedia.org/r/485696

gerritbot added a comment.Jan 21 2019, 6:22 PM

Change 485696 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db2040, promote db2047 to master of s7 section

https://gerrit.wikimedia.org/r/485696

gerritbot added a comment.Jan 21 2019, 6:26 PM

Change 485698 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db2040 after maintenance

https://gerrit.wikimedia.org/r/485698

gerritbot added a comment.Jan 21 2019, 6:34 PM

Change 485701 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Demote db2040 from being an s7 master to just a replica

https://gerrit.wikimedia.org/r/485701

gerritbot added a comment.Jan 21 2019, 6:52 PM

Change 485701 merged by Jcrespo:
[operations/puppet@production] mariadb: Demote db2040 from being an s7 master to just a replica

https://gerrit.wikimedia.org/r/485701

Stashbot added a comment.Jan 21 2019, 6:55 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-21T18:55:56Z] <jynus> stop and upgrade db2040 T214264

Stashbot added a comment.Jan 21 2019, 7:23 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-21T19:23:23Z] <jynus> mysql.py -h db1115 zarcillo -e "UPDATE masters SET instance = 'db2047' WHERE section = 's7' and dc = 'codfw'" T214264

jcrespo added a comment.Jan 21 2019, 7:27 PM
root@cumin2001:~$ ./software/dbtools/section s7 | while read instance; do echo "$instance:"; mysql.py -h $instance -e "show slave status\G" | grep 'Using_Gtid:'; done    
labsdb1011:
labsdb1010:
labsdb1009:
dbstore2001:3317:
                   Using_Gtid: Slave_Pos
dbstore1002:
db2095:3317:
                   Using_Gtid: Slave_Pos
db2087:3317:
                   Using_Gtid: Slave_Pos
db2086:3317:
                   Using_Gtid: Slave_Pos
db2077:
                   Using_Gtid: Slave_Pos
db2068:
                   Using_Gtid: Slave_Pos
db2061:
                   Using_Gtid: Slave_Pos
db2054:
                   Using_Gtid: Slave_Pos
db2040:
                   Using_Gtid: Slave_Pos
db1125:3317:
                   Using_Gtid: Slave_Pos
db1116:3317:
                   Using_Gtid: Slave_Pos
db1101:3317:
                   Using_Gtid: Slave_Pos
db1098:3317:
                   Using_Gtid: Slave_Pos
db1094:
                   Using_Gtid: Slave_Pos
db1090:3317:
                   Using_Gtid: Slave_Pos
db1086:
                   Using_Gtid: Slave_Pos
db1079:
                   Using_Gtid: Slave_Pos
db2047:
                   Using_Gtid: Slave_Pos
db1062:

I will leave semi-sync "as is" (5 active clients, the non-ssd ones)

gerritbot added a comment.Jan 21 2019, 7:33 PM

Change 485698 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool db2040 after maintenance

https://gerrit.wikimedia.org/r/485698

jcrespo removed jcrespo as the assignee of this task.Jan 22 2019, 12:21 PM
jcrespo moved this task from In progress to Blocked external/Not db team on the DBA board.
Marostegui updated the task description. (Show Details)Feb 21 2019, 10:39 AM
Marostegui updated the task description. (Show Details)Feb 21 2019, 10:43 AM
Marostegui updated the task description. (Show Details)Apr 1 2019, 9:16 AM
Marostegui updated the task description. (Show Details)May 6 2019, 8:09 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)May 22 2019, 6:15 AM
Maintenance_bot removed a project: Patch-For-Review.May 22 2019, 3:39 PM
Marostegui mentioned this in T225090: Decommission db2042.Jun 5 2019, 1:00 PM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Jun 5 2019, 2:28 PM
Marostegui mentioned this in T220002: Decommission dbstore2001.codfw.wmnet and dbstore2002.codfw.wmnet.
Marostegui closed this task as Resolved.Jun 6 2019, 5:35 AM
Marostegui claimed this task.
Marostegui updated the task description. (Show Details)

All these hosts have been handed over to DCOPs for decommissioning. Closing this.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL