Page MenuHomePhabricator
Create Task
Maniphest T230705

Replace db2044 (m2 codfw master) with db2067
Closed, ResolvedPublic
Actions

Description

db2044 is currently m2 codfw master, this host has a broken disk and has had many disks failures in the past. It will be decommissioned.
Let's replace it with db2067.

Details

SubjectRepoBranchLines +/-
dbproxy2002: Promote db2067 to m2 codfw masteroperations/puppetproduction+2 -2
db2067: Enable notificationsoperations/puppetproduction+0 -1
install_server: Do not reimage db2067operations/puppetproduction+1 -1
mariadb: Move db2067 to m2, decomm db2063operations/puppetproduction+11 -13
db-eqiad,db-codfw.php: Remove db2067 from configoperations/mediawiki-configmaster+0 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Aug 19 2019, 5:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2019, 5:40 AM
Marostegui added a parent task: T208323: Predictive failures on disk S.M.A.R.T. status.Aug 19 2019, 5:40 AM
Stashbot added a comment.Aug 19 2019, 5:46 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-19T05:46:07Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2067, will be moved to m1 T230705', diff saved to https://phabricator.wikimedia.org/P8930 and previous config saved to /var/cache/conftool/dbconfig/20190819-054606-marostegui.json

gerritbot added a comment.Aug 19 2019, 5:46 AM

Change 530789 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2067 from config

https://gerrit.wikimedia.org/r/530789

gerritbot added a project: Patch-For-Review.Aug 19 2019, 5:46 AM
gerritbot added a comment.Aug 19 2019, 5:49 AM

Change 530789 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2067 from config

https://gerrit.wikimedia.org/r/530789

Stashbot added a comment.Aug 19 2019, 5:50 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-19T05:50:17Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2067 from config T230705 (duration: 00m 50s)

Stashbot added a comment.Aug 19 2019, 5:51 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-19T05:51:09Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2067 from config T230705 (duration: 00m 47s)

gerritbot added a comment.Aug 19 2019, 5:55 AM

Change 530790 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db2067 to m2, decomm db2063

https://gerrit.wikimedia.org/r/530790

gerritbot added a comment.Aug 19 2019, 7:09 AM

Change 530790 merged by Marostegui:
[operations/puppet@production] mariadb: Move db2067 to m2, decomm db2063

https://gerrit.wikimedia.org/r/530790

Maintenance_bot removed a project: Patch-For-Review.Aug 19 2019, 7:10 AM
ops-monitoring-bot added a comment.Aug 19 2019, 7:11 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2067.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908190711_marostegui_144027.log.

Marostegui triaged this task as Medium priority.Aug 19 2019, 7:11 AM
Marostegui moved this task from Triage to In progress on the DBA board.
ops-monitoring-bot added a comment.Aug 19 2019, 7:25 AM

Completed auto-reimage of hosts:

['db2067.codfw.wmnet']

Of which those FAILED:

['db2067.codfw.wmnet']
ops-monitoring-bot added a comment.Aug 19 2019, 8:07 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2067.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908190807_marostegui_154648.log.

ops-monitoring-bot added a comment.Aug 19 2019, 8:23 AM

Completed auto-reimage of hosts:

['db2067.codfw.wmnet']

Of which those FAILED:

['db2067.codfw.wmnet']
ops-monitoring-bot added a comment.Aug 19 2019, 9:07 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2067.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908190907_marostegui_167018.log.

ops-monitoring-bot added a comment.Aug 19 2019, 9:37 AM

Completed auto-reimage of hosts:

['db2067.codfw.wmnet']

and were ALL successful.

Marostegui added a comment.Aug 19 2019, 11:00 AM

db2067 is now replicating from db2044.
I am going to give it a few hours before promoting it to m2 codfw master.

gerritbot added a comment.Aug 19 2019, 11:42 AM

Change 530839 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Do not reimage db2067

https://gerrit.wikimedia.org/r/530839

gerritbot added a project: Patch-For-Review.Aug 19 2019, 11:42 AM

Change 530839 merged by Marostegui:
[operations/puppet@production] install_server: Do not reimage db2067

https://gerrit.wikimedia.org/r/530839

Maintenance_bot removed a project: Patch-For-Review.Aug 19 2019, 12:10 PM
gerritbot added a comment.Aug 19 2019, 3:29 PM

Change 530880 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2067: Enable notifications

https://gerrit.wikimedia.org/r/530880

gerritbot added a project: Patch-For-Review.Aug 19 2019, 3:29 PM
gerritbot added a comment.Aug 19 2019, 3:47 PM

Change 530880 merged by Marostegui:
[operations/puppet@production] db2067: Enable notifications

https://gerrit.wikimedia.org/r/530880

Maintenance_bot removed a project: Patch-For-Review.Aug 19 2019, 4:10 PM
gerritbot added a comment.Aug 20 2019, 5:17 AM

Change 531025 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy2002: Promote db2067 to m2 codfw master

https://gerrit.wikimedia.org/r/531025

gerritbot added a project: Patch-For-Review.Aug 20 2019, 5:17 AM
Stashbot added a comment.Aug 20 2019, 5:18 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-20T05:18:33Z] <marostegui> Switchover m2 codfw master, db2044 -> db2067 T230705

gerritbot added a comment.Aug 20 2019, 5:21 AM

Change 531025 merged by Marostegui:
[operations/puppet@production] dbproxy2002: Promote db2067 to m2 codfw master

https://gerrit.wikimedia.org/r/531025

Stashbot added a comment.Aug 20 2019, 5:24 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-20T05:24:50Z] <marostegui> Reload haproxy on dbproxy2002 T230705

Marostegui added a comment.Aug 20 2019, 5:25 AM

db2067 is the new m2 codfw master:

./replication_tree.py db2067.codfw.wmnet
db2067, version: 10.1.39, up: 18h, RO: ON, binlog: MIXED, lag: 0, processes: 16, latency: 0.0429
+ db2044, version: 10.1.39, up: 27d, RO: ON, binlog: MIXED, lag: 0, processes: 9, latency: 0.0411
+ db2078:3322, version: 10.1.39, up: 89d, RO: ON, binlog: MIXED, lag: 0, processes: 13, latency: 0.0428
Marostegui closed this task as Resolved.Aug 20 2019, 5:26 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 20 2019, 6:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits