Page MenuHomePhabricator
Create Task
Maniphest T217396

Decommission db1061-db1073
Closed, ResolvedPublic
Actions

Description

The following hosts need to be handed over to DCOPs to get decommissioned and refreshed by the ones at T222682: Productionize db11[26-38]

Details

SubjectRepoBranchLines +/-
db1071: Disable notificationsoperations/puppetproduction+1 -0
db-eqiad,db-codfw.php: Remove db1068operations/mediawiki-configmaster+0 -3
db1068: Disable notificationsoperations/puppetproduction+1 -0
db-eqiad.php: Depool db1064operations/mediawiki-configmaster+4 -4
db-eqiad,db-codfw.php: Pool db1127 and db1137 in x1operations/mediawiki-configmaster+7 -1
mariadb: Provision db1127 and db1137 on x1operations/puppetproduction+20 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedMarosteguiT217396 Decommission db1061-db1073
 ResolvedMarosteguiT211613 rack/setup/install db11[26-38].eqiad.wmnet
 ResolvedMarosteguiT222682 Productionize db11[26-38]
 Resolved• CmjohnsonT222731 Storage problems with new host db1133
 ResolvedjcrespoT213664 correctable memory errors db1068 (commons primary master database)
 ResolvedJclark-ctrT223217 Decommission db1064
 ResolvedMarosteguiT224852 Failover s4 primary master: db1068 to db1081
 ResolvedJohanT224516 Database primary master failover on s4 (commonswiki)
 ResolvedMarosteguiT226358 Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC
 ResolvedJclark-ctrT226689 decommission db1068
 ResolvedMarosteguiT226952 Failover m2 master db1065 to db1132
 ResolvedMarosteguiT227062 Failover s8 (wikidatawiki) db primary master db1071 to db1104 (read-only required)
 ResolvedJohanT227063 Database primary master failover on s8 (wikidatawiki)
 ResolvedRequestJclark-ctrT227166 decommission db1069
 ResolvedMarosteguiT228243 Switchover m3 (phabricator) master db1072 to db1128
 Resolved• mmodellT228955 test
 ResolvedJclark-ctrT228956 decommission db1072.eqiad.wmnet
 ResolvedJclark-ctrT229381 decommission db1071.eqiad.wmnet
 ResolvedMarosteguiT230785 Switchover s2 primary database master db1066 -> db1122 - 17th Sept @05:00 UTC
 ResolvedTrizek-WMFT230788 Community Relations support needed for several read-only windows (s2, s3, s4 and s8)
 ResolvedMarosteguiT231403 Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC
 ResolvedJclark-ctrT231892 Decommission db1073.eqiad.wmnet
 ResolvedMarosteguiT229657 Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC
 ResolvedJclark-ctrT232564 Decommission db1063.eqiad.wmnet
 ResolvedJclark-ctrT233071 Decommission db1066.eqiad.wmnet
 ResolvedMarosteguiT234300 Switchover s5 primary database master db1070 -> db1100 - 15th Oct 05:00 - 05:30 UTC
 ResolvedTrizek-WMFT234303 Community Relations support needed a several read-only window for s5
 ResolvedMarosteguiT234800 Switchover s1 primary database master db1067 -> db1083 - 14th Nov 06:00 - 06:30 UTC
 ResolvedTrizek-WMFT234801 Community Relations support needed for a read-only window for s1 (enwiki)
 ResolvedJclark-ctrT235464 decommission db1070.eqiad.wmnet
 ResolvedMarosteguiT235469 Switchover s6 primary database master db1061 -> db1131 - 19th Nov 06:00 - 06:30 UTC
 ResolvedTrizek-WMFT235475 Community Relations support needed for a read-only window for s6 (frwiki, jawiki, ruwiki)
 ResolvedMarosteguiT238044 Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC
 ResolvedTrizek-WMFT238046 Community Relations support needed for a read-only window for s7 (several wikis)
 ResolvedJclark-ctrT238624 Decommission db1061.eqiad.wmnet
 ResolvedJclark-ctrT239188 Decommission db1062.eqiad.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Marostegui updated the task description. (Show Details)May 23 2019, 2:13 PM
Marostegui mentioned this in T224516: Database primary master failover on s4 (commonswiki).May 28 2019, 7:19 PM
Marostegui added a subtask: T224516: Database primary master failover on s4 (commonswiki).
Stashbot added a comment.Jun 3 2019, 5:17 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-03T05:17:07Z] <marostegui> Upgrade MariaDB on codfw hosts in preparation for s4 master failover T217396

Marostegui mentioned this in T224852: Failover s4 primary master: db1068 to db1081.Jun 3 2019, 5:40 AM
jcrespo removed a subtask: T224516: Database primary master failover on s4 (commonswiki).Jun 3 2019, 7:32 AM
Marostegui updated the task description. (Show Details)Jun 3 2019, 8:04 AM
Marostegui changed the status of subtask T211613: rack/setup/install db11[26-38].eqiad.wmnet from Open to Stalled.Jun 4 2019, 5:10 PM
Marostegui closed subtask T224852: Failover s4 primary master: db1068 to db1081 as Resolved.Jun 19 2019, 5:23 AM
gerritbot added a comment.Jun 19 2019, 5:27 AM

Change 517792 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1068: Disable notifications

https://gerrit.wikimedia.org/r/517792

gerritbot added a project: Patch-For-Review.Jun 19 2019, 5:27 AM
gerritbot added a comment.Jun 19 2019, 5:29 AM

Change 517792 merged by Marostegui:
[operations/puppet@production] db1068: Disable notifications

https://gerrit.wikimedia.org/r/517792

Maintenance_bot removed a project: Patch-For-Review.Jun 19 2019, 6:10 AM
Marostegui mentioned this in T226358: Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC.Jun 24 2019, 7:59 AM
Marostegui changed the status of subtask T211613: rack/setup/install db11[26-38].eqiad.wmnet from Stalled to Open.Jun 25 2019, 7:38 AM
Marostegui closed subtask T211613: rack/setup/install db11[26-38].eqiad.wmnet as Resolved.
gerritbot added a comment.Jun 26 2019, 9:19 AM

Change 519200 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1068

https://gerrit.wikimedia.org/r/519200

gerritbot added a project: Patch-For-Review.Jun 26 2019, 9:19 AM

Change 519200 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1068

https://gerrit.wikimedia.org/r/519200

Stashbot added a comment.Jun 26 2019, 9:22 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-26T09:22:46Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db1068 from config T217396 (duration: 01m 11s)

Stashbot added a comment.Jun 26 2019, 9:23 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-26T09:23:47Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db1068 from config T217396 (duration: 00m 55s)

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2019, 10:10 AM
Marostegui updated the task description. (Show Details)Jun 27 2019, 5:26 AM
Marostegui updated the task description. (Show Details)Jun 27 2019, 7:29 AM
Marostegui mentioned this in T226952: Failover m2 master db1065 to db1132.Jul 1 2019, 6:16 AM
Marostegui added a subtask: T227062: Failover s8 (wikidatawiki) db primary master db1071 to db1104 (read-only required).Jul 2 2019, 8:20 AM
Marostegui added a subtask: T227166: decommission db1069.Jul 3 2019, 6:28 AM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T226358: Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC as Resolved.Jul 3 2019, 12:51 PM
Marostegui updated the task description. (Show Details)Jul 5 2019, 5:11 AM
Marostegui closed subtask T226952: Failover m2 master db1065 to db1132 as Resolved.Jul 9 2019, 7:30 AM
Marostegui updated the task description. (Show Details)Jul 9 2019, 8:46 AM
Marostegui updated the task description. (Show Details)Jul 17 2019, 5:28 AM
Marostegui closed subtask T228243: Switchover m3 (phabricator) master db1072 to db1128 as Resolved.Jul 25 2019, 5:41 AM
Marostegui updated the task description. (Show Details)Jul 29 2019, 5:15 AM
gerritbot added a comment.Jul 30 2019, 5:22 AM

Change 526307 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1071: Disable notifications

https://gerrit.wikimedia.org/r/526307

gerritbot added a project: Patch-For-Review.Jul 30 2019, 5:22 AM
gerritbot added a comment.Jul 30 2019, 5:24 AM

Change 526307 merged by Marostegui:
[operations/puppet@production] db1071: Disable notifications

https://gerrit.wikimedia.org/r/526307

Marostegui closed subtask T227062: Failover s8 (wikidatawiki) db primary master db1071 to db1104 (read-only required) as Resolved.Jul 30 2019, 5:48 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 30 2019, 6:10 AM
Marostegui claimed this task.Jul 30 2019, 8:06 AM
Marostegui moved this task from Pending comment to In progress on the DBA board.
Marostegui updated the task description. (Show Details)Jul 31 2019, 5:48 AM
Marostegui mentioned this in T229657: Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC.Aug 2 2019, 11:54 AM
Marostegui updated the task description. (Show Details)Aug 7 2019, 6:00 AM
Marostegui mentioned this in T230785: Switchover s2 primary database master db1066 -> db1122 - 17th Sept @05:00 UTC.Aug 20 2019, 10:21 AM
Marostegui added a subtask: T230785: Switchover s2 primary database master db1066 -> db1122 - 17th Sept @05:00 UTC.
Marostegui closed subtask T229657: Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC as Resolved.Sep 3 2019, 2:45 PM
Marostegui updated the task description. (Show Details)Sep 4 2019, 5:07 AM
Marostegui closed subtask T231403: Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC as Resolved.Sep 10 2019, 4:22 PM
Marostegui updated the task description. (Show Details)Sep 11 2019, 5:13 AM
Marostegui updated the task description. (Show Details)Sep 12 2019, 6:02 AM
Marostegui closed subtask T230785: Switchover s2 primary database master db1066 -> db1122 - 17th Sept @05:00 UTC as Resolved.Sep 17 2019, 5:16 AM
Marostegui updated the task description. (Show Details)Sep 17 2019, 6:36 AM
Marostegui updated the task description. (Show Details)Sep 17 2019, 7:48 AM
Marostegui mentioned this in T234303: Community Relations support needed a several read-only window for s5.Oct 1 2019, 4:50 AM
Marostegui updated the task description. (Show Details)Oct 7 2019, 9:52 AM
Marostegui closed subtask T234300: Switchover s5 primary database master db1070 -> db1100 - 15th Oct 05:00 - 05:30 UTC as Resolved.Oct 15 2019, 5:22 AM
Marostegui added a subtask: T235464: decommission db1070.eqiad.wmnet.
Marostegui updated the task description. (Show Details)Oct 15 2019, 5:24 AM
Marostegui added a subtask: T235469: Switchover s6 primary database master db1061 -> db1131 - 19th Nov 06:00 - 06:30 UTC.Oct 15 2019, 6:33 AM
Marostegui updated the task description. (Show Details)Oct 24 2019, 4:54 AM
Papaul closed subtask T232564: Decommission db1063.eqiad.wmnet as Resolved.Nov 5 2019, 10:02 PM
Papaul closed subtask T223217: Decommission db1064 as Resolved.
Papaul closed subtask T226689: decommission db1068 as Resolved.Nov 5 2019, 10:06 PM
Papaul closed subtask T235464: decommission db1070.eqiad.wmnet as Resolved.
Papaul closed subtask T229381: decommission db1071.eqiad.wmnet as Resolved.Nov 5 2019, 10:08 PM
Marostegui added a subtask: T238044: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC.Nov 12 2019, 9:25 AM
Marostegui closed subtask T234800: Switchover s1 primary database master db1067 -> db1083 - 14th Nov 06:00 - 06:30 UTC as Resolved.Nov 14 2019, 6:40 AM
Marostegui updated the task description. (Show Details)Nov 14 2019, 7:31 AM
Marostegui closed subtask T235469: Switchover s6 primary database master db1061 -> db1131 - 19th Nov 06:00 - 06:30 UTC as Resolved.Nov 19 2019, 6:12 AM
Marostegui updated the task description. (Show Details)Nov 19 2019, 6:20 AM
Marostegui updated the task description. (Show Details)Nov 25 2019, 8:13 AM
Marostegui closed subtask T238044: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC as Resolved.Nov 26 2019, 6:29 AM
Marostegui updated the task description. (Show Details)Nov 26 2019, 6:53 AM
Marostegui updated the task description. (Show Details)Nov 28 2019, 6:21 AM
Papaul closed subtask T227166: decommission db1069 as Resolved.Dec 6 2019, 4:53 PM
Papaul closed subtask T228956: decommission db1072.eqiad.wmnet as Resolved.
Papaul closed subtask T231892: Decommission db1073.eqiad.wmnet as Resolved.
Agusbou2015 subscribed.Dec 10 2019, 8:06 PM
Marostegui closed this task as Resolved.Dec 11 2019, 6:12 AM
Marostegui updated the task description. (Show Details)

All these hosts have been decommissioned or are waiting for the last on-site steps.

Papaul closed subtask T238624: Decommission db1061.eqiad.wmnet as Resolved.Mar 20 2020, 3:26 PM
Papaul closed subtask T239188: Decommission db1062.eqiad.wmnet as Resolved.
Papaul closed subtask T233071: Decommission db1066.eqiad.wmnet as Resolved.Mar 20 2020, 3:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL