Page MenuHomePhabricator
Create Task
Maniphest T350458

Decommission db11[26-49]
Closed, ResolvedPublic
Actions

Description

Following T344036

  • db1126
  • db1127
  • db1128
  • db1129
  • db1130
  • db1131
  • db1132
  • db1133
  • db1134
  • db1135
  • db1136
  • db1137
  • db1138
  • db1139
  • db1140
  • db1141
  • db1142
  • db1143
  • db1144
  • db1145
  • db1146
  • db1147
  • db1148
  • db1149

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
Restricted Task
 ResolvedArnoldokothT353651 VRTS Upgrade to 6.5.3
Unknown Object (Task)
 ResolvedJhancock.wmT342166 Q1:rack/setup/install db12[34-49]
Unknown Object (Task)
 ResolvedJclark-ctrT342176 Q1:rack/setup/install db12[26-33]
 ResolvedABran-WMFT344036 Productionize db12[26-49]
 ResolvedABran-WMFT350458 Decommission db11[26-49]
 ResolvedRequestJclark-ctrT351065 decommission db1136.eqiad.wmnet
 ResolvedRequestJclark-ctrT351063 decommission db1127.eqiad.wmnet
 ResolvedRequestJclark-ctrT351067 decommission db1130.eqiad.wmnet
 ResolvedRequestJclark-ctrT352362 decommission db1126.eqiad.wmnet
 Resolved• MarosteguiT355541 Install a temporary DB host in m2 to support VRTS migration
 ResolvedRequestVRiley-WMFT355740 decommission db1134.eqiad.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Jclark-ctr closed subtask T351063: decommission db1127.eqiad.wmnet as Resolved.Nov 27 2023, 3:56 PM
• Marostegui updated the task description. (Show Details)Nov 30 2023, 1:14 PM
• Marostegui added a subtask: T352362: decommission db1126.eqiad.wmnet.
Jclark-ctr closed subtask T352362: decommission db1126.eqiad.wmnet as Resolved.Dec 8 2023, 3:01 PM
gerritbot added a comment.Dec 11 2023, 1:14 PM

Change 981440 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: decommission db1138

https://gerrit.wikimedia.org/r/981440

gerritbot added a project: Patch-For-Review.Dec 11 2023, 1:14 PM
ops-monitoring-bot added a comment.Dec 11 2023, 1:17 PM

Icinga downtime and Alertmanager silence (ID=bffacd05-8841-49a1-9e9d-e042eb304501) set by arnaudb@cumin1001 for 1 day, 0:00:00 on 1 host(s) and their services with reason: decomission pre downtime

db1138.eqiad.wmnet
gerritbot added a comment.Dec 11 2023, 1:20 PM

Change 981440 merged by Arnaudb:

[operations/puppet@production] mariadb: decommission db1138

https://gerrit.wikimedia.org/r/981440

Maintenance_bot removed a project: Patch-For-Review.Dec 11 2023, 1:30 PM
ABran-WMF updated the task description. (Show Details)Dec 11 2023, 2:24 PM
gerritbot added a comment.Dec 11 2023, 2:27 PM

Change 981441 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: remove db1141 db1142 db1143

https://gerrit.wikimedia.org/r/981441

gerritbot added a project: Patch-For-Review.Dec 11 2023, 2:27 PM
gerritbot added a comment.Dec 11 2023, 2:48 PM

Change 981441 merged by Arnaudb:

[operations/puppet@production] mariadb: remove db1141 db1142 db1143

https://gerrit.wikimedia.org/r/981441

Maintenance_bot removed a project: Patch-For-Review.Dec 11 2023, 3:10 PM
ABran-WMF updated the task description. (Show Details)Dec 11 2023, 3:35 PM
ABran-WMF updated the task description. (Show Details)Dec 13 2023, 2:33 PM
ABran-WMF updated the task description. (Show Details)Dec 13 2023, 2:37 PM
gerritbot added a comment.Dec 13 2023, 2:40 PM

Change 982198 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: decommission db1128 db1129 db1147

https://gerrit.wikimedia.org/r/982198

gerritbot added a project: Patch-For-Review.Dec 13 2023, 2:40 PM
gerritbot added a comment.Dec 13 2023, 3:02 PM

Change 982198 merged by Arnaudb:

[operations/puppet@production] mariadb: decommission db1128 db1129 db1147

https://gerrit.wikimedia.org/r/982198

Maintenance_bot removed a project: Patch-For-Review.Dec 13 2023, 3:11 PM
ABran-WMF updated the task description. (Show Details)Dec 13 2023, 3:15 PM
ABran-WMF updated the task description. (Show Details)Dec 13 2023, 3:26 PM
ABran-WMF updated the task description. (Show Details)Dec 13 2023, 3:37 PM
ABran-WMF updated the task description. (Show Details)Dec 14 2023, 1:29 PM
ABran-WMF updated the task description. (Show Details)Dec 14 2023, 1:33 PM
gerritbot added a comment.Dec 14 2023, 1:37 PM

Change 982875 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: decommission hosts

https://gerrit.wikimedia.org/r/982875

gerritbot added a project: Patch-For-Review.Dec 14 2023, 1:37 PM
gerritbot added a comment.Dec 14 2023, 1:39 PM

Change 982875 merged by Arnaudb:

[operations/puppet@production] mariadb: decommission hosts

https://gerrit.wikimedia.org/r/982875

ABran-WMF updated the task description. (Show Details)Dec 14 2023, 1:55 PM
ABran-WMF updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Dec 14 2023, 2:10 PM
ABran-WMF updated the task description. (Show Details)Dec 14 2023, 4:20 PM
ABran-WMF changed the task status from Open to In Progress.Dec 20 2023, 3:43 PM
• Marostegui added a subtask: T355541: Install a temporary DB host in m2 to support VRTS migration.Jan 22 2024, 2:07 PM
• Marostegui closed subtask T355541: Install a temporary DB host in m2 to support VRTS migration as Resolved.Jan 23 2024, 6:01 AM
• Marostegui updated the task description. (Show Details)
• Marostegui added a subtask: T355740: decommission db1134.eqiad.wmnet.Jan 24 2024, 7:22 AM
• Marostegui changed the status of subtask T355740: decommission db1134.eqiad.wmnet from Stalled to Open.Jan 29 2024, 5:45 AM
• Marostegui updated the task description. (Show Details)Jan 29 2024, 5:49 AM
• Marostegui updated the task description. (Show Details)
VRiley-WMF closed subtask T355740: decommission db1134.eqiad.wmnet as Resolved.Jan 29 2024, 5:01 PM
Stashbot added a comment.Feb 1 2024, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T13:29:06Z] <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: provisionning db1234.eqiad.wmnet - T350458

ops-monitoring-bot added a comment.Feb 1 2024, 1:29 PM

Icinga downtime and Alertmanager silence (ID=cc740753-fd98-4755-ad37-dfd92af53af6) set by arnaudb@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: provisionning db1234.eqiad.wmnet - T350458

db1144.eqiad.wmnet
Stashbot added a comment.Feb 1 2024, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T13:29:40Z] <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: provisionning db1234.eqiad.wmnet - T350458

ops-monitoring-bot added a comment.Feb 1 2024, 1:29 PM

Icinga downtime and Alertmanager silence (ID=7b097df6-7429-4e29-9f7c-164dd2eb37b4) set by arnaudb@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: provisionning db1234.eqiad.wmnet - T350458

db1244.eqiad.wmnet
Stashbot added a comment.Feb 1 2024, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T13:29:53Z] <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: provisionning db1234.eqiad.wmnet - T350458

Stashbot added a comment.Feb 1 2024, 1:30 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T13:30:25Z] <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: provisionning db1234.eqiad.wmnet - T350458

Stashbot added a comment.Feb 1 2024, 1:38 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T13:38:44Z] <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: provisionning db1244.eqiad.wmnet - T350458

ops-monitoring-bot added a comment.Feb 1 2024, 1:38 PM

Icinga downtime and Alertmanager silence (ID=da094b6a-f059-477c-957d-fee829374a3f) set by arnaudb@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: provisionning db1244.eqiad.wmnet - T350458

db1144.eqiad.wmnet
ops-monitoring-bot added a comment.Feb 1 2024, 1:38 PM

Icinga downtime and Alertmanager silence (ID=44ed7cb6-a06a-48a1-8efe-519851e204a7) set by arnaudb@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: provisionning db1244.eqiad.wmnet - T350458

db1244.eqiad.wmnet
Stashbot added a comment.Feb 1 2024, 1:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T13:39:00Z] <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: provisionning db1244.eqiad.wmnet - T350458

Stashbot added a comment.Feb 1 2024, 1:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T13:39:14Z] <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: provisionning db1244.eqiad.wmnet - T350458

Stashbot added a comment.Feb 1 2024, 1:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T13:39:28Z] <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: provisionning db1244.eqiad.wmnet - T350458

Stashbot added a comment.Feb 1 2024, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T13:41:08Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Cloning db1144 in db1244 for T350458', diff saved to https://phabricator.wikimedia.org/P56064 and previous config saved to /var/cache/conftool/dbconfig/20240201-134107-arnaudb.json

Stashbot added a comment.Feb 1 2024, 2:12 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T14:12:49Z] <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: provisionning db1246.eqiad.wmnet - T350458

ops-monitoring-bot added a comment.Feb 1 2024, 2:13 PM

Icinga downtime and Alertmanager silence (ID=fe9402f0-5956-4560-bb9a-16fb066dd913) set by arnaudb@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: provisionning db1246.eqiad.wmnet - T350458

db1146.eqiad.wmnet
Stashbot added a comment.Feb 1 2024, 2:13 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T14:13:06Z] <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: provisionning db1246.eqiad.wmnet - T350458

Stashbot added a comment.Feb 1 2024, 2:13 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T14:13:21Z] <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1246.eqiad.wmnet with reason: provisionning db1246.eqiad.wmnet - T350458

ops-monitoring-bot added a comment.Feb 1 2024, 2:13 PM

Icinga downtime and Alertmanager silence (ID=539c6327-d7ed-40e3-9363-fa3f56dcbac9) set by arnaudb@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: provisionning db1246.eqiad.wmnet - T350458

db1246.eqiad.wmnet
Stashbot added a comment.Feb 1 2024, 2:13 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T14:13:33Z] <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1246.eqiad.wmnet with reason: provisionning db1246.eqiad.wmnet - T350458

Stashbot added a comment.Feb 1 2024, 2:15 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-01T14:15:32Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Cloning db1146 in db1246 for T350458', diff saved to https://phabricator.wikimedia.org/P56067 and previous config saved to /var/cache/conftool/dbconfig/20240201-141531-arnaudb.json

ABran-WMF updated the task description. (Show Details)Feb 5 2024, 5:11 PM
jcrespo subscribed.Feb 7 2024, 4:52 PM

Technically no blocker for decom of db1139, db1140 or db1145, although I would wait 1 day to make sure backups run correctly on the new hosts at least once.

ABran-WMF updated the task description. (Show Details)Feb 8 2024, 3:01 PM
ABran-WMF added a comment.Feb 12 2024, 8:59 AM

@jcrespo everything is going OK with all the new servers? if so, I'll close this task!

jcrespo added a comment.Feb 12 2024, 10:01 AM

All backups looking good. Please proceed.

ABran-WMF awarded a token.Feb 12 2024, 10:32 AM
gerritbot added a comment.Feb 12 2024, 10:37 AM

Change 1000300 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: decom db1133

https://gerrit.wikimedia.org/r/1000300

gerritbot added a project: Patch-For-Review.Feb 12 2024, 10:37 AM
gerritbot added a comment.Feb 12 2024, 10:44 AM

Change 1000301 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: removes db1135

https://gerrit.wikimedia.org/r/1000301

gerritbot added a comment.Feb 12 2024, 11:01 AM

Change 1000302 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: removes db1139

https://gerrit.wikimedia.org/r/1000302

gerritbot added a comment.Feb 12 2024, 11:03 AM

Change 1000303 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: removes db1140

https://gerrit.wikimedia.org/r/1000303

gerritbot added a comment.Feb 12 2024, 11:05 AM

Change 1000304 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: removes db1144

https://gerrit.wikimedia.org/r/1000304

gerritbot added a comment.Feb 12 2024, 11:08 AM

Change 1000305 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: removes db1145

https://gerrit.wikimedia.org/r/1000305

gerritbot added a comment.Feb 12 2024, 11:10 AM

Change 1002406 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: removes db1146

https://gerrit.wikimedia.org/r/1002406

gerritbot added a comment.Feb 12 2024, 11:12 AM

Change 1002407 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: removes db1149

https://gerrit.wikimedia.org/r/1002407

ABran-WMF updated the task description. (Show Details)Feb 12 2024, 11:16 AM
gerritbot added a comment.Feb 12 2024, 3:59 PM

Change 1002406 merged by Arnaudb:

[operations/puppet@production] mariadb: removes db1146

https://gerrit.wikimedia.org/r/1002406

gerritbot added a comment.Feb 12 2024, 3:59 PM

Change 1000303 merged by Arnaudb:

[operations/puppet@production] mariadb: removes db1140

https://gerrit.wikimedia.org/r/1000303

gerritbot added a comment.Feb 12 2024, 4:01 PM

Change 1000305 merged by Arnaudb:

[operations/puppet@production] mariadb: removes db1145

https://gerrit.wikimedia.org/r/1000305

gerritbot added a comment.Feb 12 2024, 4:01 PM

Change 1000304 abandoned by Arnaudb:

[operations/puppet@production] mariadb: removes db1144

Reason:

https://gerrit.wikimedia.org/r/1000304

gerritbot added a comment.Feb 12 2024, 4:02 PM

Change 1000301 merged by Arnaudb:

[operations/puppet@production] mariadb: removes db1135

https://gerrit.wikimedia.org/r/1000301

gerritbot added a comment.Feb 12 2024, 4:03 PM

Change 1002407 abandoned by Arnaudb:

[operations/puppet@production] mariadb: removes db1149

Reason:

https://gerrit.wikimedia.org/r/1002407

gerritbot added a comment.Feb 12 2024, 4:06 PM

Change 1002411 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] mariadb: removes db1144 db1149

https://gerrit.wikimedia.org/r/1002411

gerritbot added a comment.Feb 12 2024, 4:07 PM

Change 1002411 merged by Arnaudb:

[operations/puppet@production] mariadb: removes db1144 db1149

https://gerrit.wikimedia.org/r/1002411

gerritbot added a comment.Feb 12 2024, 4:12 PM

Change 1000302 merged by Arnaudb:

[operations/puppet@production] mariadb: removes db1139

https://gerrit.wikimedia.org/r/1000302

gerritbot added a comment.Feb 12 2024, 4:14 PM

Change 1000300 merged by Arnaudb:

[operations/puppet@production] mariadb: decom db1133

https://gerrit.wikimedia.org/r/1000300

Maintenance_bot removed a project: Patch-For-Review.Feb 12 2024, 4:31 PM
Stashbot added a comment.Feb 12 2024, 4:34 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-12T16:34:08Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Removing instances as per T350458', diff saved to https://phabricator.wikimedia.org/P56683 and previous config saved to /var/cache/conftool/dbconfig/20240212-163407-arnaudb.json

Stashbot added a comment.Feb 13 2024, 11:45 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-13T11:45:17Z] <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1133.eqiad.wmnet with reason: T350458

Stashbot added a comment.Feb 13 2024, 11:45 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-13T11:45:31Z] <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1133.eqiad.wmnet with reason: T350458

ABran-WMF closed this task as Resolved.Feb 13 2024, 4:46 PM
ABran-WMF updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits