Page MenuHomePhabricator
Create Task
Maniphest T418979

Productionize db225[0-3]
Closed, ResolvedPublic
Actions

Description

These hosts will replace hosts in codfw

  • db2250 will replace db2141
  • db2251 will replace db2142 ms1
  • db2252 will replace db2143 ms3
  • db2253 will replace db2144 ms2

The following hosts WILL NOT BE refreshed and will be later decommissioned
Decommission tracking task: T424028: Decommission db2141-db2152

db2145 s1
db2146 s1
db2147 s4
db2148 s2
db2149 s3
db2150 s7
db2151 s6
db2152 s8

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2026, 8:38 AM
Marostegui triaged this task as Medium priority.Mar 4 2026, 8:38 AM
Marostegui moved this task from Triage to Blocked on the DBA board.
Marostegui updated the task description. (Show Details)Mar 4 2026, 8:47 AM
Marostegui renamed this task from Productionize db225[0-4] to Productionize db225[0-3].Mar 4 2026, 9:18 AM
gerritbot added a comment.Mar 4 2026, 9:19 AM

Change #1247934 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Add db225[0-3]

https://gerrit.wikimedia.org/r/1247934

gerritbot added a project: Patch-For-Review.Mar 4 2026, 9:19 AM
Marostegui added a subscriber: • jcrespo.Apr 21 2026, 1:45 PM

@jcrespo db2250 is for you to replace db2141

Marostegui created subtask T424028: Decommission db2141-db2152.Apr 21 2026, 1:47 PM
Marostegui mentioned this in T424028: Decommission db2141-db2152.
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Apr 21 2026, 1:54 PM

Change #1275930 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db2251

https://gerrit.wikimedia.org/r/1275930

gerritbot added a comment.Apr 21 2026, 1:56 PM

Change #1275930 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db2251

https://gerrit.wikimedia.org/r/1275930

gerritbot added a comment.Apr 21 2026, 2:21 PM

Change #1275945 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2142: Disable notifications

https://gerrit.wikimedia.org/r/1275945

gerritbot added a comment.Apr 21 2026, 2:24 PM

Change #1275945 merged by Marostegui:

[operations/puppet@production] db2142: Disable notifications

https://gerrit.wikimedia.org/r/1275945

Marostegui added a subtask: T424038: decommission db2142.codfw.wmnet.Apr 21 2026, 2:24 PM
gerritbot added a comment.Apr 21 2026, 2:26 PM

Change #1275946 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Remove db2142, add db2251

https://gerrit.wikimedia.org/r/1275946

gerritbot added a comment.Apr 21 2026, 2:27 PM

Change #1275946 merged by Marostegui:

[operations/puppet@production] instances.yaml: Remove db2142, add db2251

https://gerrit.wikimedia.org/r/1275946

Stashbot added a comment.Apr 21 2026, 2:29 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-21T14:29:14Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Add db2251, remove db2142 T418979', diff saved to https://phabricator.wikimedia.org/P91298 and previous config saved to /var/cache/conftool/dbconfig/20260421-142913-marostegui.json

Stashbot added a comment.Apr 21 2026, 2:30 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-21T14:30:17Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Add db2251 to ms1 T418979', diff saved to https://phabricator.wikimedia.org/P91300 and previous config saved to /var/cache/conftool/dbconfig/20260421-143017-marostegui.json

Stashbot added a comment.Apr 21 2026, 2:31 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-21T14:31:46Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Repool ms1 T418979', diff saved to https://phabricator.wikimedia.org/P91301 and previous config saved to /var/cache/conftool/dbconfig/20260421-143145-marostegui.json

gerritbot added a comment.Apr 21 2026, 2:32 PM

Change #1275947 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2251: Enable notifications

https://gerrit.wikimedia.org/r/1275947

Marostegui updated the task description. (Show Details)Apr 21 2026, 2:32 PM
gerritbot added a comment.Apr 21 2026, 2:33 PM

Change #1275947 merged by Marostegui:

[operations/puppet@production] db2251: Enable notifications

https://gerrit.wikimedia.org/r/1275947

• jcrespo added a comment.Apr 21 2026, 3:27 PM
In T418979#11844326, @Marostegui wrote:

@jcrespo db2250 is for you to replace db2141

Understood, will take care.

ops-monitoring-bot added a comment.Apr 22 2026, 10:27 AM

Icinga downtime and Alertmanager silence (ID=7e88a25f-0194-4ea2-910c-02ec207549cd) set by jynus@cumin1003 for 4:00:00 on 2 host(s) and their services with reason: clone

db[2141,2250].codfw.wmnet
gerritbot added a comment.Apr 22 2026, 10:42 AM

Change #1276382 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Set db2250 as a new codfw s1 backup source

https://gerrit.wikimedia.org/r/1276382

ops-monitoring-bot added a comment.Apr 22 2026, 3:40 PM

Icinga downtime and Alertmanager silence (ID=4f65eefe-2ac7-4c5f-a593-a3b32487e088) set by jynus@cumin1003 for 4:00:00 on 2 host(s) and their services with reason: clone

db[2141,2250].codfw.wmnet
gerritbot added a comment.Apr 22 2026, 3:55 PM

Change #1276406 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Pool db2250 for backups instead of db2141

https://gerrit.wikimedia.org/r/1276406

gerritbot added a comment.Apr 22 2026, 3:55 PM

Change #1276407 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Set db2141 as a spare for decommissioning

https://gerrit.wikimedia.org/r/1276407

gerritbot added a comment.Apr 23 2026, 5:40 AM

Change #1276478 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db2252

https://gerrit.wikimedia.org/r/1276478

gerritbot added a comment.Apr 23 2026, 5:42 AM

Change #1276478 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db2252

https://gerrit.wikimedia.org/r/1276478

Marostegui updated the task description. (Show Details)Apr 23 2026, 5:45 AM
Marostegui added a subtask: T424171: decommission db2143.codfw.wmnet.Apr 23 2026, 6:24 AM
gerritbot added a comment.Apr 23 2026, 6:28 AM

Change #1276509 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2252: Enable notifications

https://gerrit.wikimedia.org/r/1276509

gerritbot added a comment.Apr 23 2026, 6:29 AM

Change #1276510 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Replace db2143 with db2252

https://gerrit.wikimedia.org/r/1276510

gerritbot added a comment.Apr 23 2026, 6:48 AM

Change #1276509 merged by Marostegui:

[operations/puppet@production] db2252: Enable notifications

https://gerrit.wikimedia.org/r/1276509

gerritbot added a comment.Apr 23 2026, 6:49 AM

Change #1276510 merged by Marostegui:

[operations/puppet@production] instances.yaml: Replace db2143 with db2252

https://gerrit.wikimedia.org/r/1276510

Stashbot added a comment.Apr 23 2026, 6:52 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-23T06:52:15Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Remove db2143 from ms3, add db2252 T418979', diff saved to https://phabricator.wikimedia.org/P91326 and previous config saved to /var/cache/conftool/dbconfig/20260423-065214-marostegui.json

Stashbot added a comment.Apr 23 2026, 6:53 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-23T06:53:23Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Make db2252 master of ms3 T418979', diff saved to https://phabricator.wikimedia.org/P91327 and previous config saved to /var/cache/conftool/dbconfig/20260423-065323-marostegui.json

Stashbot added a comment.Apr 23 2026, 6:58 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-23T06:58:04Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Repool ms1 with db2252 as new codfw master T418979', diff saved to https://phabricator.wikimedia.org/P91328 and previous config saved to /var/cache/conftool/dbconfig/20260423-065803-marostegui.json

Marostegui updated the task description. (Show Details)Apr 23 2026, 6:58 AM
Marostegui added a subtask: T424177: decommission db2145.codfw.wmnet.Apr 23 2026, 7:10 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Apr 23 2026, 7:15 AM

Change #1276382 merged by Jcrespo:

[operations/puppet@production] mariadb: Set db2250 as a new codfw s1 backup source

https://gerrit.wikimedia.org/r/1276382

Marostegui added a subtask: T424189: decommission db2146.codfw.wmnet.Apr 23 2026, 8:52 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Apr 23 2026, 8:53 AM

Change #1276612 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Remove db2146

https://gerrit.wikimedia.org/r/1276612

gerritbot added a comment.Apr 23 2026, 8:55 AM

Change #1276612 merged by Marostegui:

[operations/puppet@production] instances.yaml: Remove db2146

https://gerrit.wikimedia.org/r/1276612

gerritbot added a comment.Apr 23 2026, 8:56 AM

Change #1276406 merged by Jcrespo:

[operations/puppet@production] mariadb: Pool db2250 for backups instead of db2141

https://gerrit.wikimedia.org/r/1276406

Marostegui added a subtask: T424226: decommission db2147.codfw.wmnet.Apr 23 2026, 1:50 PM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Apr 23 2026, 3:11 PM

Change #1276719 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Reenable notifications for db2250

https://gerrit.wikimedia.org/r/1276719

gerritbot added a comment.Apr 23 2026, 3:15 PM

Change #1276720 had a related patch set uploaded (by Jcrespo; author: Marostegui):

[operations/puppet@production] installserver: Do not format db2250

https://gerrit.wikimedia.org/r/1276720

gerritbot added a comment.Apr 23 2026, 3:25 PM

Change #1276720 merged by Marostegui:

[operations/puppet@production] installserver: Do not format db2250

https://gerrit.wikimedia.org/r/1276720

gerritbot added a comment.Apr 23 2026, 3:25 PM

Change #1276719 merged by Jcrespo:

[operations/puppet@production] mariadb: Reenable notifications for db2250

https://gerrit.wikimedia.org/r/1276719

Marostegui added a subtask: T424309: decommission db2148.codfw.wmnet.Apr 24 2026, 5:33 AM
Marostegui updated the task description. (Show Details)Apr 24 2026, 5:39 AM
• jcrespo updated the task description. (Show Details)Apr 24 2026, 11:01 AM
• jcrespo added a subtask: T424327: decommission db2141.codfw.wmnet.Apr 24 2026, 11:05 AM
Marostegui moved this task from Blocked to In progress on the DBA board.Apr 24 2026, 1:53 PM
Jhancock.wm closed subtask T424309: decommission db2148.codfw.wmnet as Resolved.Apr 27 2026, 9:30 PM
Jhancock.wm closed subtask T424177: decommission db2145.codfw.wmnet as Resolved.Apr 27 2026, 9:34 PM
gerritbot added a comment.Apr 28 2026, 1:05 PM

Change #1276407 merged by Jcrespo:

[operations/puppet@production] mariadb: Set db2141 as a spare for decommissioning

https://gerrit.wikimedia.org/r/1276407

gerritbot added a comment.Apr 29 2026, 5:44 AM

Change #1279080 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Do not reimage db2251

https://gerrit.wikimedia.org/r/1279080

gerritbot added a comment.Apr 29 2026, 5:47 AM

Change #1279080 merged by Marostegui:

[operations/puppet@production] installserver: Do not reimage db2251

https://gerrit.wikimedia.org/r/1279080

Jhancock.wm closed subtask T424189: decommission db2146.codfw.wmnet as Resolved.May 4 2026, 3:50 PM
Jhancock.wm closed subtask T424226: decommission db2147.codfw.wmnet as Resolved.
Jhancock.wm closed subtask T424327: decommission db2141.codfw.wmnet as Resolved.
Stashbot added a comment.May 6 2026, 7:58 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-06T07:58:52Z] <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool db2144: Replacing HW T418979

Stashbot added a comment.May 6 2026, 7:59 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-06T07:59:01Z] <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2144: Replacing HW T418979

gerritbot added a comment.May 6 2026, 8:01 AM

Change #1283619 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db2253

https://gerrit.wikimedia.org/r/1283619

gerritbot added a comment.May 6 2026, 8:02 AM

Change #1283619 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db2253

https://gerrit.wikimedia.org/r/1283619

gerritbot added a comment.May 6 2026, 8:38 AM

Change #1283638 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db2253 to dbctl

https://gerrit.wikimedia.org/r/1283638

gerritbot added a comment.May 6 2026, 8:39 AM

Change #1283638 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db2253 to dbctl

https://gerrit.wikimedia.org/r/1283638

gerritbot added a comment.May 6 2026, 8:52 AM

Change #1283644 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2253: Enable notifications

https://gerrit.wikimedia.org/r/1283644

gerritbot added a comment.May 6 2026, 8:53 AM

Change #1283644 merged by Marostegui:

[operations/puppet@production] db2253: Enable notifications

https://gerrit.wikimedia.org/r/1283644

Stashbot added a comment.May 6 2026, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-06T09:14:04Z] <marostegui@cumin1003> START - Cookbook sre.mysql.pool pool db2253: Replacing HW T418979

Stashbot added a comment.May 6 2026, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-06T09:14:10Z] <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2253: Replacing HW T418979

Marostegui updated the task description. (Show Details)May 6 2026, 9:16 AM
Marostegui closed this task as Resolved.May 6 2026, 9:54 AM

This is all done and decommissioning will be tracked at T424028

Maintenance_bot moved this task from In progress to Done on the DBA board.May 6 2026, 10:29 AM
gerritbot added a comment.May 7 2026, 8:30 AM

Change #1284555 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Decommission db2144

https://gerrit.wikimedia.org/r/1284555

gerritbot added a comment.May 7 2026, 8:32 AM

Change #1284555 merged by Marostegui:

[operations/puppet@production] mariadb: Decommission db2144

https://gerrit.wikimedia.org/r/1284555

Jhancock.wm closed subtask T424038: decommission db2142.codfw.wmnet as Resolved.May 12 2026, 4:22 PM
Marostegui closed subtask T424028: Decommission db2141-db2152 as Resolved.May 19 2026, 1:18 PM
Jhancock.wm closed subtask T424171: decommission db2143.codfw.wmnet as Resolved.May 22 2026, 4:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits