Page MenuHomePhabricator
Create Task
Maniphest T381475

Productionize x3 hosts
Closed, ResolvedPublic
Actions

Description

eqiad

  • db1255 (Dell - ready) - cloned in s8
  • db1256 (Dell - ready) - cloned in s8
  • db1257 (Supermicro - ready) - cloned in s8

codfw

  • db2241 (Dell - ready) - cloned in s8
  • db2242 (Dell - ready) - cloned in s8
  • db2243 (Supermicro - ready) - cloned in s8

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Krinkle subscribed.Jan 30 2025, 7:10 PM
• Marostegui updated the task description. (Show Details)Feb 7 2025, 4:37 PM
• Marostegui added a parent task: T384979: Q3:rack/setup/install db1257.
• Marostegui added a parent task: T382425: Q2:rack/setup/install db2243.Feb 7 2025, 4:40 PM
• Marostegui updated the task description. (Show Details)Feb 25 2025, 6:33 AM

db2243 is ready, which means all the hosts in codfw are ready. Only missing db1257 in eqiad to unblock this T384979

• Marostegui updated the task description. (Show Details)Feb 28 2025, 8:27 AM
gerritbot added a comment.Mar 6 2025, 6:34 AM

Change #1125029 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Do not reimage db1255

https://gerrit.wikimedia.org/r/1125029

gerritbot added a project: Patch-For-Review.Mar 6 2025, 6:34 AM
gerritbot added a comment.Mar 6 2025, 6:37 AM

Change #1125029 merged by Marostegui:

[operations/puppet@production] installserver: Do not reimage db1255

https://gerrit.wikimedia.org/r/1125029

Maintenance_bot removed a project: Patch-For-Review.Mar 6 2025, 7:30 AM
• Marostegui updated the task description. (Show Details)Mar 18 2025, 6:19 AM
• Marostegui moved this task from Blocked to Ready on the DBA board.Mar 18 2025, 6:23 AM

Extended their virtual volume

gerritbot added a comment.Mar 18 2025, 6:28 AM

Change #1128746 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] valid_section.pp: Add x3

https://gerrit.wikimedia.org/r/1128746

gerritbot added a project: Patch-For-Review.Mar 18 2025, 6:28 AM
gerritbot added a comment.Mar 18 2025, 9:36 AM

Change #1128746 merged by Marostegui:

[operations/puppet@production] valid_section.pp: Add x3

https://gerrit.wikimedia.org/r/1128746

Maintenance_bot removed a project: Patch-For-Review.Mar 18 2025, 10:31 AM
gerritbot added a comment.Mar 25 2025, 10:39 AM

Change #1130976 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2243: Productionize

https://gerrit.wikimedia.org/r/1130976

gerritbot added a project: Patch-For-Review.Mar 25 2025, 10:39 AM
gerritbot added a comment.Mar 25 2025, 10:43 AM

Change #1130976 merged by Marostegui:

[operations/puppet@production] db2243: Productionize

https://gerrit.wikimedia.org/r/1130976

ops-monitoring-bot added a comment.Mar 25 2025, 10:52 AM

Started cloning db2181.codfw.wmnet to db2243.codfw.wmnet - marostegui@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Mar 25 2025, 11:32 AM
• Marostegui added subtasks: T377853: RAID monitoring on new hardware spec requires new or updated user space cli tool, Restricted Task.Mar 25 2025, 1:36 PM
• Marostegui moved this task from Ready to In progress on the DBA board.Mar 26 2025, 6:49 AM
Stashbot added a comment.Mar 26 2025, 6:50 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-26T06:50:38Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2181 T381475', diff saved to https://phabricator.wikimedia.org/P74423 and previous config saved to /var/cache/conftool/dbconfig/20250326-065037-marostegui.json

gerritbot added a comment.Mar 26 2025, 6:52 AM

Change #1131219 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2241: Productionize

https://gerrit.wikimedia.org/r/1131219

gerritbot added a project: Patch-For-Review.Mar 26 2025, 6:52 AM

Change #1131219 merged by Marostegui:

[operations/puppet@production] db2241: Productionize

https://gerrit.wikimedia.org/r/1131219

ops-monitoring-bot added a comment.Mar 26 2025, 7:03 AM

Started cloning db2181.codfw.wmnet to db2241.codfw.wmnet - marostegui@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2025, 7:30 AM
gerritbot added a comment.Mar 26 2025, 9:36 AM

Change #1131278 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db2242

https://gerrit.wikimedia.org/r/1131278

gerritbot added a project: Patch-For-Review.Mar 26 2025, 9:36 AM
gerritbot added a comment.Mar 26 2025, 9:39 AM

Change #1131278 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db2242

https://gerrit.wikimedia.org/r/1131278

ops-monitoring-bot added a comment.Mar 26 2025, 9:44 AM

Started cloning db2181.codfw.wmnet to db2242.codfw.wmnet - marostegui@cumin1002

• Marostegui updated the task description. (Show Details)Mar 26 2025, 9:53 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 26 2025, 10:31 AM
• Marostegui updated the task description. (Show Details)Mar 27 2025, 6:04 AM
gerritbot added a comment.Mar 27 2025, 6:08 AM

Change #1131500 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Do not reimage db2242

https://gerrit.wikimedia.org/r/1131500

gerritbot added a project: Patch-For-Review.Mar 27 2025, 6:08 AM
gerritbot added a comment.Mar 27 2025, 6:10 AM

Change #1131500 merged by Marostegui:

[operations/puppet@production] installserver: Do not reimage db2242

https://gerrit.wikimedia.org/r/1131500

Maintenance_bot removed a project: Patch-For-Review.Mar 27 2025, 6:30 AM
gerritbot added a comment.Mar 27 2025, 7:32 AM

Change #1131626 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db1255

https://gerrit.wikimedia.org/r/1131626

gerritbot added a project: Patch-For-Review.Mar 27 2025, 7:32 AM

Change #1131626 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db1255

https://gerrit.wikimedia.org/r/1131626

ops-monitoring-bot added a comment.Mar 27 2025, 7:54 AM

Started cloning db1211.eqiad.wmnet to db1255.eqiad.wmnet - marostegui@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Mar 27 2025, 8:30 AM
ops-monitoring-bot added a comment.Mar 27 2025, 1:19 PM

Started cloning db1211.eqiad.wmnet to db1255.eqiad.wmnet - marostegui@cumin1002

ops-monitoring-bot added a comment.Mar 27 2025, 1:21 PM

Completed depool of db1211 - Depool db1211.eqiad.wmnet to then clone it to db1255.eqiad.wmnet - marostegui@cumin1002 - marostegui@cumin1002

ops-monitoring-bot added a comment.Mar 27 2025, 1:45 PM

Started cloning db1211.eqiad.wmnet to db1255.eqiad.wmnet - marostegui@cumin1002

ops-monitoring-bot added a comment.Mar 27 2025, 1:48 PM

Started cloning db1211.eqiad.wmnet to db1255.eqiad.wmnet - marostegui@cumin1002

ops-monitoring-bot added a comment.Mar 27 2025, 1:49 PM

Started cloning db1211.eqiad.wmnet to db1255.eqiad.wmnet - marostegui@cumin1002

• Marostegui updated the task description. (Show Details)Mar 31 2025, 8:29 AM
gerritbot added a comment.Mar 31 2025, 8:41 AM

Change #1132542 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db1256

https://gerrit.wikimedia.org/r/1132542

gerritbot added a project: Patch-For-Review.Mar 31 2025, 8:41 AM

Change #1132542 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db1256

https://gerrit.wikimedia.org/r/1132542

ops-monitoring-bot added a comment.Mar 31 2025, 8:48 AM

Started cloning db1211.eqiad.wmnet to db1256.eqiad.wmnet - marostegui@cumin1002

ops-monitoring-bot added a comment.Mar 31 2025, 8:48 AM

Completed depool of db1211 - Depool db1211.eqiad.wmnet to then clone it to db1256.eqiad.wmnet - marostegui@cumin1002 - marostegui@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Mar 31 2025, 9:31 AM
• Marostegui updated the task description. (Show Details)Mar 31 2025, 11:05 AM
• Marostegui mentioned this in T390530: Create topology for x3 hosts.
• Marostegui changed the status of subtask T390530: Create topology for x3 hosts from Open to Stalled.
gerritbot added a comment.Apr 1 2025, 6:08 AM

Change #1132815 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db1257

https://gerrit.wikimedia.org/r/1132815

gerritbot added a project: Patch-For-Review.Apr 1 2025, 6:08 AM

Change #1132815 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db1257

https://gerrit.wikimedia.org/r/1132815

ops-monitoring-bot added a comment.Apr 1 2025, 6:14 AM

Started cloning db1211.eqiad.wmnet to db1257.eqiad.wmnet - marostegui@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Apr 1 2025, 6:30 AM
ops-monitoring-bot added a comment.Apr 1 2025, 8:38 AM

Start pool of db1211 slowly with 10 steps - Pool db1211.eqiad.wmnet in after cloning - marostegui@cumin1002

ops-monitoring-bot added a comment.Apr 1 2025, 10:55 AM

Completed pool of db1211 slowly with 10 steps - Pool db1211.eqiad.wmnet in after cloning - marostegui@cumin1002

ops-monitoring-bot added a comment.Apr 1 2025, 10:55 AM

Finished cloning db1211.eqiad.wmnet to db1257.eqiad.wmnet - marostegui@cumin1002

• Marostegui closed subtask T388684: Test hot disk swap on Supermicro database hosts as Resolved.Apr 1 2025, 4:24 PM
• Marostegui reopened subtask T388684: Test hot disk swap on Supermicro database hosts as Open.Apr 1 2025, 6:14 PM
• Marostegui updated the task description. (Show Details)Apr 2 2025, 9:25 AM
gerritbot added a comment.Apr 2 2025, 9:36 AM

Change #1133345 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db1257

https://gerrit.wikimedia.org/r/1133345

gerritbot added a project: Patch-For-Review.Apr 2 2025, 9:36 AM

Change #1133345 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db1257

https://gerrit.wikimedia.org/r/1133345

Stashbot added a comment.Apr 2 2025, 9:41 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-02T09:41:10Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add db1257 to dbctl depooled T381475', diff saved to https://phabricator.wikimedia.org/P74555 and previous config saved to /var/cache/conftool/dbconfig/20250402-094109-marostegui.json

gerritbot added a comment.Apr 2 2025, 9:41 AM

Change #1133347 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1257: Enable notifications

https://gerrit.wikimedia.org/r/1133347

gerritbot added a comment.Apr 2 2025, 9:42 AM

Change #1133347 merged by Marostegui:

[operations/puppet@production] db1257: Enable notifications

https://gerrit.wikimedia.org/r/1133347

gerritbot added a comment.Apr 2 2025, 9:45 AM

Change #1133348 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2243: Enable notifications

https://gerrit.wikimedia.org/r/1133348

gerritbot added a comment.Apr 2 2025, 9:45 AM

Change #1133348 merged by Marostegui:

[operations/puppet@production] db2243: Enable notifications

https://gerrit.wikimedia.org/r/1133348

gerritbot added a comment.Apr 2 2025, 9:48 AM

Change #1133349 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db2243

https://gerrit.wikimedia.org/r/1133349

gerritbot added a comment.Apr 2 2025, 9:49 AM

Change #1133349 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db2243

https://gerrit.wikimedia.org/r/1133349

Stashbot added a comment.Apr 2 2025, 9:52 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-02T09:52:13Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add db2243 to dbctl depooled T381475', diff saved to https://phabricator.wikimedia.org/P74557 and previous config saved to /var/cache/conftool/dbconfig/20250402-095213-marostegui.json

• Marostegui added a comment.Apr 2 2025, 9:56 AM

I am pooling db2243 and db1257 (both supermicro) into s8 to start also testing their performance.

Maintenance_bot removed a project: Patch-For-Review.Apr 2 2025, 10:30 AM
• Marostegui closed this task as Resolved.Apr 2 2025, 1:32 PM
• Marostegui updated the task description. (Show Details)

I am going to close this. The hosts have data and further, db1257 and db2243 are serving reads in production.
Next steps is build the topology and adapt puppet for them, that will be tracked at T390530

• Marostegui changed the status of subtask T390530: Create topology for x3 hosts from Stalled to Open.Apr 2 2025, 1:34 PM
Maintenance_bot moved this task from In progress to Done on the DBA board.Apr 2 2025, 2:29 PM
• Marostegui closed subtask T388684: Test hot disk swap on Supermicro database hosts as Resolved.Apr 2 2025, 4:00 PM
MoritzMuehlenhoff closed subtask Restricted Task as Resolved.Apr 29 2025, 1:05 PM
• Marostegui closed subtask T390530: Create topology for x3 hosts as Resolved.May 28 2025, 2:40 PM
elukey closed subtask T377853: RAID monitoring on new hardware spec requires new or updated user space cli tool as Resolved.Nov 4 2025, 3:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits