Page MenuHomePhabricator
Create Task
Maniphest T390530

Create topology for x3 hosts
Closed, ResolvedPublic
Actions

Description

x3 hosts [T381475] are currently living in s8 and are normal replicas.
We need to split them into x3 so we have to create the puppet side of things and also create the topology for x3 to replicate from s8 master.

  • eqiad:
    • db1255 master
    • db1256 slave
    • db1257 slave
    • db1258 slave T393989
  • codfw:
    • db2241 master
    • db2242 slave
    • db2243 slave
    • db2244 slave T393989
  • Make masters have the master role in puppet
  • Make masters as such in zarcillo
  • dbctl config changes
  • conftool changes

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
mariadb: Switch from s8 to x3operations/puppetproduction+6 -12
db1211,db2162: Make them candidate mastersoperations/puppetproduction+2 -2
db2187: Enable notificationsoperations/puppetproduction+0 -1
x3 masters: Make them SBRoperations/puppetproduction+2 -0
db1211,db2162: Move them under x3operations/puppetproduction+6 -4
x3: Make db1255 and db2241 mastersoperations/puppetproduction+7 -7
x3 eqiad replicas: Make them SBRoperations/puppetproduction+6 -3
x3 codfw replicas: Make them SBRoperations/puppetproduction+6 -3
check_depooled: Add x3operations/softwaremaster+1 -1
db1258: Enable notificationsoperations/puppetproduction+0 -1
wmnet: Add x3-masteroperations/dnsmaster+1 -0
dbconfig.schema: Add x3operations/puppetproduction+1 -1
instances.yaml: Add db1258 to dbctloperations/puppetproduction+1 -0
mariadb: Productionize db1258operations/puppetproduction+3 -6
instances.yaml: Add db1256 to dbctloperations/puppetproduction+1 -0
db1256: Enable notificationsoperations/puppetproduction+0 -1
instances.yaml: Add db1255 to dbctloperations/puppetproduction+1 -0
db1255: Enable notificationsoperations/puppetproduction+0 -1
db2241,db2242: Enable notificationsoperations/puppetproduction+0 -2
instances: Add db2241, db2242operations/puppetproduction+2 -0
instance.schema: Add x3operations/puppetproduction+1 -1
site: Reorganize x3 hostsoperations/puppetproduction+6 -1
sections.yaml: Add x3operations/puppetproduction+2 -0
db2241: Make note about its future roleoperations/puppetproduction+1 -0
db1255: Make note about its future roleoperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT379757 Q2:rack/setup/install db224[12]
Unknown Object (Task)
Unknown Object (Task)
 ResolvedJhancock.wmT382425 Q2:rack/setup/install db2243
Unknown Object (Task)
 ResolvedNoneT351820 Move Wikidata term store to separate database cluster
 ResolvedJclark-ctrT384979 Q3:rack/setup/install db1257
 ResolvedMarosteguiT381475 Productionize x3 hosts
 ResolvedMarosteguiT390530 Create topology for x3 hosts
 ResolvedMarosteguiT393989 Productionize new x3 hosts

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a project: Patch-For-Review.Apr 24 2025, 2:03 PM
gerritbot added a comment.Apr 24 2025, 2:06 PM

Change #1138774 merged by Marostegui:

[operations/puppet@production] site: Reorganize x3 hosts

https://gerrit.wikimedia.org/r/1138774

Maintenance_bot removed a project: Patch-For-Review.Apr 24 2025, 2:30 PM
Marostegui updated the task description. (Show Details)Apr 28 2025, 7:30 AM
gerritbot added a comment.Apr 28 2025, 7:33 AM

Change #1139350 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instance.schema: Add x3

https://gerrit.wikimedia.org/r/1139350

gerritbot added a project: Patch-For-Review.Apr 28 2025, 7:33 AM
gerritbot added a comment.Apr 29 2025, 6:26 AM

Change #1139350 merged by Marostegui:

[operations/puppet@production] instance.schema: Add x3

https://gerrit.wikimedia.org/r/1139350

Maintenance_bot removed a project: Patch-For-Review.Apr 29 2025, 6:30 AM
Marostegui moved this task from Ready to In progress on the DBA board.May 13 2025, 11:18 AM
gerritbot added a comment.May 13 2025, 11:21 AM

Change #1145151 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances: Add db2241, db2242

https://gerrit.wikimedia.org/r/1145151

gerritbot added a project: Patch-For-Review.May 13 2025, 11:21 AM
Marostegui added a subtask: T393989: Productionize new x3 hosts.May 13 2025, 11:21 AM
gerritbot added a comment.May 13 2025, 11:23 AM

Change #1145151 merged by Marostegui:

[operations/puppet@production] instances: Add db2241, db2242

https://gerrit.wikimedia.org/r/1145151

Maintenance_bot removed a project: Patch-For-Review.May 13 2025, 11:31 AM
Stashbot added a comment.May 13 2025, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-13T11:31:39Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add db2241 and db2242 future x3 hosts, to s8 T390530', diff saved to https://phabricator.wikimedia.org/P75996 and previous config saved to /var/cache/conftool/dbconfig/20250513-113138-marostegui.json

gerritbot added a comment.May 13 2025, 11:38 AM

Change #1145165 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2241,db2242: Enable notifications

https://gerrit.wikimedia.org/r/1145165

gerritbot added a project: Patch-For-Review.May 13 2025, 11:38 AM

Change #1145165 merged by Marostegui:

[operations/puppet@production] db2241,db2242: Enable notifications

https://gerrit.wikimedia.org/r/1145165

gerritbot added a comment.May 13 2025, 11:59 AM

Change #1145171 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1255: Enable notifications

https://gerrit.wikimedia.org/r/1145171

gerritbot added a comment.May 13 2025, 12:01 PM

Change #1145171 merged by Marostegui:

[operations/puppet@production] db1255: Enable notifications

https://gerrit.wikimedia.org/r/1145171

gerritbot added a comment.May 13 2025, 12:04 PM

Change #1145173 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db1255 to dbctl

https://gerrit.wikimedia.org/r/1145173

gerritbot added a comment.May 13 2025, 12:06 PM

Change #1145173 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db1255 to dbctl

https://gerrit.wikimedia.org/r/1145173

Stashbot added a comment.May 13 2025, 12:08 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-13T12:08:54Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add db1255 future x3 hosts, to s8 T390530', diff saved to https://phabricator.wikimedia.org/P76009 and previous config saved to /var/cache/conftool/dbconfig/20250513-120853-marostegui.json

gerritbot added a comment.May 13 2025, 12:25 PM

Change #1145182 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1256: Enable notifications

https://gerrit.wikimedia.org/r/1145182

gerritbot added a comment.May 13 2025, 12:31 PM

Change #1145182 merged by Marostegui:

[operations/puppet@production] db1256: Enable notifications

https://gerrit.wikimedia.org/r/1145182

gerritbot added a comment.May 13 2025, 12:34 PM

Change #1145185 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db1256 to dbctl

https://gerrit.wikimedia.org/r/1145185

gerritbot added a comment.May 13 2025, 12:36 PM

Change #1145185 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db1256 to dbctl

https://gerrit.wikimedia.org/r/1145185

Stashbot added a comment.May 13 2025, 12:39 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-13T12:39:18Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add db1256 future x3 hosts, to s8 T390530', diff saved to https://phabricator.wikimedia.org/P76022 and previous config saved to /var/cache/conftool/dbconfig/20250513-123917-marostegui.json

Marostegui added a comment.May 13 2025, 12:40 PM

All these hosts are now serving traffic in s8 as replicas.

Maintenance_bot removed a project: Patch-For-Review.May 13 2025, 1:31 PM
Marostegui updated the task description. (Show Details)May 14 2025, 5:15 AM
gerritbot added a comment.May 14 2025, 5:18 AM

Change #1145428 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db1258

https://gerrit.wikimedia.org/r/1145428

gerritbot added a project: Patch-For-Review.May 14 2025, 5:18 AM
gerritbot added a comment.May 14 2025, 5:29 AM

Change #1145428 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db1258

https://gerrit.wikimedia.org/r/1145428

Maintenance_bot removed a project: Patch-For-Review.May 14 2025, 5:32 AM
Stashbot added a comment.May 14 2025, 5:49 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-14T05:49:11Z] <marostegui> Mark db1255 as x3 master in zarcillo T390530

gerritbot added a comment.May 14 2025, 7:48 AM

Change #1145799 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db1258 to dbctl

https://gerrit.wikimedia.org/r/1145799

gerritbot added a project: Patch-For-Review.May 14 2025, 7:48 AM
gerritbot added a comment.May 14 2025, 7:50 AM

Change #1145799 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db1258 to dbctl

https://gerrit.wikimedia.org/r/1145799

Maintenance_bot removed a project: Patch-For-Review.May 14 2025, 8:31 AM
Stashbot added a comment.May 14 2025, 8:53 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-14T08:53:57Z] <marostegui> Mark db2241 as x3 master in zarcillo T390530

Marostegui updated the task description. (Show Details)May 14 2025, 8:54 AM
gerritbot added a comment.May 14 2025, 9:31 AM

Change #1145834 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbconfig.schema: Add x3

https://gerrit.wikimedia.org/r/1145834

gerritbot added a project: Patch-For-Review.May 14 2025, 9:31 AM
gerritbot added a comment.May 14 2025, 9:35 AM

Change #1145834 merged by Marostegui:

[operations/puppet@production] dbconfig.schema: Add x3

https://gerrit.wikimedia.org/r/1145834

Stashbot added a comment.May 14 2025, 9:40 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-14T09:40:39Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add x3 eqiad T390530', diff saved to https://phabricator.wikimedia.org/P76131 and previous config saved to /var/cache/conftool/dbconfig/20250514-094038-marostegui.json

Marostegui updated the task description. (Show Details)May 14 2025, 9:40 AM
Stashbot added a comment.May 14 2025, 9:50 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-14T09:50:31Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add x3 codfw T390530', diff saved to https://phabricator.wikimedia.org/P76135 and previous config saved to /var/cache/conftool/dbconfig/20250514-095031-marostegui.json

Marostegui updated the task description. (Show Details)May 14 2025, 9:51 AM

x3 codfw and eqiad have been pushed.
So far all the hosts in s8 are also serving x3.

Pending:

  • Disconnect both s8 and x3
  • Make the masters, real masters in puppet and dbctl
Marostegui added a comment.May 14 2025, 9:53 AM

Also pending db2244, which still needs to be racked and installed (T393195)

gerritbot added a comment.May 14 2025, 9:54 AM

Change #1145841 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Add x3-master

https://gerrit.wikimedia.org/r/1145841

gerritbot added a comment.May 14 2025, 9:57 AM

Change #1145841 merged by Marostegui:

[operations/dns@master] wmnet: Add x3-master

https://gerrit.wikimedia.org/r/1145841

Maintenance_bot removed a project: Patch-For-Review.May 14 2025, 10:31 AM
gerritbot added a comment.May 14 2025, 12:12 PM

Change #1145873 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1258: Enable notifications

https://gerrit.wikimedia.org/r/1145873

gerritbot added a project: Patch-For-Review.May 14 2025, 12:12 PM

Change #1145873 merged by Marostegui:

[operations/puppet@production] db1258: Enable notifications

https://gerrit.wikimedia.org/r/1145873

gerritbot added a comment.May 14 2025, 12:20 PM

Change #1145877 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] check_depooled: Add x3

https://gerrit.wikimedia.org/r/1145877

gerritbot added a comment.May 14 2025, 12:22 PM

Change #1145877 merged by jenkins-bot:

[operations/software@master] check_depooled: Add x3

https://gerrit.wikimedia.org/r/1145877

Maintenance_bot removed a project: Patch-For-Review.May 14 2025, 12:31 PM
Marostegui mentioned this in T351820: Move Wikidata term store to separate database cluster.May 20 2025, 11:39 AM
Marostegui added a comment.May 28 2025, 12:24 PM

I have "moved" the hosts in zarcillo from s8 to x3

gerritbot added a comment.May 28 2025, 12:31 PM

Change #1151668 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] x3: Make db1255 and db2241 masters

https://gerrit.wikimedia.org/r/1151668

gerritbot added a project: Patch-For-Review.May 28 2025, 12:31 PM
gerritbot added a comment.May 28 2025, 12:37 PM

Change #1151670 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] x3 codfw replicas: Make them SBR

https://gerrit.wikimedia.org/r/1151670

Stashbot added a comment.May 28 2025, 12:38 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-28T12:38:46Z] <marostegui> dbmaint x3 codfw make it SBR T390530

gerritbot added a comment.May 28 2025, 12:39 PM

Change #1151670 merged by Marostegui:

[operations/puppet@production] x3 codfw replicas: Make them SBR

https://gerrit.wikimedia.org/r/1151670

gerritbot added a comment.May 28 2025, 12:51 PM

Change #1151674 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] x3 eqiad replicas: Make them SBR

https://gerrit.wikimedia.org/r/1151674

gerritbot added a comment.May 28 2025, 12:53 PM

Change #1151674 merged by Marostegui:

[operations/puppet@production] x3 eqiad replicas: Make them SBR

https://gerrit.wikimedia.org/r/1151674

Stashbot added a comment.May 28 2025, 12:53 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-28T12:53:44Z] <marostegui> dbmaint x3 eqiad make it SBR T383795 T390530

Stashbot mentioned this in T383795: Move sX to STATEMENT based replication.May 28 2025, 12:53 PM
gerritbot added a comment.May 28 2025, 1:40 PM

Change #1151668 merged by Marostegui:

[operations/puppet@production] x3: Make db1255 and db2241 masters

https://gerrit.wikimedia.org/r/1151668

Stashbot added a comment.May 28 2025, 1:44 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-28T13:44:31Z] <marostegui> Move db1211 and db2162 under x3 masters T390530 T351820

gerritbot added a comment.May 28 2025, 1:45 PM

Change #1151692 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1211,db2162: Move them under x3

https://gerrit.wikimedia.org/r/1151692

gerritbot added a comment.May 28 2025, 1:47 PM

Change #1151692 merged by Marostegui:

[operations/puppet@production] db1211,db2162: Move them under x3

https://gerrit.wikimedia.org/r/1151692

gerritbot added a comment.May 28 2025, 1:56 PM

Change #1151695 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] x3 masters: Make them SBR

https://gerrit.wikimedia.org/r/1151695

gerritbot added a comment.May 28 2025, 1:57 PM

Change #1151695 merged by Marostegui:

[operations/puppet@production] x3 masters: Make them SBR

https://gerrit.wikimedia.org/r/1151695

Maintenance_bot removed a project: Patch-For-Review.May 28 2025, 2:32 PM
gerritbot added a comment.May 28 2025, 2:36 PM

Change #1151712 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2187: Enable notifications

https://gerrit.wikimedia.org/r/1151712

gerritbot added a project: Patch-For-Review.May 28 2025, 2:36 PM

Change #1151712 merged by Marostegui:

[operations/puppet@production] db2187: Enable notifications

https://gerrit.wikimedia.org/r/1151712

Marostegui updated the task description. (Show Details)May 28 2025, 2:37 PM

This is done. Just pending to productionize db2244 which has not been racked yet (T393989 T393195)

Marostegui closed this task as Resolved.May 28 2025, 2:40 PM
gerritbot added a comment.May 28 2025, 2:42 PM

Change #1151714 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1211,db2162: Make them candidate masters

https://gerrit.wikimedia.org/r/1151714

gerritbot added a comment.May 28 2025, 2:43 PM

Change #1151714 merged by Marostegui:

[operations/puppet@production] db1211,db2162: Make them candidate masters

https://gerrit.wikimedia.org/r/1151714

gerritbot added a comment.May 28 2025, 3:26 PM

Change #1151726 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Switch from s8 to x3

https://gerrit.wikimedia.org/r/1151726

Maintenance_bot moved this task from In progress to Done on the DBA board.May 28 2025, 3:29 PM
gerritbot added a comment.May 28 2025, 3:29 PM

Change #1151726 merged by Marostegui:

[operations/puppet@production] mariadb: Switch from s8 to x3

https://gerrit.wikimedia.org/r/1151726

Maintenance_bot removed a project: Patch-For-Review.May 28 2025, 3:31 PM
Marostegui closed subtask T393989: Productionize new x3 hosts as Resolved.Jun 9 2025, 9:38 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits