Page MenuHomePhabricator
Create Task
Maniphest T382569

Productionize es104[1-6]
Closed, ResolvedPublic
Actions

Description

  • es1041 to replace es1020
  • es1042 to replace es1021
  • es1043 to replace es1022
  • es1044 to replace es1023
  • es1045 to replace es1024
  • es1046 to replace es1025

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT378146 Q2:rack/setup/install es204[1-6]
Unknown Object (Task)
 ResolvedJhancock.wmT378143 Q2:rack/setup/install es104[1-6]
 ResolvedMarosteguiT382569 Productionize es104[1-6]
 ResolvedMarosteguiT383199 Decommission es102[0-5]
 ResolvedRequestPapaulT383578 decommission es1020.eqiad.wmnet
 ResolvedRequestPapaulT384418 decommission es1021.eqiad.wmnet
 ResolvedRequestPapaulT384566 decommission es1022.eqiad.wmnet
 ResolvedRequestJclark-ctrT384679 decommission es1023.eqiad.wmnet
 ResolvedRequestPapaulT384820 decommission es1024.eqiad.wmnet
 ResolvedRequestPapaulT384912 decommission es1025.eqiad.wmnet
 ResolvedVRiley-WMFT382485 InterfaceSpeedError

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Maintenance_bot removed a project: Patch-For-Review.Jan 8 2025, 7:30 AM
Marostegui updated the task description. (Show Details)Jan 8 2025, 8:08 AM
gerritbot added a comment.Jan 9 2025, 6:14 AM

Change #1109302 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es1042

https://gerrit.wikimedia.org/r/1109302

gerritbot added a project: Patch-For-Review.Jan 9 2025, 6:14 AM

Change #1109302 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es1042

https://gerrit.wikimedia.org/r/1109302

Stashbot added a comment.Jan 9 2025, 6:17 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-09T06:17:24Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add es1042 depooled T382569', diff saved to https://phabricator.wikimedia.org/P71892 and previous config saved to /var/cache/conftool/dbconfig/20250109-061724-marostegui.json

Marostegui added a comment.Jan 9 2025, 6:28 AM

es1042 is being slowly pooled.

Marostegui updated the task description. (Show Details)Jan 9 2025, 6:28 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 9 2025, 6:30 AM
gerritbot added a comment.Jan 9 2025, 6:31 AM

Change #1109305 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize es1043

https://gerrit.wikimedia.org/r/1109305

gerritbot added a project: Patch-For-Review.Jan 9 2025, 6:31 AM
gerritbot added a comment.Jan 9 2025, 6:38 AM

Change #1109305 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize es1043

https://gerrit.wikimedia.org/r/1109305

Stashbot added a comment.Jan 9 2025, 6:41 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-09T06:41:18Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1022 T382569', diff saved to https://phabricator.wikimedia.org/P71895 and previous config saved to /var/cache/conftool/dbconfig/20250109-064117-marostegui.json

Marostegui added a comment.Jan 9 2025, 6:42 AM

Starting the transfer from es1022 to es1043

Maintenance_bot removed a project: Patch-For-Review.Jan 9 2025, 7:30 AM
Marostegui added a comment.Jan 10 2025, 9:36 AM

I stopped the transfer to es1043, cause it is taking ages due to: T382485

Marostegui merged a task: T382485: InterfaceSpeedError.Jan 10 2025, 9:37 AM
Marostegui added a subtask: T382485: InterfaceSpeedError.
Marostegui added subscribers: phaultfinder, Papaul, VRiley-WMF, elukey.
Marostegui removed subscribers: elukey, VRiley-WMF, Papaul, phaultfinder.Jan 10 2025, 4:34 PM
Stashbot added a comment.Jan 13 2025, 9:48 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-13T09:48:33Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71987 and previous config saved to /var/cache/conftool/dbconfig/20250113-094833-marostegui.json

gerritbot added a comment.Jan 13 2025, 9:50 AM

Change #1110725 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize es1044

https://gerrit.wikimedia.org/r/1110725

gerritbot added a project: Patch-For-Review.Jan 13 2025, 9:50 AM

Change #1110725 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize es1044

https://gerrit.wikimedia.org/r/1110725

Maintenance_bot removed a project: Patch-For-Review.Jan 13 2025, 10:32 AM
Marostegui updated the task description. (Show Details)Jan 13 2025, 11:01 AM
Marostegui updated the task description. (Show Details)
Stashbot added a comment.Jan 13 2025, 3:51 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-13T15:51:35Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Set es1041 as es4 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P72014 and previous config saved to /var/cache/conftool/dbconfig/20250113-155135-marostegui.json

Stashbot added a comment.Jan 13 2025, 3:51 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-13T15:51:53Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1020 T382569', diff saved to https://phabricator.wikimedia.org/P72015 and previous config saved to /var/cache/conftool/dbconfig/20250113-155153-marostegui.json

Marostegui closed subtask T382485: InterfaceSpeedError as Resolved.Jan 14 2025, 8:00 AM
gerritbot added a comment.Jan 14 2025, 9:29 AM

Change #1111168 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es1044

https://gerrit.wikimedia.org/r/1111168

gerritbot added a project: Patch-For-Review.Jan 14 2025, 9:29 AM

Change #1111168 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es1044

https://gerrit.wikimedia.org/r/1111168

Maintenance_bot removed a project: Patch-For-Review.Jan 14 2025, 9:31 AM
Stashbot added a comment.Jan 14 2025, 9:31 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-14T09:31:48Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add es1044 to dbctl depooled T382569', diff saved to https://phabricator.wikimedia.org/P72025 and previous config saved to /var/cache/conftool/dbconfig/20250114-093147-marostegui.json

Marostegui added a comment.Jan 14 2025, 9:32 AM

es1044 is being pooled automatically.

Marostegui updated the task description. (Show Details)Jan 14 2025, 9:32 AM
Stashbot added a comment.Jan 14 2025, 9:33 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-14T09:33:16Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1022 T382569', diff saved to https://phabricator.wikimedia.org/P72027 and previous config saved to /var/cache/conftool/dbconfig/20250114-093315-marostegui.json

Marostegui updated the task description. (Show Details)Jan 14 2025, 9:34 AM
gerritbot added a comment.Jan 14 2025, 12:38 PM

Change #1111210 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Do not format es1042

https://gerrit.wikimedia.org/r/1111210

gerritbot added a project: Patch-For-Review.Jan 14 2025, 12:38 PM
gerritbot added a comment.Jan 14 2025, 12:41 PM

Change #1111210 merged by Marostegui:

[operations/puppet@production] installserver: Do not format es1042

https://gerrit.wikimedia.org/r/1111210

Maintenance_bot removed a project: Patch-For-Review.Jan 14 2025, 1:31 PM
gerritbot added a comment.Jan 15 2025, 8:20 AM

Change #1111567 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es1043 to dbctl

https://gerrit.wikimedia.org/r/1111567

gerritbot added a project: Patch-For-Review.Jan 15 2025, 8:20 AM
gerritbot added a comment.Jan 15 2025, 8:23 AM

Change #1111567 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es1043 to dbctl

https://gerrit.wikimedia.org/r/1111567

Stashbot added a comment.Jan 15 2025, 8:25 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-15T08:25:55Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add es1043 to dbctl depooled T382569', diff saved to https://phabricator.wikimedia.org/P72050 and previous config saved to /var/cache/conftool/dbconfig/20250115-082554-marostegui.json

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2025, 8:30 AM
gerritbot added a comment.Jan 15 2025, 9:17 AM

Change #1111576 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1043: Enable notifications

https://gerrit.wikimedia.org/r/1111576

gerritbot added a project: Patch-For-Review.Jan 15 2025, 9:17 AM
Marostegui updated the task description. (Show Details)Jan 15 2025, 9:17 AM
gerritbot added a comment.Jan 15 2025, 9:18 AM

Change #1111576 merged by Marostegui:

[operations/puppet@production] es1043: Enable notifications

https://gerrit.wikimedia.org/r/1111576

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2025, 9:31 AM
Stashbot added a comment.Jan 15 2025, 10:02 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-15T10:02:07Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote es1025 to eqiad es5 master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P72059 and previous config saved to /var/cache/conftool/dbconfig/20250115-100207-marostegui.json

Stashbot added a comment.Jan 15 2025, 10:02 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-15T10:02:28Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1024 T382569', diff saved to https://phabricator.wikimedia.org/P72060 and previous config saved to /var/cache/conftool/dbconfig/20250115-100228-marostegui.json

gerritbot added a comment.Jan 15 2025, 10:04 AM

Change #1111584 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize es1045

https://gerrit.wikimedia.org/r/1111584

gerritbot added a project: Patch-For-Review.Jan 15 2025, 10:04 AM
gerritbot added a comment.Jan 15 2025, 10:09 AM

Change #1111584 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize es1045

https://gerrit.wikimedia.org/r/1111584

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2025, 10:31 AM
Marostegui updated the task description. (Show Details)Jan 15 2025, 10:49 AM
gerritbot added a comment.Jan 16 2025, 8:31 AM

Change #1111918 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es1045 to dbctl

https://gerrit.wikimedia.org/r/1111918

gerritbot added a project: Patch-For-Review.Jan 16 2025, 8:31 AM

Change #1111918 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es1045 to dbctl

https://gerrit.wikimedia.org/r/1111918

Stashbot added a comment.Jan 16 2025, 8:36 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-16T08:36:00Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add es1045 to dbctl depooled T382569', diff saved to https://phabricator.wikimedia.org/P72093 and previous config saved to /var/cache/conftool/dbconfig/20250116-083559-marostegui.json

Marostegui updated the task description. (Show Details)Jan 16 2025, 8:50 AM
Stashbot added a comment.Jan 16 2025, 8:53 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-16T08:53:05Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote es1023 to es5 primary T382569', diff saved to https://phabricator.wikimedia.org/P72096 and previous config saved to /var/cache/conftool/dbconfig/20250116-085305-root.json

gerritbot added a comment.Jan 16 2025, 8:59 AM

Change #1111921 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize es1046

https://gerrit.wikimedia.org/r/1111921

gerritbot added a comment.Jan 16 2025, 8:59 AM

Change #1111921 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize es1046

https://gerrit.wikimedia.org/r/1111921

Marostegui updated the task description. (Show Details)Jan 16 2025, 9:06 AM
gerritbot added a comment.Jan 17 2025, 11:19 AM

Change #1112190 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1046: Enable notifications

https://gerrit.wikimedia.org/r/1112190

gerritbot added a comment.Jan 17 2025, 11:21 AM

Change #1112190 merged by Marostegui:

[operations/puppet@production] es1046: Enable notifications

https://gerrit.wikimedia.org/r/1112190

gerritbot added a comment.Jan 17 2025, 11:27 AM

Change #1112192 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es1046

https://gerrit.wikimedia.org/r/1112192

gerritbot added a comment.Jan 17 2025, 12:09 PM

Change #1112192 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es1046

https://gerrit.wikimedia.org/r/1112192

Stashbot added a comment.Jan 17 2025, 12:31 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-17T12:31:54Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add es1046 to dbctl depooled T382569', diff saved to https://phabricator.wikimedia.org/P72136 and previous config saved to /var/cache/conftool/dbconfig/20250117-123153-marostegui.json

Marostegui updated the task description. (Show Details)Jan 17 2025, 12:32 PM

es1046 being automatically slowly pooled

Maintenance_bot removed a project: Patch-For-Review.Jan 17 2025, 1:30 PM
Marostegui closed this task as Resolved.Jan 17 2025, 2:22 PM
Marostegui updated the task description. (Show Details)

This is all done

Maintenance_bot moved this task from In progress to Done on the DBA board.Jan 17 2025, 2:29 PM
gerritbot added a comment.Jan 24 2025, 5:47 AM

Change #1113913 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] backup1002.cnf.erb: Replace es1022 with es1043

https://gerrit.wikimedia.org/r/1113913

gerritbot added a project: Patch-For-Review.Jan 24 2025, 5:47 AM
gerritbot added a comment.Jan 24 2025, 5:49 AM

Change #1113913 merged by Marostegui:

[operations/puppet@production] backup1002.cnf.erb: Replace es1022 with es1043

https://gerrit.wikimedia.org/r/1113913

Marostegui closed subtask T383199: Decommission es102[0-5] as Resolved.Jan 29 2025, 12:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits