Page MenuHomePhabricator
Create Task
Maniphest T391354

Q4 object storage hardware tasks
Closed, ResolvedPublic
Actions

Description

There's quite a lot of hardware work being done in FY2024-25/Q4, this task is to keep track of it all

Thanos refresh

Apus expansion

  • codfw frontend apus-fe2003 T390578
  • codfw backend apus-be2004 T388242 T392845 (installed)
  • eqiad frontend apus-fe1003 T388239 T389632 (installed)
  • eqiad backend apus-be1004 T388241 T392844 (installed)

MS frontend expansion

  • codfw frontend ms-fe201[56] T388887
  • eqiad frontend ms-fe101[56] T385040 T388886 (ready to be pooled)

MS hardware refresh (pulled forward from Q1 of next FY due to T392796 (failure of ms-be1060)

  • decommission ms-be1060 - handed off to DC-Ops in T393609
  • bring ms-be109[2-5] into service T393046 T393104 (being loaded into the rings)
  • decommission ms-be106[1-3] - handed off to DC-Ops in T401368

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedMatthewVernonT391354 Q4 object storage hardware tasks
 ResolvedMatthewVernonT391352 Q4 Thanos hardware refresh

Event Timeline

MatthewVernon created this task.Apr 8 2025, 2:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 8 2025, 2:11 PM
Aklapper updated the task description. (Show Details)Apr 8 2025, 2:21 PM
MatthewVernon moved this task from Inbox to In progress on the SRE-swift-storage board.Apr 8 2025, 2:39 PM
MatthewVernon updated the task description. (Show Details)Apr 11 2025, 9:32 AM
gerritbot added a comment.May 2 2025, 3:57 PM

Change #1140752 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: add ms-fe101[5,6] as new proxy nodes

https://gerrit.wikimedia.org/r/1140752

gerritbot added a project: Patch-For-Review.May 2 2025, 3:57 PM
MatthewVernon updated the task description. (Show Details)May 2 2025, 3:59 PM
MatthewVernon updated the task description. (Show Details)May 7 2025, 1:22 PM
gerritbot added a comment.May 7 2025, 1:23 PM

Change #1140752 merged by MVernon:

[operations/puppet@production] swift: add ms-fe101[5,6] as new proxy nodes

https://gerrit.wikimedia.org/r/1140752

Maintenance_bot removed a project: Patch-For-Review.May 7 2025, 1:32 PM
MatthewVernon updated the task description. (Show Details)May 7 2025, 1:45 PM
Stashbot added a comment.May 7 2025, 3:04 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-07T15:04:01Z] <Emperor> pool ms-fe1015 ms-fe1016 new frontends T388886 T391354

Stashbot mentioned this in T388886: Q4:rack/setup/install ms-fe101[56].May 7 2025, 3:04 PM
MatthewVernon updated the task description. (Show Details)May 7 2025, 3:05 PM
gerritbot added a comment.May 7 2025, 3:09 PM

Change #1143118 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] hiera: remove ms-be1060 from swift storagehosts

https://gerrit.wikimedia.org/r/1143118

gerritbot added a project: Patch-For-Review.May 7 2025, 3:09 PM
gerritbot added a comment.May 7 2025, 3:24 PM

Change #1143118 merged by MVernon:

[operations/puppet@production] hiera: remove ms-be1060 from swift storagehosts

https://gerrit.wikimedia.org/r/1143118

Maintenance_bot removed a project: Patch-For-Review.May 7 2025, 3:30 PM
MatthewVernon updated the task description. (Show Details)May 7 2025, 3:32 PM
MatthewVernon updated the task description. (Show Details)May 9 2025, 7:27 AM
MatthewVernon updated the task description. (Show Details)May 9 2025, 10:28 AM
MatthewVernon updated the task description. (Show Details)
gerritbot added a comment.May 9 2025, 1:30 PM

Change #1143821 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] apus: bring new frontend apus-fe1003 into service

https://gerrit.wikimedia.org/r/1143821

gerritbot added a project: Patch-For-Review.May 9 2025, 1:30 PM
MatthewVernon added a subtask: T391352: Q4 Thanos hardware refresh.May 9 2025, 1:42 PM
gerritbot added a comment.May 12 2025, 8:19 AM

Change #1143821 merged by MVernon:

[operations/puppet@production] apus: bring new frontend apus-fe1003 into service

https://gerrit.wikimedia.org/r/1143821

MatthewVernon updated the task description. (Show Details)May 12 2025, 8:30 AM
Maintenance_bot removed a project: Patch-For-Review.May 12 2025, 8:30 AM
MatthewVernon updated the task description. (Show Details)May 13 2025, 7:16 AM
MatthewVernon updated the task description. (Show Details)May 14 2025, 8:51 AM
MatthewVernon updated the task description. (Show Details)May 16 2025, 2:24 PM
MatthewVernon updated the task description. (Show Details)May 20 2025, 9:05 AM
gerritbot added a comment.May 20 2025, 9:14 AM

Change #1148280 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] apus: add apus-be1004 to eqiad cluster as osd server

https://gerrit.wikimedia.org/r/1148280

gerritbot added a project: Patch-For-Review.May 20 2025, 9:14 AM
gerritbot added a comment.May 20 2025, 9:20 AM

Change #1148280 merged by MVernon:

[operations/puppet@production] apus: add apus-be1004 to eqiad cluster as osd server

https://gerrit.wikimedia.org/r/1148280

Maintenance_bot removed a project: Patch-For-Review.May 20 2025, 9:30 AM
gerritbot added a comment.May 20 2025, 10:33 AM

Change #1148296 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] cephadm: handle storage servers with BOSS card

https://gerrit.wikimedia.org/r/1148296

gerritbot added a project: Patch-For-Review.May 20 2025, 10:33 AM
gerritbot added a comment.May 20 2025, 1:11 PM

Change #1148296 merged by MVernon:

[operations/puppet@production] cephadm: handle storage servers with BOSS card

https://gerrit.wikimedia.org/r/1148296

Maintenance_bot removed a project: Patch-For-Review.May 20 2025, 1:30 PM
MatthewVernon updated the task description. (Show Details)May 20 2025, 1:59 PM
gerritbot added a comment.May 27 2025, 10:56 AM

Change #1151166 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] hiera: add apus-be2004 to codfw apus cluster

https://gerrit.wikimedia.org/r/1151166

gerritbot added a project: Patch-For-Review.May 27 2025, 10:56 AM
gerritbot added a comment.May 27 2025, 11:53 AM

Change #1151166 merged by MVernon:

[operations/puppet@production] hiera: add apus-be2004 to codfw apus cluster

https://gerrit.wikimedia.org/r/1151166

Stashbot added a comment.May 27 2025, 12:00 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-27T12:00:55Z] <Emperor> ceph orch apply to bring apus-be2004 into service T391354

MatthewVernon updated the task description. (Show Details)May 27 2025, 12:02 PM
Maintenance_bot removed a project: Patch-For-Review.May 27 2025, 12:30 PM
gerritbot added a comment.Jun 4 2025, 2:59 PM

Change #1153648 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] ms-be eqiad: add 2 new backends, drain 2 old ones

https://gerrit.wikimedia.org/r/1153648

gerritbot added a project: Patch-For-Review.Jun 4 2025, 2:59 PM
gerritbot added a comment.Jun 4 2025, 3:12 PM

Change #1153648 merged by MVernon:

[operations/puppet@production] ms-be eqiad: add 2 new backends, drain 2 old ones

https://gerrit.wikimedia.org/r/1153648

MatthewVernon updated the task description. (Show Details)Jun 4 2025, 3:13 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 4 2025, 3:31 PM
gerritbot added a comment.Jul 2 2025, 10:46 AM

Change #1165851 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] hiera: add ms-be109[2-5] to swift::storagehosts

https://gerrit.wikimedia.org/r/1165851

gerritbot added a project: Patch-For-Review.Jul 2 2025, 10:46 AM

Change #1165852 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift/eqiad: add ms-be109[2,3], drain ms-be1063

https://gerrit.wikimedia.org/r/1165852

gerritbot added a comment.Jul 2 2025, 10:52 AM

Change #1165851 merged by MVernon:

[operations/puppet@production] hiera: add ms-be109[2-5] to swift::storagehosts

https://gerrit.wikimedia.org/r/1165851

gerritbot added a comment.Jul 2 2025, 10:59 AM

Change #1165852 merged by MVernon:

[operations/puppet@production] swift/eqiad: add ms-be109[2,3], drain ms-be1063

https://gerrit.wikimedia.org/r/1165852

MatthewVernon updated the task description. (Show Details)Jul 2 2025, 11:04 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 2 2025, 11:30 AM
MatthewVernon closed subtask T391352: Q4 Thanos hardware refresh as Resolved.Jul 7 2025, 3:38 PM
MatthewVernon updated the task description. (Show Details)
gerritbot added a comment.Aug 6 2025, 2:29 PM

Change #1176253 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: remove old nodes, drain & reweight SM C-J nodes

https://gerrit.wikimedia.org/r/1176253

gerritbot added a project: Patch-For-Review.Aug 6 2025, 2:29 PM
gerritbot added a comment.Aug 6 2025, 2:41 PM

Change #1176253 merged by MVernon:

[operations/puppet@production] swift: remove old nodes, drain & reweight SM C-J nodes

https://gerrit.wikimedia.org/r/1176253

gerritbot added a comment.Aug 6 2025, 2:57 PM

Change #1176258 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: remove ms-be106[1-3]

https://gerrit.wikimedia.org/r/1176258

gerritbot added a comment.Aug 7 2025, 7:58 AM

Change #1176258 merged by MVernon:

[operations/puppet@production] swift: remove ms-be106[1-3]

https://gerrit.wikimedia.org/r/1176258

MatthewVernon closed this task as Resolved.Aug 7 2025, 8:05 AM
MatthewVernon updated the task description. (Show Details)
ops-monitoring-bot added a comment.Aug 7 2025, 8:25 AM

cookbooks.sre.hosts.decommission executed by mvernon@cumin1003 for hosts: ms-be[1061-1063].eqiad.wmnet

  • ms-be1061.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1062.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1063.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Aug 7 2025, 8:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits