Page MenuHomePhabricator
Create Task
Maniphest T391352

Q4 Thanos hardware refresh
Closed, ResolvedPublic
Actions

Description

We are refreshing almost the entire thanos estate this quarter.

  • Frontend refresh
    • codfw
      • Bring thanos-fe200[5-7] into service T389634
      • Decommission thanos-fe200[1-3] (handed over to DC-ops via T393870)
    • eqiad
      • Bring thanos-fe100[5-7] into service T389501 T389635
      • Decommission thanos-fe100[1-3] (handed over to DC-ops via T394894)
  • Backend refresh
    • codfw
      • Bring thanos-be200[6-9] into service T389836 T392908 (being loaded into the rings)
      • Decommission thanos-be200[1-4] (handed over to DC-ops via T398849)
    • eqiad
      • Bring thanos-be100[6-9] into service T389837 T392909
      • Decommission thanos-be100[1-4] (handed over to DC-ops via T397414)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
hiera: remove thanos-be200[1-4] from thanos::swift::backendsoperations/puppetproduction+0 -4
thanos: remove now-drained thanos-be200[1-4] from ringsoperations/puppetproduction+0 -9
thanos: add new nodes to ring, drain old onesoperations/puppetproduction+12 -4
thanos: add new backends, remove old ones gone from ringsoperations/puppetproduction+4 -4
thanos: remove drained thanos-be100[1-4] from ringsoperations/puppetproduction+0 -8
thanos: add new backends, drain old onesoperations/puppetproduction+12 -4
thanos: add new backends to hieraoperations/puppetproduction+4 -0
thanos: remove old frontends thanos-fe100[1-3]operations/puppetproduction+2 -14
Thanos: add new thanos-fe100[5-7] nodesoperations/puppetproduction+12 -0
thanos: remove thanos-fe200[1-3]operations/puppetproduction+0 -12
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedMatthewVernonT391354 Q4 object storage hardware tasks
 ResolvedMatthewVernonT391352 Q4 Thanos hardware refresh

Event Timeline

MatthewVernon created this task.Apr 8 2025, 2:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 8 2025, 2:02 PM
MatthewVernon mentioned this in T391354: Q4 object storage hardware tasks.Apr 8 2025, 2:11 PM
MatthewVernon moved this task from Inbox to In progress on the SRE-swift-storage board.Apr 8 2025, 2:39 PM
MatthewVernon updated the task description. (Show Details)Apr 11 2025, 9:28 AM
Stashbot added a comment.Apr 24 2025, 9:27 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-24T09:27:18Z] <Emperor> depool thanos-fe200[1-3] pending decommissioning T391352

RobH mentioned this in Unknown Object (Task).Apr 24 2025, 6:05 PM
RobH mentioned this in Unknown Object (Task).
MatthewVernon updated the task description. (Show Details)May 9 2025, 10:32 AM
MatthewVernon updated the task description. (Show Details)
gerritbot added a comment.May 9 2025, 1:33 PM

Change #1143824 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] thanos: remove thanos-fe200[1-3]

https://gerrit.wikimedia.org/r/1143824

gerritbot added a project: Patch-For-Review.May 9 2025, 1:33 PM
MatthewVernon added a parent task: T391354: Q4 object storage hardware tasks.May 9 2025, 1:42 PM
gerritbot added a comment.May 12 2025, 8:33 AM

Change #1143824 merged by MVernon:

[operations/puppet@production] thanos: remove thanos-fe200[1-3]

https://gerrit.wikimedia.org/r/1143824

MatthewVernon updated the task description. (Show Details)May 12 2025, 8:55 AM
ops-monitoring-bot added a comment.May 12 2025, 9:04 AM

cookbooks.sre.hosts.decommission executed by mvernon@cumin1002 for hosts: thanos-fe[2001-2003].codfw.wmnet

  • thanos-fe2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • thanos-fe2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • thanos-fe2003.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Maintenance_bot removed a project: Patch-For-Review.May 12 2025, 9:30 AM
gerritbot added a comment.May 15 2025, 8:32 AM

Change #1146511 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] Thanos: add new thanos-fe100[5-7] nodes

https://gerrit.wikimedia.org/r/1146511

gerritbot added a project: Patch-For-Review.May 15 2025, 8:32 AM
gerritbot added a comment.May 15 2025, 9:02 AM

Change #1146511 merged by MVernon:

[operations/puppet@production] Thanos: add new thanos-fe100[5-7] nodes

https://gerrit.wikimedia.org/r/1146511

Stashbot added a comment.May 15 2025, 9:07 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-15T09:07:55Z] <Emperor> reboot thanos-fe100[5-7] prior to bringing into service T391352

Maintenance_bot removed a project: Patch-For-Review.May 15 2025, 9:30 AM
MatthewVernon updated the task description. (Show Details)May 15 2025, 9:31 AM
Stashbot added a comment.May 15 2025, 10:08 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-15T10:08:26Z] <Emperor> depool thanos-fe100[1-3] prior to decom T391352

MatthewVernon updated the task description. (Show Details)May 16 2025, 9:08 AM
MatthewVernon updated the task description. (Show Details)May 20 2025, 9:02 AM
gerritbot added a comment.May 20 2025, 1:08 PM

Change #1148330 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] thanos: remove old frontends thanos-fe100[1-3]

https://gerrit.wikimedia.org/r/1148330

gerritbot added a project: Patch-For-Review.May 20 2025, 1:08 PM
Stashbot added a comment.May 21 2025, 8:29 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-21T08:29:30Z] <Emperor> disable puppet on thanos-fe1001 and thanos-fe1004 T391352

gerritbot added a comment.May 21 2025, 8:32 AM

Change #1148330 merged by MVernon:

[operations/puppet@production] thanos: remove old frontends thanos-fe100[1-3]

https://gerrit.wikimedia.org/r/1148330

MatthewVernon updated the task description. (Show Details)May 21 2025, 8:56 AM
MatthewVernon updated the task description. (Show Details)
ops-monitoring-bot added a comment.May 21 2025, 9:12 AM

cookbooks.sre.hosts.decommission executed by mvernon@cumin1002 for hosts: thanos-fe[1001-1003].eqiad.wmnet

  • thanos-fe1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • thanos-fe1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • thanos-fe1003.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Maintenance_bot removed a project: Patch-For-Review.May 21 2025, 9:30 AM
MatthewVernon updated the task description. (Show Details)May 23 2025, 1:04 PM
MatthewVernon updated the task description. (Show Details)May 27 2025, 10:27 AM
gerritbot added a comment.May 27 2025, 10:36 AM

Change #1151159 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] thanos: add new backends to hiera

https://gerrit.wikimedia.org/r/1151159

gerritbot added a project: Patch-For-Review.May 27 2025, 10:36 AM

Change #1151160 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] thanos: add new backends, drain old ones

https://gerrit.wikimedia.org/r/1151160

CodeReviewBot added a comment.May 27 2025, 10:45 AM

mvernon opened https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/12

Add Eqiad E8

CodeReviewBot added a comment.May 27 2025, 11:45 AM

mvernon merged https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/12

Add Eqiad E8

gerritbot added a comment.May 27 2025, 11:46 AM

Change #1151159 merged by MVernon:

[operations/puppet@production] thanos: add new backends to hiera

https://gerrit.wikimedia.org/r/1151159

Stashbot added a comment.May 27 2025, 11:52 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-27T11:52:45Z] <Emperor> reboot thanos-be100[6-9] before bringing into the rings T391352

gerritbot added a comment.May 27 2025, 11:56 AM

Change #1151160 merged by MVernon:

[operations/puppet@production] thanos: add new backends, drain old ones

https://gerrit.wikimedia.org/r/1151160

MatthewVernon updated the task description. (Show Details)May 27 2025, 11:59 AM
MatthewVernon updated the task description. (Show Details)May 27 2025, 12:04 PM
CodeReviewBot added a comment.May 27 2025, 12:25 PM

mvernon opened https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/13

add eqiad F8

CodeReviewBot added a comment.May 27 2025, 1:10 PM

mvernon merged https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/13

add eqiad F8

Maintenance_bot removed a project: Patch-For-Review.May 27 2025, 1:30 PM
MatthewVernon updated the task description. (Show Details)Jun 18 2025, 1:40 PM
gerritbot added a comment.Jun 18 2025, 2:56 PM

Change #1160824 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] thanos: remove drained thanos-be100[1-4] from rings

https://gerrit.wikimedia.org/r/1160824

gerritbot added a project: Patch-For-Review.Jun 18 2025, 2:56 PM
gerritbot added a comment.Jun 18 2025, 3:36 PM

Change #1160824 merged by MVernon:

[operations/puppet@production] thanos: remove drained thanos-be100[1-4] from rings

https://gerrit.wikimedia.org/r/1160824

gerritbot added a comment.Jun 18 2025, 3:46 PM

Change #1160855 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] thanos: add new backends, remove old ones gone from rings

https://gerrit.wikimedia.org/r/1160855

gerritbot added a comment.Jun 18 2025, 3:46 PM

Change #1160856 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] thanos: add new nodes to ring, drain old ones

https://gerrit.wikimedia.org/r/1160856

CodeReviewBot added a comment.Jun 18 2025, 3:55 PM

mvernon opened https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/14

add 4 more racks to find_ip_zone

CodeReviewBot added a comment.Jun 18 2025, 4:42 PM

mvernon merged https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/14

add 4 more racks to find_ip_zone

gerritbot added a comment.Jun 19 2025, 7:55 AM

Change #1160855 merged by MVernon:

[operations/puppet@production] thanos: add new backends, remove old ones gone from rings

https://gerrit.wikimedia.org/r/1160855

gerritbot added a comment.Jun 19 2025, 7:59 AM

Change #1160856 merged by MVernon:

[operations/puppet@production] thanos: add new nodes to ring, drain old ones

https://gerrit.wikimedia.org/r/1160856

Maintenance_bot removed a project: Patch-For-Review.Jun 19 2025, 8:30 AM
ops-monitoring-bot added a comment.Jun 19 2025, 8:33 AM

cookbooks.sre.hosts.decommission executed by mvernon@cumin1003 for hosts: thanos-be[1001-1004].eqiad.wmnet

  • thanos-be1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • thanos-be1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • thanos-be1003.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • thanos-be1004.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
MatthewVernon updated the task description. (Show Details)Jun 19 2025, 8:34 AM
gerritbot added a comment.Jul 7 2025, 1:05 PM

Change #1166822 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] thanos: remove now-drained thanos-be200[1-4] from rings

https://gerrit.wikimedia.org/r/1166822

gerritbot added a project: Patch-For-Review.Jul 7 2025, 1:05 PM

Change #1166823 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] hiera: remove thanos-be200[1-4] from thanos::swift::backends

https://gerrit.wikimedia.org/r/1166823

gerritbot added a comment.Jul 7 2025, 1:20 PM

Change #1166822 merged by MVernon:

[operations/puppet@production] thanos: remove now-drained thanos-be200[1-4] from rings

https://gerrit.wikimedia.org/r/1166822

gerritbot added a comment.Jul 7 2025, 2:39 PM

Change #1166823 merged by MVernon:

[operations/puppet@production] hiera: remove thanos-be200[1-4] from thanos::swift::backends

https://gerrit.wikimedia.org/r/1166823

ops-monitoring-bot added a comment.Jul 7 2025, 3:25 PM

cookbooks.sre.hosts.decommission executed by mvernon@cumin2002 for hosts: thanos-be[2001-2004].codfw.wmnet

  • thanos-be2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • thanos-be2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • thanos-be2003.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • thanos-be2004.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Jul 7 2025, 3:31 PM
MatthewVernon closed this task as Resolved.Jul 7 2025, 3:36 PM
MatthewVernon mentioned this in T398849: decommission thanos-be200[1-4].codfw.wmnet.
MatthewVernon updated the task description. (Show Details)

All done!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits