Page MenuHomePhabricator
Create Task
Maniphest T424680

Add ganeti105[5678] and decom ganeti102[3456]
Closed, ResolvedPublic
Actions

Description

Racked in https://phabricator.wikimedia.org/T418903. Followup task for adding to the eqiad cluster.

Add

  • ganeti1055 (row A)
  • ganeti1056 (row A)
  • ganeti1057 (row A)
  • ganeti1058 (row C)

Remove

  • ganeti1023 (row A)
  • ganeti1024 (row C)
  • ganeti1025 (row A)
  • ganeti1026 (row A)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove ganeti1025/1026operations/puppetproduction+1 -8
Remove ganeti1024 from the eqiad clusteroperations/puppetproduction+2 -3
Make ganeti1057/1058 Ganeti nodesoperations/puppetproduction+3 -6
Remove ganeti1023 from Ganeti clusteroperations/puppetproduction+7 -2
Make ganeti1055/1056 Ganeti nodesoperations/puppetproduction+4 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.Apr 28 2026, 4:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 28 2026, 4:13 PM
MoritzMuehlenhoff triaged this task as Medium priority.May 4 2026, 11:43 AM
MoritzMuehlenhoff updated the task description. (Show Details)Wed, May 20, 6:55 AM
gerritbot added a comment.Wed, May 20, 7:55 AM

Change #1289890 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make ganeti1055/1056 Ganeti nodes

https://gerrit.wikimedia.org/r/1289890

gerritbot added a project: Patch-For-Review.Wed, May 20, 7:55 AM
gerritbot added a comment.Wed, May 20, 8:10 AM

Change #1289890 merged by Muehlenhoff:

[operations/puppet@production] Make ganeti1055/1056 Ganeti nodes

https://gerrit.wikimedia.org/r/1289890

Maintenance_bot removed a project: Patch-For-Review.Wed, May 20, 8:30 AM
MoritzMuehlenhoff renamed this task from Add ganeti105[5678] to Add ganeti105[5678] and decom ganeti102[3456].Wed, May 20, 1:26 PM
MoritzMuehlenhoff updated the task description. (Show Details)Thu, May 21, 6:23 AM
ops-monitoring-bot added a comment.Thu, May 21, 6:25 AM

VM aux-k8s-etcd1003.eqiad.wmnet switching disk type to drbd

ops-monitoring-bot added a comment.Thu, May 21, 6:52 AM

VM aux-k8s-etcd1003.eqiad.wmnet switching disk type to plain

ops-monitoring-bot added a comment.Thu, May 21, 6:54 AM

VM dse-k8s-etcd1001.eqiad.wmnet switching disk type to drbd

ops-monitoring-bot added a comment.Thu, May 21, 7:18 AM

VM dse-k8s-etcd1001.eqiad.wmnet switching disk type to plain

ops-monitoring-bot added a comment.Thu, May 21, 7:28 AM

VM kubestagemaster1005.eqiad.wmnet switching disk type to drbd

ops-monitoring-bot added a comment.Thu, May 21, 7:44 AM

VM kubestagemaster1005.eqiad.wmnet switching disk type to plain

ops-monitoring-bot added a comment.Thu, May 21, 7:47 AM

Draining ganeti1023.eqiad.wmnet of running VMs

gerritbot added a comment.Thu, May 21, 8:47 AM

Change #1290683 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti1023 from Ganeti cluster

https://gerrit.wikimedia.org/r/1290683

gerritbot added a project: Patch-For-Review.Thu, May 21, 8:47 AM
Stashbot added a comment.Thu, May 21, 9:18 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-21T09:18:52Z] <moritzm> remove ganeti1023 foom eqiad Ganeti cluster T424680

gerritbot added a comment.Thu, May 21, 9:19 AM

Change #1290683 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti1023 from Ganeti cluster

https://gerrit.wikimedia.org/r/1290683

Maintenance_bot removed a project: Patch-For-Review.Thu, May 21, 9:30 AM
MoritzMuehlenhoff updated the task description. (Show Details)Thu, May 21, 9:37 AM
ops-monitoring-bot added a comment.Fri, May 22, 6:08 AM

Draining ganeti1024.eqiad.wmnet of running VMs

gerritbot added a comment.Fri, May 22, 6:11 AM

Change #1291630 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make ganeti1057/1058 Ganeti nodes

https://gerrit.wikimedia.org/r/1291630

gerritbot added a project: Patch-For-Review.Fri, May 22, 6:11 AM
gerritbot added a comment.Fri, May 22, 6:29 AM

Change #1291630 merged by Muehlenhoff:

[operations/puppet@production] Make ganeti1057/1058 Ganeti nodes

https://gerrit.wikimedia.org/r/1291630

Maintenance_bot removed a project: Patch-For-Review.Fri, May 22, 6:30 AM
ops-monitoring-bot added a comment.Fri, May 22, 7:32 AM

Draining ganeti1024.eqiad.wmnet of running VMs

MoritzMuehlenhoff updated the task description. (Show Details)Fri, May 22, 7:59 AM
gerritbot added a comment.Fri, May 22, 9:37 AM

Change #1291905 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti1024 from the eqiad cluster

https://gerrit.wikimedia.org/r/1291905

gerritbot added a project: Patch-For-Review.Fri, May 22, 9:37 AM
Stashbot added a comment.Fri, May 22, 10:37 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-22T10:37:31Z] <moritzm> remove ganeti1024 foom eqiad Ganeti cluster T424680

gerritbot added a comment.Fri, May 22, 10:38 AM

Change #1291905 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti1024 from the eqiad cluster

https://gerrit.wikimedia.org/r/1291905

MoritzMuehlenhoff updated the task description. (Show Details)Fri, May 22, 10:44 AM
Maintenance_bot removed a project: Patch-For-Review.Fri, May 22, 11:31 AM
MoritzMuehlenhoff updated the task description. (Show Details)Fri, May 22, 12:11 PM
ops-monitoring-bot added a comment.Tue, May 26, 7:10 AM

Draining ganeti1025.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Tue, May 26, 7:44 AM

Draining ganeti1025.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Tue, May 26, 7:59 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti1023.eqiad.wmnet

  • ganeti1023.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
ops-monitoring-bot added a comment.Tue, May 26, 8:35 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti1024.eqiad.wmnet

  • ganeti1024.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
Stashbot added a comment.Tue, May 26, 2:52 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-26T14:52:02Z] <moritzm> remove ganeti1025 from eqiad Ganeti cluster T424680

MoritzMuehlenhoff updated the task description. (Show Details)Tue, May 26, 2:52 PM
ops-monitoring-bot added a comment.Tue, May 26, 2:53 PM

Draining ganeti1026.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Tue, May 26, 2:55 PM

Draining ganeti1026.eqiad.wmnet of running VMs

Jclark-ctr subscribed.Tue, May 26, 8:02 PM
Stashbot added a comment.Wed, May 27, 5:38 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-27T05:38:07Z] <moritzm> remove ganeti1026 from eqiad Ganeti cluster T424680

gerritbot added a comment.Wed, May 27, 6:54 AM

Change #1294115 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti1025/1026

https://gerrit.wikimedia.org/r/1294115

gerritbot added a project: Patch-For-Review.Wed, May 27, 6:54 AM
ops-monitoring-bot added a comment.Wed, May 27, 7:14 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti1025.eqiad.wmnet

  • ganeti1025.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
ops-monitoring-bot added a comment.Wed, May 27, 7:43 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti1026.eqiad.wmnet

  • ganeti1026.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
gerritbot added a comment.Wed, May 27, 7:53 AM

Change #1294115 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti1025/1026

https://gerrit.wikimedia.org/r/1294115

MoritzMuehlenhoff updated the task description. (Show Details)Wed, May 27, 8:07 AM
MoritzMuehlenhoff created subtask T427353: Repurpose ganeti102[3456] for Zuul migration.Wed, May 27, 8:28 AM
MoritzMuehlenhoff mentioned this in T427353: Repurpose ganeti102[3456] for Zuul migration.
Maintenance_bot removed a project: Patch-For-Review.Wed, May 27, 8:30 AM
MoritzMuehlenhoff closed this task as Resolved.Wed, May 27, 9:36 AM

All done

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits