Page MenuHomePhabricator
Create Task
Maniphest T376594

Add ganeti2035 to ganeti2044 and decom ganeti2009 to ganeti2018
Closed, ResolvedPublic
Actions

Description

Add these:

  • ganeti2035 (row C)
  • ganeti2036 (row C)
  • ganeti2037 (row C)
  • ganeti2038 (row C)
  • ganeti2039 (row C)
  • ganeti2040 (row C)
  • ganeti2041 (row D)
  • ganeti2042 (row D) (broken CPU: T378358)
  • ganeti2043 (row D)
  • ganeti2044 (row D)

And eventually remove these from the ganeti cluster (hardware decom task listed separately)

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Oct 23 2024, 9:10 AM

Change #1082424 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add ganeti2039/ganeti2040 as Ganeti nodes

https://gerrit.wikimedia.org/r/1082424

gerritbot added a project: Patch-For-Review.Oct 23 2024, 9:10 AM
gerritbot added a comment.Oct 23 2024, 9:22 AM

Change #1082424 merged by Muehlenhoff:

[operations/puppet@production] Add ganeti2039/ganeti2040 as Ganeti nodes

https://gerrit.wikimedia.org/r/1082424

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2024, 9:31 AM
gerritbot added a comment.Oct 23 2024, 11:45 AM

Change #1082454 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Move ganeti2012 to insetup role

https://gerrit.wikimedia.org/r/1082454

gerritbot added a project: Patch-For-Review.Oct 23 2024, 11:45 AM
gerritbot added a comment.Oct 23 2024, 11:50 AM

Change #1082454 merged by Muehlenhoff:

[operations/puppet@production] Move ganeti2012 to insetup role

https://gerrit.wikimedia.org/r/1082454

gerritbot added a comment.Oct 23 2024, 12:18 PM

Change #1082458 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add ganeti2039/2040 to list of Ganeti nodes

https://gerrit.wikimedia.org/r/1082458

gerritbot added a comment.Oct 23 2024, 12:21 PM

Change #1082458 merged by Muehlenhoff:

[operations/puppet@production] Add ganeti2039/2040 to list of Ganeti nodes

https://gerrit.wikimedia.org/r/1082458

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2024, 12:31 PM
MoritzMuehlenhoff updated the task description. (Show Details)Oct 23 2024, 2:08 PM
gerritbot added a comment.Oct 24 2024, 7:26 AM

Change #1082691 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch ganeti2041-ganeti2044 to nftables

https://gerrit.wikimedia.org/r/1082691

gerritbot added a project: Patch-For-Review.Oct 24 2024, 7:26 AM
gerritbot added a comment.Oct 24 2024, 7:32 AM

Change #1082691 merged by Muehlenhoff:

[operations/puppet@production] Switch ganeti2041-ganeti2044 to nftables

https://gerrit.wikimedia.org/r/1082691

ops-monitoring-bot added a comment.Oct 24 2024, 7:57 AM

Draining ganeti2035.codfw.wmnet of running VMs

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2024, 8:30 AM
ops-monitoring-bot added a comment.Oct 24 2024, 10:56 AM

Draining ganeti2038.codfw.wmnet of running VMs

ops-monitoring-bot added a comment.Oct 24 2024, 12:13 PM

Draining ganeti2036.codfw.wmnet of running VMs

ops-monitoring-bot added a comment.Oct 24 2024, 1:42 PM

Draining ganeti2037.codfw.wmnet of running VMs

ops-monitoring-bot added a comment.Oct 24 2024, 2:08 PM

Draining ganeti2013.codfw.wmnet of running VMs

ops-monitoring-bot added a comment.Oct 24 2024, 2:20 PM

Draining ganeti2013.codfw.wmnet of running VMs

gerritbot added a comment.Oct 24 2024, 2:21 PM

Change #1082795 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti2013 from active Ganeti nodes

https://gerrit.wikimedia.org/r/1082795

gerritbot added a project: Patch-For-Review.Oct 24 2024, 2:21 PM
MoritzMuehlenhoff updated the task description. (Show Details)Oct 25 2024, 6:46 AM
ops-monitoring-bot added a comment.Oct 25 2024, 6:47 AM

Draining ganeti2014.codfw.wmnet of running VMs

gerritbot added a comment.Oct 25 2024, 6:54 AM

Change #1082795 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti2013 from active Ganeti nodes

https://gerrit.wikimedia.org/r/1082795

Maintenance_bot removed a project: Patch-For-Review.Oct 25 2024, 7:30 AM
ops-monitoring-bot added a comment.Oct 25 2024, 7:43 AM

VM kubestagemaster2004.codfw.wmnet switching disk type to drbd

ops-monitoring-bot added a comment.Oct 25 2024, 8:01 AM

VM kubestagemaster2004.codfw.wmnet switching disk type to plain

ops-monitoring-bot added a comment.Oct 25 2024, 8:03 AM

VM kubestagemaster2004.codfw.wmnet switching disk type to drbd

ops-monitoring-bot added a comment.Oct 25 2024, 8:21 AM

Draining ganeti2014.codfw.wmnet of running VMs

ops-monitoring-bot added a comment.Oct 25 2024, 8:23 AM

VM kubestagemaster2004.codfw.wmnet switching disk type to plain

ops-monitoring-bot added a comment.Oct 25 2024, 8:24 AM

Draining ganeti2014.codfw.wmnet of running VMs

MoritzMuehlenhoff updated the task description. (Show Details)Oct 25 2024, 8:40 AM
gerritbot added a comment.Oct 25 2024, 9:58 AM

Change #1083151 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti2014 from active Ganeti nodes

https://gerrit.wikimedia.org/r/1083151

gerritbot added a project: Patch-For-Review.Oct 25 2024, 9:58 AM
gerritbot added a comment.Oct 25 2024, 11:38 AM

Change #1083151 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti2014 from active Ganeti nodes

https://gerrit.wikimedia.org/r/1083151

MoritzMuehlenhoff updated the task description. (Show Details)Oct 25 2024, 12:16 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 25 2024, 12:30 PM
gerritbot added a comment.Oct 28 2024, 10:20 AM

Change #1083767 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Assign the ganeti role to ganeti2041/ganeti2042

https://gerrit.wikimedia.org/r/1083767

gerritbot added a project: Patch-For-Review.Oct 28 2024, 10:20 AM

Change #1083767 merged by Muehlenhoff:

[operations/puppet@production] Assign the ganeti role to ganeti2041/ganeti2042

https://gerrit.wikimedia.org/r/1083767

Maintenance_bot removed a project: Patch-For-Review.Oct 28 2024, 10:30 AM
MoritzMuehlenhoff updated the task description. (Show Details)Oct 28 2024, 1:12 PM
ops-monitoring-bot added a comment.Oct 29 2024, 10:05 AM

Draining ganeti2015.codfw.wmnet of running VMs

ops-monitoring-bot added a comment.Oct 29 2024, 10:15 AM

Draining ganeti2015.codfw.wmnet of running VMs

gerritbot added a comment.Oct 29 2024, 10:16 AM

Change #1084046 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti role from ganeti2015

https://gerrit.wikimedia.org/r/1084046

gerritbot added a project: Patch-For-Review.Oct 29 2024, 10:16 AM
MoritzMuehlenhoff updated the task description. (Show Details)Oct 29 2024, 12:18 PM
MoritzMuehlenhoff updated the task description. (Show Details)
gerritbot added a comment.Oct 29 2024, 12:19 PM

Change #1084046 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti role from ganeti2015

https://gerrit.wikimedia.org/r/1084046

Maintenance_bot removed a project: Patch-For-Review.Oct 29 2024, 12:30 PM
gerritbot added a comment.Oct 30 2024, 8:03 AM

Change #1084705 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Apply the ganeti role to ganeti204[34]

https://gerrit.wikimedia.org/r/1084705

gerritbot added a project: Patch-For-Review.Oct 30 2024, 8:03 AM
gerritbot added a comment.Oct 30 2024, 8:26 AM

Change #1084705 merged by Muehlenhoff:

[operations/puppet@production] Apply the ganeti role to ganeti204[34]

https://gerrit.wikimedia.org/r/1084705

Maintenance_bot removed a project: Patch-For-Review.Oct 30 2024, 8:31 AM
MoritzMuehlenhoff updated the task description. (Show Details)Oct 30 2024, 11:03 AM
ops-monitoring-bot added a comment.Oct 30 2024, 11:19 AM

Draining ganeti2016.codfw.wmnet of running VMs

ops-monitoring-bot added a comment.Oct 30 2024, 11:26 AM

VM ml-etcd2003.codfw.wmnet switching disk type to drbd

ops-monitoring-bot added a comment.Oct 30 2024, 11:37 AM

Draining ganeti2016.codfw.wmnet of running VMs

ops-monitoring-bot added a comment.Oct 30 2024, 11:39 AM

VM ml-etcd2003.codfw.wmnet switching disk type to plain

ops-monitoring-bot added a comment.Oct 30 2024, 11:40 AM

Draining ganeti2016.codfw.wmnet of running VMs

gerritbot added a comment.Oct 30 2024, 12:11 PM

Change #1084749 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti role from ganeti2016

https://gerrit.wikimedia.org/r/1084749

gerritbot added a project: Patch-For-Review.Oct 30 2024, 12:11 PM
gerritbot added a comment.Oct 30 2024, 2:48 PM

Change #1084749 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti role from ganeti2016

https://gerrit.wikimedia.org/r/1084749

MoritzMuehlenhoff updated the task description. (Show Details)Oct 30 2024, 2:49 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 30 2024, 3:30 PM
ops-monitoring-bot added a comment.Nov 4 2024, 8:11 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti2013.codfw.wmnet

  • ganeti2013.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Nov 4 2024, 8:23 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti2014.codfw.wmnet

  • ganeti2014.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
MoritzMuehlenhoff updated the task description. (Show Details)Nov 4 2024, 8:23 AM
MoritzMuehlenhoff updated the task description. (Show Details)Nov 8 2024, 10:54 AM
ops-monitoring-bot added a comment.Nov 14 2024, 7:34 AM

Draining ganeti2017.codfw.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 14 2024, 7:42 AM

Draining ganeti2017.codfw.wmnet of running VMs

Stashbot added a comment.Nov 14 2024, 10:16 AM

Mentioned in SAL (#wikimedia-operations) [2024-11-14T10:16:48Z] <moritzm> remove ganeti2017 from active ganeti nodes T376594

MoritzMuehlenhoff updated the task description. (Show Details)Nov 15 2024, 9:19 AM
ops-monitoring-bot added a comment.Dec 4 2024, 8:13 AM

Draining ganeti2018.codfw.wmnet of running VMs

ops-monitoring-bot added a comment.Dec 4 2024, 8:23 AM

Draining ganeti2018.codfw.wmnet of running VMs

Stashbot added a comment.Dec 4 2024, 10:33 AM

Mentioned in SAL (#wikimedia-operations) [2024-12-04T10:33:13Z] <moritzm> removing ganeti2018 from active Ganeti nodes T376594

MoritzMuehlenhoff updated the task description. (Show Details)Dec 4 2024, 10:33 AM
gerritbot added a comment.Dec 4 2024, 10:38 AM

Change #1100414 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] ganeti2018: Update site.pp

https://gerrit.wikimedia.org/r/1100414

gerritbot added a project: Patch-For-Review.Dec 4 2024, 10:38 AM
gerritbot added a comment.Dec 4 2024, 11:41 AM

Change #1100414 merged by Muehlenhoff:

[operations/puppet@production] ganeti2018: Update site.pp

https://gerrit.wikimedia.org/r/1100414

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2024, 12:32 PM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 9 2024, 10:08 AM
Stashbot added a comment.Dec 9 2024, 10:10 AM

Mentioned in SAL (#wikimedia-operations) [2024-12-09T10:10:34Z] <moritzm> rebalance Ganeti cluster in codfw/A following server refresh T376594

Stashbot added a comment.Dec 9 2024, 4:12 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-09T16:12:33Z] <moritzm> rebalance Ganeti cluster in codfw/B following server refresh T376594

Stashbot added a comment.Dec 10 2024, 9:34 AM

Mentioned in SAL (#wikimedia-operations) [2024-12-10T09:34:25Z] <moritzm> rebalance Ganeti cluster in codfw/c following server refresh T376594

Stashbot added a comment.Dec 12 2024, 1:32 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-12T13:32:53Z] <moritzm> rebalance Ganeti cluster in codfw/D following server refresh T376594

MoritzMuehlenhoff updated the task description. (Show Details)Dec 13 2024, 8:33 AM
gerritbot added a comment.Dec 13 2024, 8:45 AM

Change #1103255 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti2018 from cluster list

https://gerrit.wikimedia.org/r/1103255

gerritbot added a project: Patch-For-Review.Dec 13 2024, 8:45 AM

Change #1103255 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti2018 from cluster list

https://gerrit.wikimedia.org/r/1103255

MoritzMuehlenhoff closed this task as Resolved.Dec 13 2024, 9:42 AM

All new servers added, all old server decommissioned and clusters rebalanced.

Maintenance_bot removed a project: Patch-For-Review.Dec 13 2024, 10:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits