Page MenuHomePhabricator
Create Task
Maniphest T378921

Add ganeti1039 to ganeti1052 and decom ganeti1009 to ganeti1022
Closed, ResolvedPublic
Actions

Description

Add these:

  • ganeti1039 (row B)
  • ganeti1040 (row B)
  • ganeti1041 (row B)
  • ganeti1042 (row B)
  • ganeti1043 (row B)
  • ganeti1044 (row B)
  • ganeti1045 (row C)
  • ganeti1046 (row C)
  • ganeti1047 (row C)
  • ganeti1048 (row C)
  • ganeti1049 (row D)
  • ganeti1050 (row D)
  • ganeti1051 (row D)
  • ganeti1052 (row D)

And eventually remove these from the ganeti cluster (hardware decom task listed separately)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update site.pp for ganeti1016operations/puppetproduction+2 -3
Update site.ppoperations/puppetproduction+2 -6
Add ganeti1051/ganeti1052operations/puppetproduction+3 -5
Add ganeti1049/ganeti1050 as Ganeti serversoperations/puppetproduction+4 -2
Remove ganeti role from to-be-decommed serversoperations/puppetproduction+6 -1
Add ganeti1047/1048 as Ganeti nodesoperations/puppetproduction+4 -2
Add ganeti1045/ganeti1046 as Ganeti nodesoperations/puppetproduction+4 -2
Add the ganeti role to ganeti1043/ganeti1044operations/puppetproduction+4 -2
Assign ganeti role for ganeti1041/ganeti1042operations/puppetproduction+4 -2
Assign ganeti role to ganeti1039/ganeti1040operations/puppetproduction+5 -3
Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Nov 7 2024, 11:09 AM

Draining ganeti1010.eqiad.wmnet of running VMs

gerritbot added a comment.Nov 7 2024, 12:06 PM

Change #1088246 merged by Muehlenhoff:

[operations/puppet@production] Add ganeti1047/1048 as Ganeti nodes

https://gerrit.wikimedia.org/r/1088246

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2024, 12:31 PM
Stashbot added a comment.Nov 7 2024, 3:54 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-07T15:54:53Z] <moritzm> remove ganeti1010 from active ganeti nodes T378921

MoritzMuehlenhoff updated the task description. (Show Details)Nov 7 2024, 3:55 PM
gerritbot added a comment.Nov 7 2024, 3:59 PM

Change #1088325 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti role from to-be-decommed servers

https://gerrit.wikimedia.org/r/1088325

gerritbot added a project: Patch-For-Review.Nov 7 2024, 3:59 PM
• herron mentioned this in T378989: eqiad: (2x) aux-k8s-worker nodes.Nov 7 2024, 7:17 PM
MoritzMuehlenhoff updated the task description. (Show Details)Nov 8 2024, 7:48 AM
ops-monitoring-bot added a comment.Nov 8 2024, 10:16 AM

Draining ganeti1011.eqiad.wmnet of running VMs

gerritbot added a comment.Nov 11 2024, 7:43 AM

Change #1088325 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti role from to-be-decommed servers

https://gerrit.wikimedia.org/r/1088325

ops-monitoring-bot added a comment.Nov 11 2024, 7:52 AM

Draining ganeti1011.eqiad.wmnet of running VMs

MoritzMuehlenhoff updated the task description. (Show Details)Nov 11 2024, 8:13 AM
Maintenance_bot removed a project: Patch-For-Review.Nov 11 2024, 8:30 AM
Stashbot added a comment.Nov 11 2024, 9:10 AM

Mentioned in SAL (#wikimedia-operations) [2024-11-11T09:10:11Z] <moritzm> remove ganeti1011 from active ganeti nodes T378921

gerritbot added a comment.Nov 11 2024, 9:35 AM

Change #1089665 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add ganeti1049/ganeti1050 as Ganeti servers

https://gerrit.wikimedia.org/r/1089665

gerritbot added a project: Patch-For-Review.Nov 11 2024, 9:35 AM
gerritbot added a comment.Nov 11 2024, 11:25 AM

Change #1089665 merged by Muehlenhoff:

[operations/puppet@production] Add ganeti1049/ganeti1050 as Ganeti servers

https://gerrit.wikimedia.org/r/1089665

Maintenance_bot removed a project: Patch-For-Review.Nov 11 2024, 11:30 AM
MoritzMuehlenhoff updated the task description. (Show Details)Nov 11 2024, 1:22 PM
ops-monitoring-bot added a comment.Nov 12 2024, 8:17 AM

Draining ganeti1009.eqiad.wmnet of running VMs

MoritzMuehlenhoff updated the task description. (Show Details)Nov 12 2024, 8:23 AM
ops-monitoring-bot added a comment.Nov 12 2024, 8:25 AM

Draining ganeti1015.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 12 2024, 8:32 AM

Draining ganeti1015.eqiad.wmnet of running VMs

MoritzMuehlenhoff updated the task description. (Show Details)Nov 12 2024, 11:47 AM
Stashbot added a comment.Nov 12 2024, 12:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-12T12:09:01Z] <moritzm> remove ganeti1015 from active ganeti nodes T378921

MoritzMuehlenhoff updated the task description. (Show Details)Nov 12 2024, 12:09 PM
ops-monitoring-bot added a comment.Nov 12 2024, 12:25 PM

Draining ganeti1012.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 12 2024, 12:35 PM

VM ml-etcd1002.eqiad.wmnet switching disk type to drbd

ops-monitoring-bot added a comment.Nov 12 2024, 12:52 PM

Draining ganeti1012.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 12 2024, 12:54 PM

VM ml-etcd1002.eqiad.wmnet switching disk type to plain

ops-monitoring-bot added a comment.Nov 12 2024, 1:00 PM

VM dse-k8s-etcd1003.eqiad.wmnet switching disk type to drbd

ops-monitoring-bot added a comment.Nov 12 2024, 1:11 PM

Draining ganeti1012.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 12 2024, 1:15 PM

VM dse-k8s-etcd1003.eqiad.wmnet switching disk type to plain

ops-monitoring-bot added a comment.Nov 12 2024, 1:21 PM

Draining ganeti1012.eqiad.wmnet of running VMs

MoritzMuehlenhoff updated the task description. (Show Details)Nov 12 2024, 1:47 PM
Stashbot added a comment.Nov 12 2024, 6:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-12T18:01:43Z] <moritzm> remove ganeti1012 from active ganeti nodes T378921

MoritzMuehlenhoff updated the task description. (Show Details)Nov 12 2024, 6:04 PM
gerritbot added a comment.Nov 13 2024, 11:13 AM

Change #1090821 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add ganeti1051/ganeti1052

https://gerrit.wikimedia.org/r/1090821

gerritbot added a project: Patch-For-Review.Nov 13 2024, 11:13 AM
gerritbot added a comment.Nov 13 2024, 11:24 AM

Change #1090821 merged by Muehlenhoff:

[operations/puppet@production] Add ganeti1051/ganeti1052

https://gerrit.wikimedia.org/r/1090821

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2024, 11:31 AM
MoritzMuehlenhoff updated the task description. (Show Details)Nov 13 2024, 2:05 PM
MoritzMuehlenhoff updated the task description. (Show Details)Nov 13 2024, 2:43 PM
ops-monitoring-bot added a comment.Nov 18 2024, 7:48 AM

Draining ganeti1020.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 18 2024, 7:52 AM

Draining ganeti1020.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 18 2024, 7:56 AM

Draining ganeti1021.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 18 2024, 8:01 AM

Draining ganeti1021.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 18 2024, 8:05 AM

Draining ganeti1021.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 18 2024, 8:07 AM

Draining ganeti1021.eqiad.wmnet of running VMs

Stashbot added a comment.Nov 18 2024, 1:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-18T13:01:58Z] <moritzm> removing ganeti1021 from active Ganeti nodes T378921

gerritbot added a comment.Nov 18 2024, 1:42 PM

Change #1092250 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update site.pp

https://gerrit.wikimedia.org/r/1092250

gerritbot added a project: Patch-For-Review.Nov 18 2024, 1:42 PM
gerritbot added a comment.Nov 18 2024, 4:12 PM

Change #1092250 merged by Muehlenhoff:

[operations/puppet@production] Update site.pp

https://gerrit.wikimedia.org/r/1092250

MoritzMuehlenhoff updated the task description. (Show Details)Nov 19 2024, 7:22 AM
ops-monitoring-bot added a comment.Nov 19 2024, 7:24 AM

Draining ganeti1016.eqiad.wmnet of running VMs

Maintenance_bot removed a project: Patch-For-Review.Nov 19 2024, 7:31 AM
ops-monitoring-bot added a comment.Nov 19 2024, 7:32 AM

Draining ganeti1016.eqiad.wmnet of running VMs

Stashbot added a comment.Nov 19 2024, 12:35 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-19T12:35:02Z] <moritzm> removing ganeti1016 from active Ganeti nodes T378921

gerritbot added a comment.Nov 19 2024, 12:36 PM

Change #1092819 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update site.pp for ganeti1016

https://gerrit.wikimedia.org/r/1092819

gerritbot added a project: Patch-For-Review.Nov 19 2024, 12:36 PM
MoritzMuehlenhoff updated the task description. (Show Details)Nov 19 2024, 12:36 PM
gerritbot added a comment.Nov 19 2024, 12:40 PM

Change #1092819 merged by Muehlenhoff:

[operations/puppet@production] Update site.pp for ganeti1016

https://gerrit.wikimedia.org/r/1092819

Maintenance_bot removed a project: Patch-For-Review.Nov 19 2024, 1:30 PM
ops-monitoring-bot added a comment.Nov 20 2024, 12:49 PM

Draining ganeti1017.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 20 2024, 1:01 PM

Draining ganeti1017.eqiad.wmnet of running VMs

Stashbot added a comment.Nov 21 2024, 7:48 AM

Mentioned in SAL (#wikimedia-operations) [2024-11-21T07:48:31Z] <moritzm> removing ganeti1017 from active Ganeti nodes T378921

MoritzMuehlenhoff updated the task description. (Show Details)Nov 21 2024, 7:49 AM
MoritzMuehlenhoff updated the task description. (Show Details)Nov 22 2024, 10:11 AM
ops-monitoring-bot added a comment.Nov 26 2024, 3:20 PM

Draining ganeti1020.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 26 2024, 3:22 PM

Draining ganeti1020.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 27 2024, 2:39 PM

Draining ganeti1022.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 27 2024, 2:51 PM

VM ml-etcd1003.eqiad.wmnet switching disk type to drbd

ops-monitoring-bot added a comment.Nov 27 2024, 3:02 PM

Draining ganeti1022.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 27 2024, 3:03 PM

VM ml-etcd1003.eqiad.wmnet switching disk type to plain

ops-monitoring-bot added a comment.Nov 28 2024, 7:56 AM

Draining ganeti1022.eqiad.wmnet of running VMs

Stashbot added a comment.Nov 28 2024, 11:11 AM

Mentioned in SAL (#wikimedia-operations) [2024-11-28T11:11:29Z] <moritzm> removing ganeti1022 from active Ganeti nodes T378921

MoritzMuehlenhoff updated the task description. (Show Details)Nov 28 2024, 11:12 AM
ops-monitoring-bot added a comment.Nov 28 2024, 11:14 AM

Draining ganeti1018.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Nov 28 2024, 11:32 AM

Draining ganeti1018.eqiad.wmnet of running VMs

Stashbot added a comment.Nov 28 2024, 3:21 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-28T15:21:38Z] <moritzm> removing ganeti1018 from active Ganeti nodes T378921

MoritzMuehlenhoff updated the task description. (Show Details)Nov 29 2024, 12:21 PM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 2 2024, 7:56 AM
ops-monitoring-bot added a comment.Dec 2 2024, 8:07 AM

Draining ganeti1009.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Dec 2 2024, 8:11 AM

Draining ganeti1020.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Dec 2 2024, 9:13 AM

Draining ganeti1020.eqiad.wmnet of running VMs

Stashbot added a comment.Dec 2 2024, 2:00 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-02T14:00:58Z] <moritzm> removing ganeti1020 from active Ganeti nodes T378921

ops-monitoring-bot added a comment.Dec 3 2024, 7:25 AM

Draining ganeti1009.eqiad.wmnet of running VMs

ops-monitoring-bot added a comment.Dec 3 2024, 7:28 AM

Draining ganeti1009.eqiad.wmnet of running VMs

Stashbot added a comment.Dec 3 2024, 9:24 AM

Mentioned in SAL (#wikimedia-operations) [2024-12-03T09:24:47Z] <moritzm> removing ganeti1009 from active Ganeti nodes T378921

MoritzMuehlenhoff updated the task description. (Show Details)Dec 3 2024, 12:46 PM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 3 2024, 12:48 PM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 6 2024, 11:45 AM
MoritzMuehlenhoff closed this task as Resolved.Dec 6 2024, 2:34 PM

All new servers added, all old server decommissioned and clusters rebalanced.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits