Page MenuHomePhabricator
Create Task
Maniphest T365650

Q4:rack/setup/install ganeti1039 to ganeti1052
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of ganeti1039 to ganeti1052

Hostname / Racking / Installation Details

Hostnames: ganeti1039 to ganeti1052
Racking Proposal: These are replacing four nodes in C, so these need to get to C again. And four in D, which also need to go in D. And six in B, which again need to go to B.
Networking Setup: 10G - VLAN setup like existing Ganeti nodes
Partitioning/Raid: partman/custom/ganeti-raid5.cfg
OS Distro: Bookworm
Sub-team Technical Contact: Moritz

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

ganeti1039
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1040
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1041
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1042
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1043
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1044
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1045
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1046
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1047
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1048
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1049
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1050
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1051
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti1052
  • Receive in system on procurement task T361357 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

SubjectRepoBranchLines +/-
Add ganeti1039 - ganeti1052 to site.ppoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedMoritzMuehlenhoffT365650 Q4:rack/setup/install ganeti1039 to ganeti1052

Event Timeline

RobH created this task.May 22 2024, 9:09 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.
Maintenance_bot added a project: SRE.May 22 2024, 9:29 PM
VRiley-WMF updated the task description. (Show Details)Aug 19 2024, 11:17 AM
Jclark-ctr added subscribers: • MMuhlenhoff, Jclark-ctr.Sep 5 2024, 4:02 PM

@MMuhlenhoff is there any reason we can not rack these in row e/f ?

Jclark-ctr added a comment.Sep 6 2024, 3:44 PM

ganeti1039
b2
u4
cableid 4893
port 2

VRiley-WMF subscribed.Sep 10 2024, 1:26 PM

ganeti1039
B2
U4
CableID 4893
Port 3

ganeti1040
B2
U15
CableID 5005
Port 29

ganeti1041
B2
U16
CableID 20220202
Port 28

ganeti1042
B4
U4
CableID 4880
Port 33

ganeti1043
B4
U8
CableID 4778
Port 35

ganeti1044
B4
U12
CableID 5017
Port 9

ganeti1045
C4
U7
CableID 1972
Port 15

ganeti1046
C4
U41
CableID 20220173
Port 3p

ganeti1047
C7
U3
CableID 20220163
Port 21

ganeti1048
C7
U28
CableID 20220184
Port 19

ganeti1049
D2
U31
CableID4907
Port 11

ganeti1050
D2
U32
CableID 4882
Port 7

ganeti1051
D7
U17
CableID 20220168
Port 32

ganeti1052
D7
U18
CableID 20220167
Port 33

VRiley-WMF updated the task description. (Show Details)Sep 10 2024, 1:27 PM
ops-monitoring-bot added a comment.Sep 11 2024, 10:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1052.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 11 2024, 10:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1041.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 11 2024, 11:13 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1052.eqiad.wmnet with OS bookworm executed with errors:

  • ganeti1052 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console ganeti1052.eqiad.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 20 2024, 11:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1040.eqiad.wmnet with OS bookworm

Jclark-ctr added a subscriber: MoritzMuehlenhoff.Sep 20 2024, 11:25 PM

@MoritzMuehlenhoff can you update puppet site.pp is missing these servers. also please verify preseed.yaml is updated
Thanks

gerritbot added a comment.Sep 23 2024, 9:40 AM

Change #1074957 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add ganeti1039 - ganeti1052 to site.pp

https://gerrit.wikimedia.org/r/1074957

gerritbot added a project: Patch-For-Review.Sep 23 2024, 9:40 AM

Change #1074957 merged by Muehlenhoff:

[operations/puppet@production] Add ganeti1039 - ganeti1052 to site.pp

https://gerrit.wikimedia.org/r/1074957

MoritzMuehlenhoff added a comment.Sep 23 2024, 10:00 AM
In T365650#10165298, @Jclark-ctr wrote:

@MoritzMuehlenhoff can you update puppet site.pp is missing these servers. also please verify preseed.yaml is updated
Thanks

Sure thing! site.pp has been updated, the preseed config already covered the new hostnames.

Maintenance_bot removed a project: Patch-For-Review.Sep 23 2024, 10:30 AM
ops-monitoring-bot added a comment.Sep 23 2024, 8:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1039.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1040.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 9:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1041.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 9:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1046.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 9:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1045.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 10:24 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1045.eqiad.wmnet with OS bookworm completed:

  • ganeti1045 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409232209_jclark_1592007_ganeti1045.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 23 2024, 10:28 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1046.eqiad.wmnet with OS bookworm completed:

  • ganeti1046 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409232213_jclark_1591421_ganeti1046.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 23 2024, 10:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1048.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 10:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1039.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 10:32 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1041.eqiad.wmnet with OS bookworm completed:

  • ganeti1041 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409232217_jclark_1586647_ganeti1041.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 23 2024, 10:37 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1040.eqiad.wmnet with OS bookworm completed:

  • ganeti1040 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409232221_jclark_1585508_ganeti1040.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 23 2024, 10:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1047.eqiad.wmnet with OS bookworm

Jclark-ctr updated the task description. (Show Details)Sep 23 2024, 10:43 PM
ops-monitoring-bot added a comment.Sep 23 2024, 10:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1049.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 11:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1050.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 11:05 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1048.eqiad.wmnet with OS bookworm completed:

  • ganeti1048 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409232249_jclark_1608708_ganeti1048.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Sep 23 2024, 11:05 PM
ops-monitoring-bot added a comment.Sep 23 2024, 11:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1052.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 11:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1051.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 23 2024, 11:14 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1047.eqiad.wmnet with OS bookworm completed:

  • ganeti1047 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409232257_jclark_1609787_ganeti1047.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Sep 23 2024, 11:15 PM
ops-monitoring-bot added a comment.Sep 23 2024, 11:33 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1049.eqiad.wmnet with OS bookworm completed:

  • ganeti1049 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409232314_jclark_1611173_ganeti1049.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Sep 23 2024, 11:34 PM
ops-monitoring-bot added a comment.Sep 23 2024, 11:34 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1039.eqiad.wmnet with OS bookworm completed:

  • ganeti1039 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409232317_jclark_1608810_ganeti1039.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Sep 23 2024, 11:35 PM
ops-monitoring-bot added a comment.Sep 23 2024, 11:43 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1050.eqiad.wmnet with OS bookworm completed:

  • ganeti1050 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409232321_jclark_1613437_ganeti1050.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 24 2024, 12:15 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1042.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 24 2024, 12:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1043.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 24 2024, 12:23 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1051.eqiad.wmnet with OS bookworm completed:

  • ganeti1051 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409240008_jclark_1616610_ganeti1051.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 24 2024, 12:52 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1042.eqiad.wmnet with OS bookworm completed:

  • ganeti1042 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409240032_jclark_1631094_ganeti1042.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Sep 24 2024, 12:56 AM
Jclark-ctr updated the task description. (Show Details)
ops-monitoring-bot added a comment.Sep 24 2024, 12:58 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1043.eqiad.wmnet with OS bookworm completed:

  • ganeti1043 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409240043_jclark_1633521_ganeti1043.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Sep 24 2024, 12:59 AM
Jclark-ctr updated the task description. (Show Details)
ops-monitoring-bot added a comment.Sep 24 2024, 1:08 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1052.eqiad.wmnet with OS bookworm completed:

  • ganeti1052 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409232327_jclark_1615964_ganeti1052.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Sep 24 2024, 2:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ganeti1044.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 24 2024, 2:47 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ganeti1044.eqiad.wmnet with OS bookworm completed:

  • ganeti1044 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409241423_jclark_2264872_ganeti1044.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
elukey subscribed.Oct 30 2024, 7:55 AM

Hey folks! I've uploaded all the Redfish licenses to these ganeti nodes, and ran provision again up to ganeti1043. I tried 1044 but it seems that it still needs to run the first provision run, and I don't have the custom BMC password, so I cannot proceed. Feel free to go ahead and report issues if any!

elukey added a comment.Oct 31 2024, 9:39 AM

Found the issue with 1044 (see T376121#10280005), I'll post an update as soon as Supermicro replies with the correct license.

elukey added a comment.EditedNov 1 2024, 1:36 PM

@VRiley-WMF @Jclark-ctr I fixed the issue with ganeti1044 and ran provision, all good! The rest of the nodes should be fine as well :)

Edit: spoke too soon, I still have an issue configuring the mgmt interface, will post an update when I sort out this remaining issue..

elukey added a comment.Nov 4 2024, 9:00 AM

Fixed 1044. For some reason IPv6 support was disabled, so our settings like IPv6AutoConfigEnabled: False led to a HTTP 400. I connected to the Web UI, turned IPv6 on and re-run provision, all good. I've also set the host's status to Active in Netbox.

@VRiley-WMF @Jclark-ctr I think that this batch of Supermicro nodes should be ok now, please recheck everything and lemme know if anything is missing :)

Jclark-ctr added a comment.Nov 5 2024, 11:42 PM

I have gone through and rerun all provision scripts on these i believe this is good to close @elukey

MoritzMuehlenhoff added a comment.Nov 6 2024, 9:56 AM
In T365650#10287518, @elukey wrote:

Fixed 1044. For some reason IPv6 support was disabled, so our settings like IPv6AutoConfigEnabled: False led to a HTTP 400. I connected to the Web UI, turned IPv6 on and re-run provision, all good. I've also set the host's status to Active in Netbox.

@VRiley-WMF @Jclark-ctr I think that this batch of Supermicro nodes should be ok now, please recheck everything and lemme know if anything is missing :)

Something is still missing with the provisioning of this host, it's missing the /dev/kvm device, maybe this got lost in the last re-reprovisioning?

MoritzMuehlenhoff added a comment.Nov 6 2024, 11:06 AM

I had been running into issues with moving VMs to ganeti1041 this morning (which is already added to the Ganeti cluster) and after debugging various OS-level aspects I finally realised that ganeti1041 also lost /dev/kvm? Was it also re-re-reprovisioned? It's not mentioned on this task at all.

Please don't run the reprovision cookbook on any of ganeti1039-1052 before checking with me beforehand! This has an impact on running VMs, If the cookbook had been re-run on 1039 we'd have lost one of the Dragonfly nodes, e.g.

ops-monitoring-bot added a comment.Nov 6 2024, 11:08 AM

Draining ganeti1041.eqiad.wmnet of running VMs

Jclark-ctr added a comment.Nov 6 2024, 8:53 PM

@MoritzMuehlenhoff these where not handed over to service owner while. Luca and dceng researched License /provisioning issue. The reprovisioned this was brought up in irc last week while luca was troubleshooting supermicro licenses and

Wednesday, October 30th, 2024
4:09 AM <elukey> luca
VRiley, jclark-ctr: o/ - ganeti1039->1052 should have the supermicro license uploaded, so you are free to run provision on those. I went up to 1043 since IIUC provision ran at least once on those, but after (1044+) I got some auth errors since I suspect I'd need the server's label password to proceed.

MoritzMuehlenhoff closed this task as Resolved.Nov 7 2024, 11:32 AM
MoritzMuehlenhoff claimed this task.
In T365650#10297670, @Jclark-ctr wrote:

@MoritzMuehlenhoff these where not handed over to service owner while. Luca and dceng researched License /provisioning issue. The reprovisioned this was brought up in irc last week while luca was troubleshooting supermicro licenses and

Ok! These are now partly moved into production and the racking is complete. Let's close this task and if there should be a need for any other DC ops step, then please ping me so that I can drain live nodes off the servers

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits