Page MenuHomePhabricator

Put cloudvirt10[77-80] in service
Closed, ResolvedPublic

Description

Hosts are provisioned and ready to go (T425088).

The one small caveat is that currently reimage must be coordinated as per below

Just reimaged cloudvirt1078, I think all hosts are ready now!

Please note: the provision/reimage changes to make this happen are not merged yet, I test-cookbooked them, so you cannot re-run immediately those if needed (ping me in case).

though I don't think we require reimage, nor should we.

Here's what I did yesterday to put cloudvirt1077 in service:

  • Make sure the host has no firewall (insetup_noferm) and reimage if needed
  • Allocate cloud-private v4/v6 addresses in the host rack vlan, specifically https://netbox.wikimedia.org/ipam/ip-addresses/23745/ and https://netbox.wikimedia.org/ipam/ip-addresses/23746/
  • Run the dns cookbook to propagate the names/addresses
  • The host can be transitioned to wmcs::openstack::eqiad1::virt_ceph role
  • Run puppet with the new role, reboot
  • Verify nova-compute is up and running
  • On any cloudcontrol host, put the host in service:
    • Allocate the host to a cell nova-manage cell_v2 discover_hosts --verbose
    • Add the host to ceph and network-ovs aggregates: wmcs-openstack aggregate add host ceph cloudvirt1077 wmcs-openstack aggregate add host network-ovs cloudvirt1077
    • Sync the aggregates (TBD if this happens periodically in the background) nova-manage placement sync_aggregates
    • Verify the hypervisor shows up: wmcs-openstack hypervisor list | grep 1077
    • Verify nova-compute service is enabled, VMs will be scheduled wmcs-openstack compute service list --host cloudvirt1077
    • Verify live migration works, pick a test instance, then its server ID with wmcs-server-id <fqdn> and (in this case for pontoon-demo-puppet-01.testlabs.eqiad1.wikimedia.cloud) then run wmcs-openstack server migrate --live --host cloudvirt1077 24e5b10a-80df-4bbc-807c-97d4e935d1f4 --wait and verify the operation is successful (no errors reported, VM stays up and moves to the new host)
  • Once the host is in service, hardcode its nova id (/etc/nova/compute_id) in puppet with a patch like https://gerrit.wikimedia.org/r/c/operations/puppet/+/1305668

Event Timeline

Change #1303962 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] Put cloudvirt10[77-80] in service

https://gerrit.wikimedia.org/r/1303962

Change #1303962 merged by Filippo Giunchedi:

[operations/puppet@production] Put cloudvirt10[77-80] in service

https://gerrit.wikimedia.org/r/1303962

Hosts are provisioned, though not yet on cloud-private. @cmooney has a cookbook in the testing phase which will help automate cloud hosts provisioning in their respective vlans ( https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1303397 ). My understanding is that hte missing bits right now are IP/DNS allocations; we can use this as a great opportunity to be beta-testers of the cookbook

Following up from the network syncup from yesterday: I'll be trying to provision cloudvirt1077 end to end and see what is required on the netbox side. Once that's established we'll evaluate what to do with the other hosts (cookbook automation or manual for now)

Change #1305598 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] site: put cloudvirt1077 in service

https://gerrit.wikimedia.org/r/1305598

Change #1305598 merged by Filippo Giunchedi:

[operations/puppet@production] site: put cloudvirt1077 in service

https://gerrit.wikimedia.org/r/1305598

Change #1305668 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: add nova id for cloudvirt1077

https://gerrit.wikimedia.org/r/1305668

Change #1305668 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: add nova id for cloudvirt1077

https://gerrit.wikimedia.org/r/1305668

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt1077.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1077.eqiad.wmnet with OS trixie completed:

  • cloudvirt1077 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606251319_filippo_2789129_cloudvirt1077.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Change #1305862 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] site: make sure new cloudvirts have no firewall

https://gerrit.wikimedia.org/r/1305862

Here's what I did yesterday to put cloudvirt1077 in service:

  • Make sure the host has no firewall (insetup_noferm) and reimage if needed
  • Allocate cloud-private v4/v6 addresses in the host rack vlan, specifically https://netbox.wikimedia.org/ipam/ip-addresses/23745/ and https://netbox.wikimedia.org/ipam/ip-addresses/23746/
  • Run the dns cookbook to propagate the names/addresses
  • The host can be transitioned to wmcs::openstack::eqiad1::virt_ceph role
  • Run puppet with the new role, reboot
  • Verify nova-compute is up and running
  • On any cloudcontrol host, put the host in service:
    • Allocate the host to a cell nova-manage cell_v2 discover_hosts --verbose
    • Add the host to ceph and network-ovs aggregates: wmcs-openstack aggregate add host ceph cloudvirt1077 wmcs-openstack aggregate add host network-ovs cloudvirt1077
    • Sync the aggregates (TBD if this happens periodically in the background) nova-manage placement sync_aggregates
    • Verify the hypervisor shows up: wmcs-openstack hypervisor list | grep 1077
    • Verify nova-compute service is enabled, VMs will be scheduled wmcs-openstack compute service list --host cloudvirt1077
    • Verify live migration works, pick a test instance, then its server ID with wmcs-server-id <fqdn> and (in this case for pontoon-demo-puppet-01.testlabs.eqiad1.wikimedia.cloud) then run wmcs-openstack server migrate --live --host cloudvirt1077 24e5b10a-80df-4bbc-807c-97d4e935d1f4 --wait and verify the operation is successful (no errors reported, VM stays up and moves to the new host)
  • Once the host is in service, hardcode its nova id (/etc/nova/compute_id) in puppet with a patch like https://gerrit.wikimedia.org/r/c/operations/puppet/+/1305668

Change #1305862 merged by Filippo Giunchedi:

[operations/puppet@production] site: make sure new cloudvirts have no firewall

https://gerrit.wikimedia.org/r/1305862

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt1078.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt1079.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt1080.eqiad.wmnet with OS trixie

I went ahead and added all v4/v6 for the remaining hosts and run the dns cookbook

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1078.eqiad.wmnet with OS trixie completed:

  • cloudvirt1078 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606261104_filippo_3540936_cloudvirt1078.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1079.eqiad.wmnet with OS trixie completed:

  • cloudvirt1079 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606261109_filippo_3540991_cloudvirt1079.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1080.eqiad.wmnet with OS trixie completed:

  • cloudvirt1080 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606261113_filippo_3541017_cloudvirt1080.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Change #1306159 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] site: put cloudvirt10[78-80] in service

https://gerrit.wikimedia.org/r/1306159

Change #1306159 merged by Filippo Giunchedi:

[operations/puppet@production] site: put cloudvirt10[78-80] in service

https://gerrit.wikimedia.org/r/1306159

Mentioned in SAL (#wikimedia-operations) [2026-06-29T12:28:09Z] <godog> add cloudvirt10[78-80] to nova -- with compute disabled - T429563

Change #1306289 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: add hypervisor IDs for cloudvirt10[78-80]

https://gerrit.wikimedia.org/r/1306289

Change #1306289 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: add hypervisor IDs for cloudvirt10[78-80]

https://gerrit.wikimedia.org/r/1306289

fgiunchedi claimed this task.

hosts are in service. I have updated https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Maintenance#new_cloudvirt_install with the instructions, manual for now