Page MenuHomePhabricator
Create Task
Maniphest T355343

Q3:rack/setup/install es[2035-2040]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of es[2035-2040]

Hostname / Racking / Installation Details

This section should list the racking restrictions for these hosts. If they shouldn't share a rack/row with one another, or any existing hosts. This section should also list the other details listed below.

Hostnames: es2035 es2036 es2037 es2038 es2039 es2040
Racking Proposal: Different racks and rows if possible.
Networking Setup: # of Connections:1 - Speed:1G. - VLAN:Private : AAAA records:N, Additional IP records (Cassandra)? No
Partitioning/Raid: HW Raid: Y, Partman recipe and/or desired Raid Level: Hardware Raid level 10 - @Marostegui will assign the hosts to the correct recipe in puppet
OS Distro: Bookworm
Sub-team Technical Contact: @Marostegui

Quote Review

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

es2035:
  • Receive in system on procurement task T354674 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2036:
  • Receive in system on procurement task T354674 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2037:
  • Receive in system on procurement task T354674 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2038:
  • Receive in system on procurement task T354674 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2039:
  • Receive in system on procurement task T354674 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2040:
  • Receive in system on procurement task T354674 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedVolansT355343 Q3:rack/setup/install es[2035-2040]
 ResolvedMarosteguiT355424 Productionize es[2035-2040]

Event Timeline

Papaul created this task.Jan 18 2024, 4:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 18 2024, 4:29 PM
Papaul updated the task description. (Show Details)Jan 18 2024, 4:30 PM
Marostegui added a comment.Jan 18 2024, 4:30 PM

@Papaul all the puppet changes are in place :-)

Papaul added a parent task: Unknown Object (Task).Jan 18 2024, 4:31 PM

@Marostegui thank you

Papaul updated the task description. (Show Details)Jan 18 2024, 4:33 PM
Papaul mentioned this in Unknown Object (Task).Jan 18 2024, 4:47 PM
Maintenance_bot added a project: SRE.Jan 18 2024, 5:29 PM
Marostegui changed the status of subtask T355424: Productionize es[2035-2040] from Open to Stalled.Jan 19 2024, 1:39 PM
Papaul moved this task from Backlog to Racking Tasks on the ops-codfw board.Jan 22 2024, 3:41 PM
Jhancock.wm updated the task description. (Show Details)Feb 20 2024, 3:21 PM
Marostegui added a subscriber: Jhancock.wm.Feb 27 2024, 8:50 AM

@Jhancock.wm when installing these hosts, when asked about which puppet version you want, go for 5 instead of 7

Marostegui mentioned this in T358594: Remove IPV6 dns records from new database hosts.Feb 27 2024, 2:50 PM
Jhancock.wm updated the task description. (Show Details)Feb 29 2024, 2:32 PM
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm updated the task description. (Show Details)Mar 1 2024, 11:24 PM
Jhancock.wm updated the task description. (Show Details)Mar 4 2024, 6:24 PM
ops-monitoring-bot added a comment.Mar 4 2024, 11:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2035.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 4 2024, 11:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2036.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 4 2024, 11:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2037.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 4 2024, 11:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2038.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 4 2024, 11:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2039.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 4 2024, 11:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2040.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 5 2024, 12:34 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2036.codfw.wmnet with OS bookworm completed:

  • es2036 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403050015_jhancock_2643157_es2036.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 5 2024, 12:38 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2039.codfw.wmnet with OS bookworm completed:

  • es2039 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403050020_jhancock_2643176_es2039.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 5 2024, 12:40 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2038.codfw.wmnet with OS bookworm completed:

  • es2038 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403050022_jhancock_2643110_es2038.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 5 2024, 12:42 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2037.codfw.wmnet with OS bookworm completed:

  • es2037 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403050025_jhancock_2643171_es2037.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 5 2024, 1:10 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2035.codfw.wmnet with OS bookworm executed with errors:

  • es2035 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" es2035.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 5 2024, 3:43 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2040.codfw.wmnet with OS bookworm completed:

  • es2040 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403050033_jhancock_2643197_es2040.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Jhancock.wm updated the task description. (Show Details)Mar 5 2024, 3:44 PM
ops-monitoring-bot added a comment.Mar 5 2024, 3:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2035.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 5 2024, 4:47 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2035.codfw.wmnet with OS bookworm completed:

  • es2035 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403051627_jhancock_3318644_es2035.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Jhancock.wm closed this task as Resolved.Mar 5 2024, 4:48 PM
Jhancock.wm updated the task description. (Show Details)

@Marostegui this is completed

Marostegui added a comment.Mar 5 2024, 4:53 PM

Thank you so much!

Marostegui changed the status of subtask T355424: Productionize es[2035-2040] from Stalled to Open.Mar 5 2024, 4:55 PM
Volans reopened this task as Open.Mar 5 2024, 8:42 PM
Volans claimed this task.
Volans subscribed.

Re-opening as AAAA records were erroneously added to the hosts (AAAA records:N). I'll remove them programmatically.

Volans closed this task as Resolved.Mar 5 2024, 8:58 PM

Got the list of affected hosts with nodeset -S '","' -e "es20[35-40]" on a cumin host, then I run the following code on Netbox:

>>> import uuid
>>> request_id = uuid.uuid4()
>>> user = User.objects.get(username='volans')
>>> def update(d):
...     ip = d.primary_ip6
...     ip.dns_name = ""
...     ip.save()
...     log = ip.to_objectchange('update')
...     log.request_id = request_id
...     log.user = user
...     log.save()
...
>>> devices = Device.objects.filter(name__in=["es2035","es2036","es2037","es2038","es2039","es2040"])
>>> len(devices)
6
>>> [d.name for d in devices]
['es2035', 'es2036', 'es2037', 'es2038', 'es2039', 'es2040']
>>> for device in devices:
...     update(device)
...

The changes can be seen in https://netbox.wikimedia.org/extras/changelog/?request_id=3ba2bf1d-ab44-447e-bbc5-b3c82ef23f93
And then I run the sre.dns.netbox cookbook to propagate the deleted records to the DNS.

I've also run the sre.dns.wipe-cache cookbook to clean the dns recursors's cache getting the list with nodeset -S '" "' -e "es20[35-40].codfw.wmnet"

sudo cookbook sre.dns.wipe-cache "es2035.codfw.wmnet" "es2036.codfw.wmnet" "es2037.codfw.wmnet" "es2038.codfw.wmnet" "es2039.codfw.wmnet" "es2040.codfw.wmnet"
Marostegui closed subtask T355424: Productionize es[2035-2040] as Resolved.May 13 2024, 5:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits