Site/Location: eqiad
Number of systems: 1
Service: Ceph cluster administration using cephadm
Networking Requirements: internal IP
Processor Requirements: How many VCPUs? 2
Memory: 4GB
Disks: 20 GB
Other Requirements: none
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| Remove the cephadm role | Btullis | operations/puppet | production | +0 -22 | |
| Add cephadm globbing to partition selection | Muehlenhoff | operations/puppet | production | +3 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gehel | T327267 Create a DSE Kubernetes cluster with support for persistent storage from Ceph | |||
| Resolved | Gehel | T324660 Install Ceph Cluster for Data Platform Engineering | |||
| Declined | BTullis | T363558 Switch the DPE Ceph cluster to use cephadm management | |||
| Resolved | BTullis | T363559 eqiad: 1 VMs requested for ceph cluster administration (cephadm) |
Event Timeline
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephadm1001.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephadm1001.eqiad.wmnet with OS bookworm executed with errors:
- cephadm1001 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" cephadm1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
Comment Actions
Change #1024689 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Add cephadm globbing to partition selection
Comment Actions
Change #1024689 merged by Btullis:
[operations/puppet@production] Add cephadm globbing to partition selection
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephadm1001.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephadm1001.eqiad.wmnet with OS bookworm completed:
- cephadm1001 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404261402_btullis_2498070_cephadm1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephadm1001.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephadm1001.eqiad.wmnet with OS bullseye executed with errors:
- cephadm1001 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" cephadm1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephadm1001.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephadm1001.eqiad.wmnet with OS bookworm completed:
- cephadm1001 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404291040_btullis_3023774_cephadm1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1026100 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Remove the cephadm role
Comment Actions
Change #1026100 merged by Btullis:
[operations/puppet@production] Remove the cephadm role