Page MenuHomePhabricator
Create Task
Maniphest T393733

Q4:rack/setup/install clouddb102[2-5]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of clouddb102[1-4]

Hostname / Racking / Installation Details

Hostnames: clouddb102[2-5].eqiad.wmnet
Racking Proposal: Ideally each host in a separate row. Failing that, at least keep 1022/1023 in separate rows from each other and 1024/1025 separate as well.
Networking Setup: # of Connections:1 - Speed: 10G. - VLAN: Private
OS Distro: Bookworm (default unless otherwise specified)
Boot Method: Legacy BIOS
Sub-team Technical Contact: Francesco Negri (preferred) or Andrew

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

clouddb1022
  • Receive in system on procurement task T391363 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
clouddb1023
  • Receive in system on procurement task T391363 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
clouddb1024
  • Receive in system on procurement task T391363 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
clouddb1025
  • Receive in system on procurement task T391363 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
clouddb102[25]: Add hieradata fileoperations/puppetproduction+12 -0
installserver: setup new hosts clouddb102[2-5]operations/puppetproduction+6 -1
site.pp: skip clouddb1021 as it existed in the pastoperations/puppetproduction+2 -1
site.pp: Add clouddb102[1-4] as insetup::wmcs_fermoperations/puppetproduction+3 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedVRiley-WMFT393733 Q4:rack/setup/install clouddb102[2-5]

Event Timeline

RobH created this task.May 8 2025, 7:25 PM
RobH assigned this task to fnegri.
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.

Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-team receiving the new servers. This is due to the majority of DC Ops not having root/merge puppet rights.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

Maintenance_bot added a project: SRE.May 8 2025, 7:29 PM
gerritbot added a comment.May 9 2025, 5:03 PM

Change #1143871 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] site.pp: Add clouddb102[1-4] as insetup::wmcs_ferm

https://gerrit.wikimedia.org/r/1143871

gerritbot added a project: Patch-For-Review.May 9 2025, 5:03 PM
gerritbot added a comment.May 9 2025, 5:21 PM

Change #1143871 merged by FNegri:

[operations/puppet@production] site.pp: Add clouddb102[1-4] as insetup::wmcs_ferm

https://gerrit.wikimedia.org/r/1143871

fnegri removed fnegri as the assignee of this task.May 9 2025, 5:28 PM
fnegri subscribed.

update the site.pp file with the insetup role for your team

Done in https://gerrit.wikimedia.org/r/1143871

and add the new servers to preseed.yml for partition info

This was not necessary as preseed.yml already contains an entry for clouddb1* that matches both old and new hosts.

fnegri renamed this task from Q4:rack/setup/install clouddb102[1-4] to Q4:rack/setup/install clouddb102[2-5].May 9 2025, 5:29 PM
fnegri updated the task description. (Show Details)

Please note that we should skip clouddb1021 as it existed in the past and was decom'd in T368518: decommission clouddb1021.

I updated the task to rename the 4 new hosts clouddb1022 to clouddb1025.

Maintenance_bot removed a project: Patch-For-Review.May 9 2025, 5:30 PM
gerritbot added a comment.May 9 2025, 5:31 PM

Change #1143876 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] site.pp: skip clouddb1021 as it existed in the past

https://gerrit.wikimedia.org/r/1143876

gerritbot added a project: Patch-For-Review.May 9 2025, 5:31 PM
gerritbot added a comment.May 9 2025, 5:36 PM

Change #1143876 merged by FNegri:

[operations/puppet@production] site.pp: skip clouddb1021 as it existed in the past

https://gerrit.wikimedia.org/r/1143876

Maintenance_bot removed a project: Patch-For-Review.May 9 2025, 6:30 PM
VRiley-WMF updated the task description. (Show Details)Jul 21 2025, 7:53 PM
VRiley-WMF claimed this task.Jul 22 2025, 7:50 PM
VRiley-WMF updated the task description. (Show Details)
VRiley-WMF added a comment.EditedJul 22 2025, 8:00 PM

clouddb1022
Rack A4
U33
CableID: 20220030
Port: 43

clouddb1023
Rack B2
U32
CableID: 5253
Port: 39

clouddb1024
Rack E8
U35
CableID 240707900050
Port 35

clouddb1025
Rack F8
U24
CableID 240707900035
Port 24

VRiley-WMF updated the task description. (Show Details)Jul 23 2025, 10:14 PM
ops-monitoring-bot added a comment.Jul 24 2025, 6:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 24 2025, 6:52 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm executed with errors:

  • clouddb1022 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console clouddb1022.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jul 24 2025, 6:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 24 2025, 7:42 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm executed with errors:

  • clouddb1022 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console clouddb1022.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jul 24 2025, 10:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 24 2025, 11:04 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm executed with errors:

  • clouddb1022 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console clouddb1022.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jul 24 2025, 11:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 25 2025, 12:00 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm executed with errors:

  • clouddb1022 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console clouddb1022.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
VRiley-WMF added a comment.Jul 25 2025, 7:12 PM

While attempting to image this server (clouddb1022) and got this error.

kVh9WNyR.png (296×500 px, 32 KB)

fnegri added a comment.Jul 28 2025, 2:11 PM

I think the partman recipe is incompatible with the new servers, I'll look into it.

VRiley-WMF moved this task from Racking Tasks to Blocked on the ops-eqiad board.Jul 28 2025, 9:17 PM
gerritbot added a comment.Jul 29 2025, 4:22 PM

Change #1173974 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] installserver: setup new hosts clouddb102[2-5]

https://gerrit.wikimedia.org/r/1173974

gerritbot added a project: Patch-For-Review.Jul 29 2025, 4:22 PM
ops-monitoring-bot added a comment.Jul 29 2025, 5:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm

VRiley-WMF updated the task description. (Show Details)Jul 29 2025, 6:01 PM

Tried to run reimage on clouddb1022, to no avail. Running through clouddb1023 to see if there is a difference.

VRiley-WMF updated the task description. (Show Details)Jul 29 2025, 6:03 PM
VRiley-WMF updated the task description. (Show Details)Jul 29 2025, 6:58 PM
ops-monitoring-bot added a comment.Jul 29 2025, 7:45 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm executed with errors:

  • clouddb1022 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console clouddb1022.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jul 29 2025, 8:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host clouddb1023.eqiad.wmnet with OS bookworm

VRiley-WMF added a comment.Jul 29 2025, 8:17 PM

I have it the same issue with the same error on clouddb1023

Jclark-ctr moved this task from Blocked to Decommission on the ops-eqiad board.Jul 30 2025, 4:13 AM
Jclark-ctr moved this task from Decommission to Blocked on the ops-eqiad board.
Jclark-ctr mentioned this in T400758: Alert for device lsw1-e8-eqiad.mgmt.eqiad.wmnet - Port with no description on access switch.Jul 30 2025, 5:28 AM
Jclark-ctr mentioned this in T400763: Alert for device lsw1-f8-eqiad.mgmt.eqiad.wmnet - Port with no description on access switch.
fnegri added a comment.Jul 30 2025, 8:23 AM

@VRiley-WMF I think that my patch above (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1173974) should fix this, but it isn't merged yet.

ops-monitoring-bot added a comment.Jul 30 2025, 2:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm

fnegri added a subscriber: Papaul.Jul 30 2025, 2:39 PM

@Papaul the reimage of clouddb1022 will fail until my patch above is merged, I'm waiting for a review from Data-Persistence .

Papaul added a comment.Jul 30 2025, 2:42 PM

@fnegri thanks i was about to ping you also on that.

gerritbot added a comment.Jul 30 2025, 3:13 PM

Change #1173974 merged by FNegri:

[operations/puppet@production] installserver: setup new hosts clouddb102[2-5]

https://gerrit.wikimedia.org/r/1173974

Maintenance_bot removed a project: Patch-For-Review.Jul 30 2025, 3:31 PM
ops-monitoring-bot added a comment.Jul 30 2025, 4:01 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm executed with errors:

  • clouddb1022 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console clouddb1022.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jul 30 2025, 6:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 30 2025, 6:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host clouddb1023.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 30 2025, 7:05 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host clouddb1022.eqiad.wmnet with OS bookworm completed:

  • clouddb1022 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507301847_vriley_3936284_clouddb1022.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF moved this task from Blocked to Racking Tasks on the ops-eqiad board.Jul 30 2025, 7:05 PM
VRiley-WMF updated the task description. (Show Details)
ops-monitoring-bot added a comment.Jul 30 2025, 7:25 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host clouddb1023.eqiad.wmnet with OS bookworm completed:

  • clouddb1023 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507301907_vriley_3959389_clouddb1023.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF updated the task description. (Show Details)Jul 30 2025, 10:59 PM
VRiley-WMF added a comment.Jul 30 2025, 11:02 PM

Was able to run though 2 of these. Running into issues with BMC password.

clouddb1022 - Finished, no issues
clouddb1023 - Finished, Password was set to the BMC default
clouddb1024 and clouddb1025 - Tried multiple combinations of usernames and passwords with no luck.

ops-monitoring-bot added a comment.Jul 31 2025, 1:55 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host clouddb1025.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 31 2025, 2:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host clouddb1024.eqiad.wmnet with OS bookworm

VRiley-WMF updated the task description. (Show Details)Jul 31 2025, 2:23 AM
ops-monitoring-bot added a comment.Jul 31 2025, 2:28 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host clouddb1025.eqiad.wmnet with OS bookworm completed:

  • clouddb1025 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507310213_vriley_4070441_clouddb1025.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jul 31 2025, 2:33 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host clouddb1024.eqiad.wmnet with OS bookworm completed:

  • clouddb1024 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507310218_vriley_4072907_clouddb1024.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF closed this task as Resolved.Jul 31 2025, 2:34 AM

This has been completed

fnegri mentioned this in T401295: Decide how to use the new clouddb hosts (clouddb102[2-5]).Aug 6 2025, 9:02 AM
ops-monitoring-bot added a comment.Aug 19 2025, 10:30 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host clouddb1023.eqiad.wmnet with OS bookworm executed with errors:

  • clouddb1023 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console clouddb1023.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Marostegui subscribed.Oct 22 2025, 1:24 PM

I've expanded their logical volume to use most of the disk as we normally do

root@clouddb1022:~# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sda3  tank lvm2 a--  <8.69t 56.36g
gerritbot added a comment.Oct 22 2025, 1:30 PM

Change #1198064 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb102[25]: Add hieradata file

https://gerrit.wikimedia.org/r/1198064

gerritbot added a project: Patch-For-Review.Oct 22 2025, 1:30 PM

Change #1198064 merged by Marostegui:

[operations/puppet@production] clouddb102[25]: Add hieradata file

https://gerrit.wikimedia.org/r/1198064

Maintenance_bot removed a project: Patch-For-Review.Oct 22 2025, 2:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits