Page MenuHomePhabricator
Create Task
Maniphest T355571

Q#:rack/setup/install an-redacteddb1001
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of an-redacteddb1001

Hostname / Racking / Installation Details

Hostnames: an-redacteddb1001 (That's the best I could think of)
Racking Proposal: Anywhere within eqiad A-F
Networking Setup: # of Connections:1/2 - Speed:1G/10G. - VLAN:Private/Public/Other(Specify) : AAAA records:Y/N, Additional IP records (Cassandra)? Yes/No
Partitioning/Raid: HW Raid: Y/N, Partman recipe and/or desired Raid Level: Hardware RAID 10 with partman/custom/db.cfg
OS Distro: Bullseye (default unless otherwise specified) Bookworm
Sub-team Technical Contact: Who should our on-sites contact with any questions involving system racking and setup? @BTullis

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

an-redacteddb1001:
  • Receive in system on procurement task T354452 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

SubjectRepoBranchLines +/-
Update the contacts for an-redacteddb1001operations/puppetproduction+1 -1
Install an-redacteddb1001 with puppet 7operations/puppetproduction+2 -0
Add a partition recipe for an-redacteddboperations/puppetproduction+2 -0
add an-redacteddb1001 to site.ppoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jan 22 2024, 5:30 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.
Maintenance_bot added a project: SRE.Jan 22 2024, 6:29 PM
VRiley-WMF claimed this task.Feb 15 2024, 5:59 PM
VRiley-WMF updated the task description. (Show Details)Feb 15 2024, 9:02 PM
VRiley-WMF added a comment.Feb 15 2024, 9:46 PM

an-redacteddb1001
Rack D2
U25
Port 28
CableID 5365

VRiley-WMF updated the task description. (Show Details)Feb 15 2024, 9:49 PM
VRiley-WMF updated the task description. (Show Details)Feb 15 2024, 10:43 PM
ops-monitoring-bot added a comment.Feb 16 2024, 8:38 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bullseye

Jclark-ctr updated the task description. (Show Details)Feb 16 2024, 8:40 AM
Jclark-ctr subscribed.

Firmware cookbook seems to be broken Manually downgraded firmware on server for Nic.

Jclark-ctr added a comment.Feb 16 2024, 8:47 AM
This comment was removed by Jclark-ctr.
Jclark-ctr updated the task description. (Show Details)Feb 16 2024, 8:51 AM
Jclark-ctr moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Feb 20 2024, 11:41 PM
gerritbot added a comment.Feb 21 2024, 4:58 PM

Change 1005560 had a related patch set uploaded (by Jclark-ctr; author: jclark):

[operations/puppet@production] add an-redacteddb1001 to site.pp

https://gerrit.wikimedia.org/r/1005560

gerritbot added a project: Patch-For-Review.Feb 21 2024, 4:58 PM

Change 1005560 merged by Jclark-ctr:

[operations/puppet@production] add an-redacteddb1001 to site.pp

https://gerrit.wikimedia.org/r/1005560

ops-monitoring-bot added a comment.Feb 21 2024, 5:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2024, 5:30 PM
Jclark-ctr reassigned this task from VRiley-WMF to BTullis.Feb 21 2024, 5:44 PM
Jclark-ctr added a subscriber: VRiley-WMF.

@BTullis this is a custom configuration and i am not having any luck with imaging 20 disk raid10. if you can asisst thank you

Jclark-ctr updated the task description. (Show Details)Feb 21 2024, 5:45 PM
ops-monitoring-bot added a comment.Feb 22 2024, 2:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 22 2024, 2:44 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bullseye executed with errors:

  • an-redacteddb1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" an-redacteddb1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
BTullis added a project: Data-Platform-SRE (2024.02.12 - 2024.03.03).Feb 23 2024, 5:52 PM
BTullis moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.02.12 - 2024.03.03) board.
gerritbot added a comment.Feb 23 2024, 6:05 PM

Change 1006052 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add a partition recipe for an-redacteddb

https://gerrit.wikimedia.org/r/1006052

gerritbot added a project: Patch-For-Review.Feb 23 2024, 6:05 PM
gerritbot added a comment.Feb 26 2024, 9:53 AM

Change 1006052 merged by Btullis:

[operations/puppet@production] Add a partition recipe for an-redacteddb

https://gerrit.wikimedia.org/r/1006052

gerritbot added a comment.Feb 26 2024, 10:30 AM

Change 1006486 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Install an-redacteddb1001 with puppet 7

https://gerrit.wikimedia.org/r/1006486

gerritbot added a comment.Feb 26 2024, 10:47 AM

Change 1006486 abandoned by Btullis:

[operations/puppet@production] Install an-redacteddb1001 with puppet 7

Reason:

Unnecessary after all.

https://gerrit.wikimedia.org/r/1006486

ops-monitoring-bot added a comment.Feb 26 2024, 10:47 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Feb 26 2024, 11:30 AM
ops-monitoring-bot added a comment.Feb 26 2024, 11:42 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bookworm executed with errors:

  • an-redacteddb1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" an-redacteddb1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 26 2024, 11:44 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 26 2024, 10:41 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bookworm executed with errors:

  • an-redacteddb1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402261207_btullis_2185392_an-redacteddb1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" an-redacteddb1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 26 2024, 10:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 26 2024, 11:26 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bookworm completed:

  • an-redacteddb1001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402262257_btullis_2300335_an-redacteddb1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Feb 26 2024, 11:26 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host an-redacteddb1001.eqiad.wmnet with OS bookworm executed with errors:

  • an-redacteddb1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402262257_btullis_2300335_an-redacteddb1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" an-redacteddb1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
gerritbot added a comment.Feb 26 2024, 11:33 PM

Change 1006600 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the contacts for an-redacteddb1001

https://gerrit.wikimedia.org/r/1006600

gerritbot added a project: Patch-For-Review.Feb 26 2024, 11:33 PM
gerritbot added a comment.Feb 26 2024, 11:56 PM

Change 1006600 merged by Btullis:

[operations/puppet@production] Update the contacts for an-redacteddb1001

https://gerrit.wikimedia.org/r/1006600

BTullis closed this task as Resolved.Feb 26 2024, 11:59 PM
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2024.02.12 - 2024.03.03) board.

@Jclark-ctr - This is all done now, I believe. I had to change one BIOS setting to boot from the hard drive, but nothing else. OK for me to resolve?

BTullis updated the task description. (Show Details)Feb 26 2024, 11:59 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 27 2024, 12:30 AM
Marostegui subscribed.May 14 2024, 2:39 PM

@BTullis what is the status of this?
I can see the host is up, but not yet provisioned?

root@an-redacteddb1001:~# df -hT /srv
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs   9.1T   65G  9.1T   1% /srv
root@an-redacteddb1001:~# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda3  tank lvm2 a--  <17.42t 8.32t

clouddb1021 is now at 81% so maybe we need to start getting this host provisioned, considering there are 12TB to transfer, it might take a while :-)

BTullis mentioned this in T365453: Bring an-redacteddb1001 into service to replace clouddb1021.May 21 2024, 4:31 PM
Marostegui closed subtask T365453: Bring an-redacteddb1001 into service to replace clouddb1021 as Resolved.Wed, Jun 19, 9:52 AM
Marostegui reopened subtask T365453: Bring an-redacteddb1001 into service to replace clouddb1021 as Open.
BTullis closed subtask T365453: Bring an-redacteddb1001 into service to replace clouddb1021 as Resolved.Thu, Jun 27, 9:03 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits