Page MenuHomePhabricator
Create Task
Maniphest T301183

Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of gitlab200[2|3] and gitlab-runner200[2|3|4]

Hostname / Racking / Installation Details

Hostnames: gitlab200[2|3] and gitlab-runner200[2|3|4]
Racking Proposal: No more than 2 hosts per row, preferably in different racks for the same row
Networking/Subnet/VLAN/IP: 1G production network
Partitioning/Raid: RAID1
OS Distro: Bullseye

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

gitlab2002: A1U29 ge-1/0/28
  • - receive in system on procurement task T297163 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
gitlab2003: B5U9 ge-5/0/8
  • - receive in system on procurement task T297163 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
gitlab-runner2002: B8U35 ge-8/0/34
  • - receive in system on procurement task T297163 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
gitlab-runner2003: C5U6 ge-5/0/5
  • - receive in system on procurement task T297163 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
gitlab-runner2004: D5U12 ge-5/0/11
  • - receive in system on procurement task T297163 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add new gitlab nodes in site.ppoperations/puppetproduction+10 -0
install_server/gitlab: separate partman recipes for physical serversoperations/puppetproduction+4 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Feb 7 2022, 8:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 7 2022, 8:50 PM
RobH mentioned this in Unknown Object (Task).Feb 7 2022, 8:50 PM
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
Maintenance_bot added a project: SRE.Feb 7 2022, 9:45 PM
RobH renamed this task from (Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] to Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4].Feb 23 2022, 6:21 PM
Papaul updated the task description. (Show Details)Apr 20 2022, 6:16 PM
Papaul updated the task description. (Show Details)Apr 26 2022, 9:42 PM
RobH unsubscribed.Apr 26 2022, 9:55 PM
Papaul updated the task description. (Show Details)Apr 26 2022, 10:05 PM
Dzahn added subscribers: Jelto, Dzahn.Apr 27 2022, 6:38 PM

cc: @Jelto

@Papaul has asked which partman recipe to use. SInce these are the first physical servers that might not be obvious yet. I will take a look though.

Dzahn added a comment.Apr 27 2022, 6:45 PM

The current install_server config is:

gitlab*) echo partman/flat.cfg virtual.cfg ;; \

so we can't keep using this "gitlab*" wildcard.

We need to separate into VM and not VM now. (or should have used different hostnames)

gerritbot added a comment.Apr 27 2022, 6:59 PM

Change 787051 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] install_server/gitlab: separate partman recipes for physical servers

https://gerrit.wikimedia.org/r/787051

gerritbot added a project: Patch-For-Review.Apr 27 2022, 6:59 PM
gerritbot added a comment.Apr 27 2022, 8:41 PM

Change 787051 merged by Dzahn:

[operations/puppet@production] install_server/gitlab: separate partman recipes for physical servers

https://gerrit.wikimedia.org/r/787051

Dzahn added a comment.EditedApr 27 2022, 8:45 PM

@Papaul You should be unblocked to install OS. partman recipe set to raid1-2dev.

Papaul added a comment.Apr 27 2022, 8:48 PM

@Dzahn thanks

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2022, 9:31 PM
Dzahn added a comment.Apr 27 2022, 9:40 PM

confirming that the "gitlab" hosts should use a public IP and the "gitlab-runner" hosts should use a private IP.

Dzahn changed the task status from Open to In Progress.Apr 27 2022, 9:49 PM
gerritbot added a comment.Apr 27 2022, 10:04 PM

Change 787085 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new gitlab nodes in site.pp

https://gerrit.wikimedia.org/r/787085

gerritbot added a project: Patch-For-Review.Apr 27 2022, 10:04 PM
gerritbot added a comment.Apr 27 2022, 10:20 PM

Change 787085 merged by Papaul:

[operations/puppet@production] Add new gitlab nodes in site.pp

https://gerrit.wikimedia.org/r/787085

ops-monitoring-bot added a comment.Apr 27 2022, 10:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab2002.wikimedia.org with OS bullseye

Papaul updated the task description. (Show Details)Apr 27 2022, 10:25 PM
Papaul updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Apr 27 2022, 10:30 PM
ops-monitoring-bot added a comment.Apr 27 2022, 10:39 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab2002.wikimedia.org with OS bullseye executed with errors:

  • gitlab2002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 27 2022, 10:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab2002.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Apr 27 2022, 10:58 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab2002.wikimedia.org with OS bullseye executed with errors:

  • gitlab2002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 27 2022, 10:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab-runner2002.codfw.wmnet with OS bullseye

Papaul added a comment.Apr 27 2022, 11:17 PM

@Dzahn OS install failed on gitlab-runner2002 because of partitioning. I think is becasue you have:
partman/raid1-2dev.cfg ;; \ and not
partman/standard.cfg partman/raid1-2dev.cfg ;; \
Please double check and let me know when I can resume the Install. Thanks

``
┌────────────────────────┤ [!!] Partition disks ├─────────────────────────┐

│                                                                         │
│ The installer can guide you through partitioning a disk (using          │
│ different standard schemes) or, if you prefer, you can do it            │
│ manually. With guided partitioning you will still have a chance later   │
│ to review and customise the results.                                    │
│                                                                         │
│ If you choose guided partitioning for an entire disk, you will next     │
│ be asked which disk should be used.                                     │
│                                                                         │
│ Partitioning method:                                                    │
│                                                                         │
│          Guided - use the largest continuous free space     -           │
│          Guided - use entire disk                           0           │
│          Guided - use entire disk and set up LVM            ▒           │
└
ops-monitoring-bot added a comment.Apr 27 2022, 11:31 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab-runner2002.codfw.wmnet with OS bullseye executed with errors:

  • gitlab-runner2002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 28 2022, 3:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab-runner2002.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 28 2022, 3:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab-runner2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 28 2022, 4:18 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab-runner2002.codfw.wmnet with OS bullseye completed:

  • gitlab-runner2002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204281544_pt1979_694875_gitlab-runner2002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Apr 28 2022, 4:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab-runner2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 28 2022, 4:33 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab-runner2003.codfw.wmnet with OS bullseye completed:

  • gitlab-runner2003 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204281559_pt1979_696080_gitlab-runner2003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Apr 28 2022, 4:53 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab-runner2004.codfw.wmnet with OS bullseye completed:

  • gitlab-runner2004 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204281620_pt1979_701758_gitlab-runner2004.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Apr 28 2022, 5:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab2002.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Apr 28 2022, 5:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye

Papaul updated the task description. (Show Details)Apr 28 2022, 5:28 PM
ops-monitoring-bot added a comment.Apr 28 2022, 5:30 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab2002.wikimedia.org with OS bullseye executed with errors:

  • gitlab2002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 28 2022, 5:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab2002.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Apr 28 2022, 6:04 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye completed:

  • gitlab2003 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204281716_pt1979_711452_gitlab2003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Apr 28 2022, 6:06 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab2002.wikimedia.org with OS bullseye completed:

  • gitlab2002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204281731_pt1979_712320_gitlab2002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul closed this task as Resolved.Apr 28 2022, 6:15 PM
Papaul updated the task description. (Show Details)

This is complete

Dzahn mentioned this in T307142: bring new gitlab hardware servers into production.Apr 28 2022, 9:10 PM
Dzahn added a parent task: T307142: bring new gitlab hardware servers into production.Apr 29 2022, 7:44 PM
Dzahn added a comment.Apr 29 2022, 7:48 PM

Thank you @Papaul. We continue on T307142

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL