Page MenuHomePhabricator
Create Task
Maniphest T301177

Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of gitlab100[2|3] and gitlab-runner100[2|3|4]

Hostname / Racking / Installation Details

Hostnames: gitlab100[3|4] and gitlab-runner100[2|3|4]
Racking Proposal: No more than 2 hosts per row, preferably in different racks for the same row
Networking/Subnet/VLAN/IP: 1G production network
Partitioning/Raid: RAID1
OS Distro: Bullseye

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

gitlab1003:
  • - receive in system on procurement task T297164 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
gitlab1004:
  • - receive in system on procurement task T297164 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
gitlab-runner1002:
  • - receive in system on procurement task T297164 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
gitlab-runner1003:
  • - receive in system on procurement task T297164 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
gitlab-runner1004:
  • - receive in system on procurement task T297164 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
add gitlab-runner1004 to site.ppoperations/puppetproduction+1 -1
adding gitlab and gitlab runner hosts to site.ppoperations/puppetproduction+10 -0
install_server/gitlab: separate partman recipes for physical serversoperations/puppetproduction+4 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Feb 7 2022, 8:18 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
Maintenance_bot added a project: SRE.Feb 7 2022, 8:45 PM
• Cmjohnson moved this task from Backlog to Racking Tasks on the ops-eqiad board.Feb 10 2022, 3:58 PM
RobH renamed this task from (Need By: TBD) rack/setup/install gitlab100[2|3] and gitlab-runner100[2|3|4] to Q3:(Need By: TBD) rack/setup/install gitlab100[2|3] and gitlab-runner100[2|3|4].Feb 23 2022, 6:19 PM
Dzahn added subscribers: Jelto, Dzahn.Feb 23 2022, 6:25 PM

@RobH (cc: @Jelto )

gitlab1002 has existed as a VM in the past, when contractors used it but then we deleted it again.

T274459#6877185

https://phabricator.wikimedia.org/search/query/127L7fu7aA6S/

It might be causing trouble or confusion to reuse the name. Maybe let's skip that one.

RobH reassigned this task from Jclark-ctr to LSobanski.Feb 23 2022, 6:35 PM
RobH added subscribers: LSobanski, Jclark-ctr.

@LSobanski: Is it ok to shift these hostnames from gitlab100[23] to gitlab100[34] due to T301177#7732970?

Please advise and assign back to me for followup.

LSobanski reassigned this task from LSobanski to RobH.Feb 24 2022, 8:28 AM

Fine by me.

RobH renamed this task from Q3:(Need By: TBD) rack/setup/install gitlab100[2|3] and gitlab-runner100[2|3|4] to Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4].Feb 24 2022, 4:49 PM
RobH reassigned this task from RobH to Jclark-ctr.
RobH updated the task description. (Show Details)
In T301177#7734357, @LSobanski wrote:

Fine by me.

Thanks, updated the racking details in the task description, kicking back to John for racking when they arrive.

Jclark-ctr updated the task description. (Show Details)Apr 5 2022, 7:02 PM
RobH unsubscribed.Apr 5 2022, 7:14 PM
Jclark-ctr added a comment.Apr 18 2022, 6:09 PM

Name Rack U Port Cableid
gitlab1003 a3 20 20 1864
gitlab1004 b1 33 31 23000018
gitlab-runner1002 b3 25 12 2615
gitlab-runner1003 c5 28 40 3324
gitlab-runner1004 d8 30. 13 3463

Jclark-ctr updated the task description. (Show Details)Apr 18 2022, 6:16 PM
Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Apr 18 2022, 6:19 PM
Dzahn added a comment.Apr 18 2022, 6:51 PM

It's sufficient if you put the "insetup" role on this and hand it over to us. Let us apply the actual gitlab puppet roles please. Thanks!

Dzahn added subscribers: thcipriani, brennen.Apr 20 2022, 5:25 PM

Tyler, Brennen, added you here per our meeting today. So that you can see status of the physical host install.

gerritbot added a comment.Apr 27 2022, 6:59 PM

Change 787051 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] install_server/gitlab: separate partman recipes for physical servers

https://gerrit.wikimedia.org/r/787051

gerritbot added a project: Patch-For-Review.Apr 27 2022, 6:59 PM
gerritbot added a comment.Apr 27 2022, 8:41 PM

Change 787051 merged by Dzahn:

[operations/puppet@production] install_server/gitlab: separate partman recipes for physical servers

https://gerrit.wikimedia.org/r/787051

Dzahn added a comment.EditedApr 27 2022, 8:43 PM

You should be unblocked to install OS. partman recipe set to raid1-2dev.

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2022, 9:31 PM
Dzahn added a comment.Apr 27 2022, 9:42 PM

confirming that the "gitlab" hosts should use a public IP and the "gitlab-runner" hosts should use a private IP.

• Cmjohnson updated the task description. (Show Details)Apr 28 2022, 8:19 PM
Dzahn mentioned this in T307142: bring new gitlab hardware servers into production.Apr 28 2022, 9:10 PM
• Cmjohnson updated the task description. (Show Details)Apr 29 2022, 12:09 PM
• Cmjohnson added a comment.Apr 29 2022, 3:13 PM
In T301177#7886110, @Dzahn wrote:

confirming that the "gitlab" hosts should use a public IP and the "gitlab-runner" hosts should use a private IP.

This should be in the top racking instructions, things like this will get missed

gerritbot added a comment.Apr 29 2022, 3:20 PM

Change 787758 had a related patch set uploaded (by Cmjohnson; author: Cmjohnson):

[operations/puppet@production] adding gitlab and gitlab runner hosts to site.pp

https://gerrit.wikimedia.org/r/787758

gerritbot added a project: Patch-For-Review.Apr 29 2022, 3:20 PM

Change 787758 merged by Cmjohnson:

[operations/puppet@production] adding gitlab and gitlab runner hosts to site.pp

https://gerrit.wikimedia.org/r/787758

• Cmjohnson updated the task description. (Show Details)Apr 29 2022, 3:22 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 29 2022, 3:31 PM
ops-monitoring-bot added a comment.Apr 29 2022, 4:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host gitlab-runner1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 4:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host gitlab-runner1003.eqiad.wmnet with OS bullseye

gerritbot added a comment.Apr 29 2022, 4:09 PM

Change 787787 had a related patch set uploaded (by Cmjohnson; author: Cmjohnson):

[operations/puppet@production] add gitlab-runner1004 to site.pp

https://gerrit.wikimedia.org/r/787787

gerritbot added a project: Patch-For-Review.Apr 29 2022, 4:09 PM

Change 787787 merged by Cmjohnson:

[operations/puppet@production] add gitlab-runner1004 to site.pp

https://gerrit.wikimedia.org/r/787787

ops-monitoring-bot added a comment.Apr 29 2022, 4:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host gitlab-runner1004.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 4:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host gitlab1003.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 4:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host gitlab1004.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 4:28 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host gitlab-runner1002.eqiad.wmnet with OS bullseye completed:

  • gitlab-runner1002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204291602_cmjohnson_578038_gitlab-runner1002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Maintenance_bot removed a project: Patch-For-Review.Apr 29 2022, 4:31 PM
ops-monitoring-bot added a comment.Apr 29 2022, 4:31 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host gitlab-runner1003.eqiad.wmnet with OS bullseye completed:

  • gitlab-runner1003 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204291603_cmjohnson_578702_gitlab-runner1003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Dzahn added a comment.Apr 29 2022, 4:34 PM
In T301177#7891791, @Cmjohnson wrote:
In T301177#7886110, @Dzahn wrote:

confirming that the "gitlab" hosts should use a public IP and the "gitlab-runner" hosts should use a private IP.

This should be in the top racking instructions, things like this will get missed

Yep, they DID get missed on the equivalent task in codfw. That made me put that here to save you from having to redo them.

ops-monitoring-bot added a comment.Apr 29 2022, 4:37 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host gitlab-runner1004.eqiad.wmnet with OS bullseye completed:

  • gitlab-runner1004 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204291612_cmjohnson_587690_gitlab-runner1004.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Apr 29 2022, 4:41 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host gitlab1003.wikimedia.org with OS bullseye completed:

  • gitlab1003 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204291614_cmjohnson_589396_gitlab1003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Apr 29 2022, 4:43 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host gitlab1004.wikimedia.org with OS bullseye completed:

  • gitlab1004 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204291618_cmjohnson_592021_gitlab1004.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
• Cmjohnson closed this task as Resolved.Apr 29 2022, 4:44 PM
• Cmjohnson updated the task description. (Show Details)

@Dzahn These have all been installed and resolving the task

Dzahn added a comment.Apr 29 2022, 7:43 PM

Thank you @Cmjohnson

We continue this on T307142

Dzahn added a parent task: T307142: bring new gitlab hardware servers into production.Apr 29 2022, 7:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits