Page MenuHomePhabricator
Create Task
Maniphest T305568

Q4:(Need By: TBD) rack/setup/install aqs2001-aqs2012
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of aqs2001-aqs2012

Hostname / Racking / Installation Details

Hostnames: aqs2001-aqs2012
Racking Proposal: Distributed over 3 racks, 4 servers each
Networking Setup: 1G in the private1 network
Partitioning/Raid: Same as used on aqs1010-aqs1015
OS Distro: Bullseye

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

aqs2001: B6U35 ge-6/0/34
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2002: B6U65 ge-6/0/35
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2003: B6U37 ge-6/0/36
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2004: B6U38 ge-6/0/37
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2005:
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2006:
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2007:
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2008:
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2009:
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2010:
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2011:
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
aqs2012:
  • - receive in system on procurement task T304182 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add new aqs node to site.pp and to netboot.cfgoperations/puppetproduction+6 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT305568 Q4:(Need By: TBD) rack/setup/install aqs2001-aqs2012

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
RobH added a parent task: Unknown Object (Task).Apr 6 2022, 5:04 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH mentioned this in Unknown Object (Task).
Maintenance_bot added a project: SRE.Apr 6 2022, 5:29 PM
Eevans added a project: Cassandra.Apr 11 2022, 11:48 PM
BTullis mentioned this in T305102: Erroneous node placement (AQS Cassandra cluster).Apr 13 2022, 2:11 PM
Papaul updated the task description. (Show Details)Apr 20 2022, 6:27 PM
Eevans added a comment.EditedApr 21 2022, 8:45 PM

@Papaul does this notation you added refer to the row (and rack) location? For example: aqs2001: B6U35 ge-6/0/34, does that mean row 'B'?

Papaul added a comment.Apr 21 2022, 9:53 PM

@Eevans yes B is row B , 6 is the rack number and U35 is the position of the server in the rack (row B rack 6 position 35)

RobH unsubscribed.Apr 21 2022, 10:21 PM
Eevans added a comment.EditedApr 22 2022, 12:32 AM
In T305568#7872880, @Papaul wrote:

@Eevans yes B is row B , 6 is the rack number and U35 is the position of the server in the rack (row B rack 6 position 35)

Gotcha; Now is probably a good time then to pull you into the discussion started in T305102: Erroneous node placement (AQS Cassandra cluster). That ticket was created mostly to get things sorted in eqiad, but with the expansion to codfw happening, we took a stab at that as well. The row numbers we put into that second table are -I think- a strawperson, that exact mapping doesn't have to match that if the distribution is otherwise Good™ (though we should probably update that table as we go).

The main thing that we wanted to do is to create pairs of rows that are equivalent. See the eqiad table in that ticket for an example: there we are proposing that [A,D], [B,E], and [C,F] are equivalent. For Cassandra purposes, we'd treat each of those pairs as a row, and in so doing, give ourselves more options for placement if/when the cluster is expanded again. AFAIK, there are only 4 rows in codfw, but we wanted to see about creating at least one equivalency there (perhaps combining the two most crowded?) so that if codfw is ever expanded the way eqiad was, we'd be prepared.

Papaul added a comment.Apr 22 2022, 1:03 AM

@Eevans since we have only for rows in codfw do think doing [AC] and [BD] with each row having 3 servers in a rack will work or not please advice I will be glad to work with you on what is the best option.

Thanks.

Eevans added a comment.Apr 22 2022, 1:48 AM
In T305568#7873122, @Papaul wrote:

@Eevans since we have only for rows in codfw do think doing [AC] and [BD] with each row having 3 servers in a rack will work or not please advice I will be glad to work with you on what is the best option.

For Cassandra, we need 3... let's call them units of failure. We typically replicate data 3-ways, so the idea is to have one replica in each of these units, so that if one fails, we continue to have 2 live copies (important because 2 is enough for quorum). We've been using a row in our datacenters as that unit of failure. So [AC] and [BD] would only give us two such units. But, we could do (and this is just an example), [A,D], [B], [C]. This might be useful in the near-term if for example A is tight on space (because it would let us use D as an equivalent to A as needed), and if we ever expand to 6 rows (like we did with eqiad), then we can make that [A,D], [B,E], [C,F] (again, just as an example).

Papaul added a comment.EditedApr 22 2022, 2:13 AM

@Eevans I do not have any space issue in codfw for now, so I think [A,D], [B], [C] should work without a problem. Now what i will like for you to give me since you understand this Cassandra setup better is some numbers. for example for :

  • [AD] how many serve in a rack in row A and now many servers in a rack for row D
  • [B ] how many servers in a rack for the row B?
  • [C ] how many servers in a rack for the row C?

Thanks

Eevans added a comment.Apr 22 2022, 9:58 PM
In T305568#7873144, @Papaul wrote:

@Eevans I do not have any space issue in codfw for now, so I think [A,D], [B], [C] should work without a problem. Now what i will like for you to give me since you understand this Cassandra setup better is some numbers. for example for :

  • [AD] how many serve in a rack in row A and now many servers in a rack for row D
  • [B ] how many servers in a rack for the row B?
  • [C ] how many servers in a rack for the row C?

There are 12 going up, so 4, 4, and 4. Any distribution of those 4 servers for [AD] is fine too (1 & 3, 2 & 2, 3 & 1, or even 0 & 4 or 4 & 0). The main thing would be that if we put any servers in D for example, that they correspond to what we consider the same unit of failure ("rack" in Cassandra parlance) as A, and vice-versa.

Papaul added a comment.Apr 26 2022, 5:54 PM

@Eevans I received those nodes today so I will be racking them tomorrow. Here is my racking proposal for tomorrow.

RowRacknodes
AA6aqs2001,aqs2002,aqs2003 and aqs2004
BB6aqs2005,aqs2006,aqs2007 and aqs2008
C
DD6aqs2009,aqs2010,aqs2011 and aqs2012
LSobanski subscribed.Apr 27 2022, 1:40 PM
Eevans added a comment.Apr 27 2022, 7:22 PM
In T305568#7881920, @Papaul wrote:

@Eevans I received those nodes today so I will be racking them tomorrow. Here is my racking proposal for tomorrow.

RowRacknodes
AA6aqs2001,aqs2002,aqs2003 and aqs2004
BB6aqs2005,aqs2006,aqs2007 and aqs2008
C
DD6aqs2009,aqs2010,aqs2011 and aqs2012

This seems fine with me; I'll update T305102 accordingly.

For posterity sake, you had suggested that these would go up in [AD], [B], [C] in a previous comment, and here it is [A], [B], [D] (with C un-utilized). I'll document that (for now) we're going to treat [AC] as equivalent, and if we ever add an E & F we can allocate them as [AC], [BE], [DF] (or similar).

Papaul added a comment.Apr 28 2022, 6:16 PM

@Eevans thanks

Papaul updated the task description. (Show Details)Apr 29 2022, 2:35 AM
BTullis subscribed.Apr 29 2022, 3:44 PM
gerritbot added a comment.Apr 29 2022, 7:25 PM

Change 787812 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new aqs node to site.pp and to netboot.cfg

https://gerrit.wikimedia.org/r/787812

gerritbot added a project: Patch-For-Review.Apr 29 2022, 7:25 PM
gerritbot added a comment.Apr 29 2022, 7:31 PM

Change 787812 merged by Papaul:

[operations/puppet@production] Add new aqs node to site.pp and to netboot.cfg

https://gerrit.wikimedia.org/r/787812

ops-monitoring-bot added a comment.Apr 29 2022, 7:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2001.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 8:08 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2001.codfw.wmnet with OS bullseye executed with errors:

  • aqs2001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 29 2022, 8:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2001.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 8:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2002.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Apr 29 2022, 8:30 PM
ops-monitoring-bot added a comment.Apr 29 2022, 8:31 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2001.codfw.wmnet with OS bullseye completed:

  • aqs2001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204292011_pt1979_906737_aqs2001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Apr 29 2022, 8:46 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2002.codfw.wmnet with OS bullseye executed with errors:

  • aqs2002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 29 2022, 9:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2002.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 9:42 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2002.codfw.wmnet with OS bullseye completed:

  • aqs2002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204292121_pt1979_915660_aqs2002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Apr 29 2022, 9:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 10:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 10:15 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2004.codfw.wmnet with OS bullseye executed with errors:

  • aqs2004 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 29 2022, 10:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 10:25 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2003.codfw.wmnet with OS bullseye executed with errors:

  • aqs2003 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 29 2022, 10:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 10:48 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2004.codfw.wmnet with OS bullseye executed with errors:

  • aqs2004 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 29 2022, 10:49 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2003.codfw.wmnet with OS bullseye completed:

  • aqs2003 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204292228_pt1979_925251_aqs2003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.May 2 2022, 1:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 2:10 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2004.codfw.wmnet with OS bullseye completed:

  • aqs2004 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205021350_pt1979_1367510_aqs2004.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.May 2 2022, 2:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2005.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 2:50 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2005.codfw.wmnet with OS bullseye executed with errors:

  • aqs2005 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 2:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2005.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 3:30 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2005.codfw.wmnet with OS bullseye executed with errors:

  • aqs2005 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 3:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2005.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 4:02 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2005.codfw.wmnet with OS bullseye completed:

  • aqs2005 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205021542_pt1979_1382030_aqs2005.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.May 2 2022, 4:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2007.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 4:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2006.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 4:34 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2007.codfw.wmnet with OS bullseye executed with errors:

  • aqs2007 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 4:37 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2006.codfw.wmnet with OS bullseye executed with errors:

  • aqs2006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 4:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2007.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 4:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2006.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 5:01 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2007.codfw.wmnet with OS bullseye completed:

  • aqs2007 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205021640_pt1979_1391365_aqs2007.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.May 2 2022, 5:08 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2006.codfw.wmnet with OS bullseye completed:

  • aqs2006 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205021648_pt1979_1391676_aqs2006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.May 2 2022, 5:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2008.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 5:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2009.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)May 2 2022, 5:31 PM
ops-monitoring-bot added a comment.May 2 2022, 5:59 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2008.codfw.wmnet with OS bullseye executed with errors:

  • aqs2008 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 6:01 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2009.codfw.wmnet with OS bullseye executed with errors:

  • aqs2009 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 6:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2008.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 6:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 6:28 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2008.codfw.wmnet with OS bullseye completed:

  • aqs2008 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205021807_pt1979_1405299_aqs2008.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.May 2 2022, 6:31 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2009.codfw.wmnet with OS bullseye completed:

  • aqs2009 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205021808_pt1979_1405353_aqs2009.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.May 2 2022, 6:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2010.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 6:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2011.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 7:06 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2010.codfw.wmnet with OS bullseye executed with errors:

  • aqs2010 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 7:12 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2011.codfw.wmnet with OS bullseye executed with errors:

  • aqs2011 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 7:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2010.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 7:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2011.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 7:14 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2011.codfw.wmnet with OS bullseye executed with errors:

  • aqs2011 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 7:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2011.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 7:16 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2011.codfw.wmnet with OS bullseye executed with errors:

  • aqs2011 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 7:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2011.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 7:36 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2010.codfw.wmnet with OS bullseye completed:

  • aqs2010 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205021913_pt1979_1415584_aqs2010.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.May 2 2022, 7:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2012.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 7:50 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2011.codfw.wmnet with OS bullseye completed:

  • aqs2011 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205021929_pt1979_1416549_aqs2011.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul updated the task description. (Show Details)May 2 2022, 7:53 PM
ops-monitoring-bot added a comment.May 2 2022, 8:15 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2012.codfw.wmnet with OS bullseye executed with errors:

  • aqs2012 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 2 2022, 8:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host aqs2012.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 2 2022, 8:40 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host aqs2012.codfw.wmnet with OS bullseye completed:

  • aqs2012 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205022020_pt1979_1425818_aqs2012.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul closed this task as Resolved.May 2 2022, 8:41 PM
Papaul updated the task description. (Show Details)

@Eevans this is complete

Eevans added a comment.May 3 2022, 6:29 PM
In T305568#7897589, @Papaul wrote:

@Eevans this is complete

Thank you!

Eevans awarded a token.May 3 2022, 6:30 PM
Eevans mentioned this in T307641: AQS multi-datacenter cluster expansion.May 4 2022, 11:42 PM
Eevans mentioned this in T307801: Bootstrap new Cassandra nodes (codfw).May 6 2022, 4:00 PM
Volans reopened this task as Open.Jun 9 2022, 2:59 PM
Volans subscribed.

Re-opening as those were not provisioned as cassandra hosts and the additional DNS records where not generated by the provisioning script.

Volans added a comment.EditedJun 9 2022, 3:45 PM

I've created the records for 2 cassandra instances (-a and -b) in Netbox.
Changelog: https://netbox.wikimedia.org/extras/changelog/?request_id=774eba10-813c-4882-b1cf-6602104a67da
Code executed:

>>> import uuid
>>> request_id = uuid.uuid4()
>>> user = User.objects.get(username='volans')
>>> def update(d):
...     prefix = Prefix.objects.get(prefix=d.primary_ip4.address.cidr)
...     for letter in ('a', 'b'):
...         extra_ip_address = prefix.get_first_available_ip()
...         extra_dns_name = f"{d.name}-{letter}.{d.site.slug}.wmnet"
...         address = IPAddress(
...             address=extra_ip_address,
...             status="active",
...             dns_name=extra_dns_name,
...             vrf=prefix.vrf.pk if prefix.vrf else None,
...             assigned_object=d.primary_ip4.assigned_object,
...             tenant=d.tenant,
...         )
...         address.save()
...         log = address.to_objectchange('create')
...         log.request_id = request_id
...         log.user = user
...         log.save()
...
>>> devices = Device.objects.filter(name__startswith='aqs2')
>>> len(devices)
12
>>> [d.name for d in devices]
['aqs2001', 'aqs2002', 'aqs2003', 'aqs2004', 'aqs2005', 'aqs2006', 'aqs2007', 'aqs2008', 'aqs2009', 'aqs2010', 'aqs2011', 'aqs2012']
>>> for d in devices:
...     update(d)
...

I've run the sre.dns.netbox cookbook to propagate those records to the DNS, they are now live.

Volans closed this task as Resolved.Jun 9 2022, 3:52 PM

Re-resolving.

Eevans mentioned this in T305570: Q4:(Need By: TBD) rack/setup/install aqs1016-aqs1021.Jun 10 2022, 6:11 PM
Volans mentioned this in T354893: Q3:rack/setup/install restbase10[34-42].Feb 21 2024, 11:38 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL