Page MenuHomePhabricator

Test haproxy as a WMF's CDN TLS terminator with real traffic
Open, In Progress, MediumPublic

Description

Haproxy is one of the candidates to replace ats-tls as the TLS terminator used in the WMF caching infrastructure. To fully validate its performance and stability a real traffic test is needed.
To be able to perform this test several requirements need to be fulfilled:

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+16 -1
operations/puppetproduction+108 -43
operations/puppetproduction+62 -6
operations/puppetproduction+8 -1
operations/puppetproduction+8 -1
operations/puppetproduction+2 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -10
operations/puppetproduction+5 -1
operations/puppetproduction+4 -2
operations/puppetproduction+1 -0
operations/puppetproduction+7 -0
operations/puppetproduction+9 -1
operations/puppetproduction+1 -1
operations/puppetproduction+15 -0
operations/puppetproduction+20 -0
operations/puppetproduction+9 -1
operations/puppetproduction+286 -0
operations/puppetproduction+2 -1
operations/puppetproduction+2 -1
operations/puppetproduction+1 -1
operations/puppetproduction+2 -1
operations/puppetproduction+120 -10
operations/puppetproduction+9 -1
operations/puppetproduction+0 -48
operations/puppetproduction+9 -1
operations/puppetproduction+5 -4
operations/puppetproduction+7 -0
operations/puppetproduction+2 -2
operations/puppetproduction+10 -0
operations/puppetproduction+9 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+53 -5
operations/puppetproduction+64 -2
operations/puppetproduction+1 -1
operations/puppetproduction+9 -1
operations/puppetproduction+7 -1
operations/puppetproduction+2 -2
operations/puppetproduction+1 -0
operations/puppetproduction+20 -0
operations/puppetproduction+6 -6
operations/puppetproduction+2 -1
operations/puppetproduction+4 -0
operations/puppetproduction+7 -1
operations/puppetproduction+20 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+23 -0
operations/puppetproduction+266 -0
operations/puppetproduction+23 -1
operations/puppetproduction+3 -2
operations/puppetproduction+2 -0
operations/puppetproduction+6 -1
operations/puppetproduction+16 -0
operations/puppetproduction+1 -1
operations/puppetproduction+121 -0
operations/puppetproduction+10 -0
operations/puppetproduction+34 -0
operations/puppetproduction+12 -0
operations/puppetproduction+20 -0
operations/puppetproduction+46 -0
operations/puppetproduction+56 -1
operations/puppetproduction+150 -7
operations/puppetproduction+31 -1
operations/puppetproduction+96 -0
operations/puppetproduction+41 -5
operations/puppetproduction+41 -5
operations/puppetproduction+16 -10
operations/puppetproduction+15 -6
operations/puppetproduction+8 -23
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2021-11-17T18:56:28Z] <vgutierrez> pool cp2042 (upload) running HAProxy as TLS terminator - T290005

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp2042.codfw.wmnet with OS buster completed:

  • cp2042 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111171805_vgutierrez_24768_cp2042.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Change 739754 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp1090 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/739754

Mentioned in SAL (#wikimedia-operations) [2021-11-18T08:46:24Z] <vgutierrez> depool cp1090 to be reimaged as cache::upload_haproxy - T290005

Change 739754 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp1090 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/739754

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp1090.eqiad.wmnet with OS buster

Mentioned in SAL (#wikimedia-operations) [2021-11-18T09:32:24Z] <vgutierrez> pool cp1090 (upload) running HAProxy as TLS terminator - T290005

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp1090.eqiad.wmnet with OS buster completed:

  • cp1090 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111180848_vgutierrez_4427_cp1090.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB

Change 738422 merged by Vgutierrez:

[operations/puppet@production] cache:haproxy: Gather TTFB metrics using mtail

https://gerrit.wikimedia.org/r/738422

Change 739775 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Fix unit_type for haproxy-mtail@tls.socket

https://gerrit.wikimedia.org/r/739775

Change 739775 abandoned by Vgutierrez:

[operations/puppet@production] cache::haproxy: Fix unit_type for haproxy-mtail@tls.socket

Reason:

not needed, even not possible with the current systemd::service puppetization

https://gerrit.wikimedia.org/r/739775

Change 739776 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] prometheus:ops: Fix cache_haproxy_tls_mtail yaml filename

https://gerrit.wikimedia.org/r/739776

Change 739776 merged by Vgutierrez:

[operations/puppet@production] prometheus:ops: Fix cache_haproxy_tls_mtail yaml filename

https://gerrit.wikimedia.org/r/739776

Change 739862 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Increase rsyslog config priority

https://gerrit.wikimedia.org/r/739862

Change 739862 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Increase rsyslog config priority

https://gerrit.wikimedia.org/r/739862

Change 740172 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Bypass systemd's journal for logging

https://gerrit.wikimedia.org/r/740172

Change 740172 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Bypass systemd's journal for logging

https://gerrit.wikimedia.org/r/740172

Change 740628 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] role:cache: Provide a text_haproxy role

https://gerrit.wikimedia.org/r/740628

Change 740638 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp4032 as cache::text_haproxy

https://gerrit.wikimedia.org/r/740638

Change 740628 merged by Vgutierrez:

[operations/puppet@production] role:cache: Provide a text_haproxy role

https://gerrit.wikimedia.org/r/740628

Mentioned in SAL (#wikimedia-operations) [2021-11-22T16:50:37Z] <vgutierrez> depol cp4032 to be reimaged as cache::text_haproxy - T290005

Change 740638 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp4032 as cache::text_haproxy

https://gerrit.wikimedia.org/r/740638

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp4032.ulsfo.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp4032.ulsfo.wmnet with OS buster completed:

  • cp4032 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111221654_vgutierrez_23511_cp4032.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB

Change 740652 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] prometheus::ops: Add cache::text_haproxy nodes

https://gerrit.wikimedia.org/r/740652

Change 740652 merged by Vgutierrez:

[operations/puppet@production] prometheus::ops: Add cache::text_haproxy nodes

https://gerrit.wikimedia.org/r/740652

Mentioned in SAL (#wikimedia-operations) [2021-11-22T19:01:29Z] <vgutierrez> pool cp4032 (text) using HAProxy as TLS terminator - T290005

Change 740781 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] prometheus::ops: Gather varnish mtail metrics on text_haproxy

https://gerrit.wikimedia.org/r/740781

Change 740781 merged by Vgutierrez:

[operations/puppet@production] prometheus::ops: Gather varnish mtail metrics on text_haproxy

https://gerrit.wikimedia.org/r/740781

Change 740783 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cumin: Add cache::text_haproxy nodes on A:cp-text

https://gerrit.wikimedia.org/r/740783

Change 740796 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp5012 as cache::text_haproxy

https://gerrit.wikimedia.org/r/740796

Change 740783 merged by Vgutierrez:

[operations/puppet@production] cumin: Add cache::text_haproxy nodes on A:cp-text

https://gerrit.wikimedia.org/r/740783

Mentioned in SAL (#wikimedia-operations) [2021-11-23T10:01:04Z] <vgutierrez> depool cp5012 to be reimaged as cache::text_haproxy - T290005

Change 740796 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp5012 as cache::text_haproxy

https://gerrit.wikimedia.org/r/740796

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp5012.eqsin.wmnet with OS buster

Mentioned in SAL (#wikimedia-operations) [2021-11-23T11:15:30Z] <vgutierrez> pool cp5012 (text) using HAProxy as TLS terminator - T290005

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp5012.eqsin.wmnet with OS buster completed:

  • cp5012 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111231008_vgutierrez_23189_cp5012.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Change 740893 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] haproxy:tls_teminator: Allow configuring http-use

https://gerrit.wikimedia.org/r/740893

Change 740893 merged by Vgutierrez:

[operations/puppet@production] haproxy:tls_teminator: Allow configuring http-use

https://gerrit.wikimedia.org/r/740893

Change 740898 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache:haproxy: Set http-reuse to always

https://gerrit.wikimedia.org/r/740898

Change 740898 merged by Vgutierrez:

[operations/puppet@production] cache:haproxy: Set http-reuse to always

https://gerrit.wikimedia.org/r/740898

Change 741083 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cp5006: Disable PROXY protocol

https://gerrit.wikimedia.org/r/741083

Change 741083 merged by Vgutierrez:

[operations/puppet@production] cp5006: Disable PROXY protocol

https://gerrit.wikimedia.org/r/741083

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:53:42Z] <vgutierrez> restart varnish/haproxy on cp5006 - T290005

Change 741109 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Fix UDS check cmd

https://gerrit.wikimedia.org/r/741109

Change 741109 merged by Vgutierrez:

[operations/puppet@production] varnish: Fix UDS check cmd

https://gerrit.wikimedia.org/r/741109

Change 741693 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Disable PROXY protocol

https://gerrit.wikimedia.org/r/741693

Mentioned in SAL (#wikimedia-operations) [2021-11-25T10:25:11Z] <vgutierrez> rolling restart of varnish and HAProxy on cp2042.codfw.wmnet,cp1090.eqiad.wmnet,cp[5012].eqsin.wmnet,cp3065.esams.wmnet,cp[4026,4032].ulsfo.wmnet' to disable PROXY protocol - T290005

Change 741693 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Disable PROXY protocol

https://gerrit.wikimedia.org/r/741693

Change 742122 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Set stat socket privileve level to admin

https://gerrit.wikimedia.org/r/742122

Change 742122 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Set stat socket privilege level to admin

https://gerrit.wikimedia.org/r/742122

Mentioned in SAL (#wikimedia-operations) [2021-11-26T11:25:50Z] <vgutierrez> restarting HAProxy on O:cache::(text|upload)_haproxy - T290005

Change 742128 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Relax HTTP parsing rules

https://gerrit.wikimedia.org/r/742128

Change 742128 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Relax HTTP parsing rules

https://gerrit.wikimedia.org/r/742128

Mentioned in SAL (#wikimedia-operations) [2021-11-26T12:21:33Z] <vgutierrez> restarting HAProxy on O:cache::upload_haproxy - T290005

Change 742422 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp2041 as cache::text_haproxy

https://gerrit.wikimedia.org/r/742422

Mentioned in SAL (#wikimedia-operations) [2021-11-29T08:56:40Z] <vgutierrez> depool cp2041 to be reimaged as cache::text_haproxy - T290005

Change 742422 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp2041 as cache::text_haproxy

https://gerrit.wikimedia.org/r/742422

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp2041.codfw.wmnet with OS buster

Mentioned in SAL (#wikimedia-operations) [2021-11-29T09:52:40Z] <vgutierrez> pool cp2041 with HAProxy as TLS terminator - T290005

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp2041.codfw.wmnet with OS buster completed:

  • cp2041 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111290859_vgutierrez_28363_cp2041.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB

Change 742429 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp3064 as cache::text_haproxy

https://gerrit.wikimedia.org/r/742429

Mentioned in SAL (#wikimedia-operations) [2021-11-29T10:01:08Z] <vgutierrez> depool cp3064 to be reimaged as cache::text_haproxy - T290005

Change 742429 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp3064 as cache::text_haproxy

https://gerrit.wikimedia.org/r/742429

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp3064.esams.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp3064.esams.wmnet with OS buster executed with errors:

  • cp3064 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp3064.esams.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp3064.esams.wmnet with OS buster completed:

  • cp3064 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111291045_vgutierrez_21874_cp3064.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2021-11-29T12:11:29Z] <vgutierrez> pool cp3064 (text) using HAProxy as TLS terminator - T290005

Change 742749 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::haproxy: Avoid using lua

https://gerrit.wikimedia.org/r/742749

Change 742749 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Avoid using lua

https://gerrit.wikimedia.org/r/742749

Mentioned in SAL (#wikimedia-operations) [2021-11-30T18:08:39Z] <vgutierrez> restart haproxy on cp3064 - T290005

Change 742785 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Listen on several Unix Domain Sockets

https://gerrit.wikimedia.org/r/742785

Change 742785 merged by Vgutierrez:

[operations/puppet@production] varnish: Listen on several Unix Domain Sockets

https://gerrit.wikimedia.org/r/742785

Mentioned in SAL (#wikimedia-operations) [2021-12-01T09:03:30Z] <vgutierrez> rolling restart of haproxy and varnish on O:cache::text_haproxy and O:cache::upload_haproxy - T290005

Mentioned in SAL (#wikimedia-operations) [2021-12-01T10:23:29Z] <vgutierrez> test haproxy_2.2.19-1~bpo10+1 on cp3064 - T290005

Mentioned in SAL (#wikimedia-operations) [2021-12-01T11:31:38Z] <vgutierrez> test HAProxy 2.4.9 on cp3064 - T290005

Mentioned in SAL (#wikimedia-operations) [2021-12-01T13:19:38Z] <vgutierrez> restore haproxy 2.2.9 on cp3064 - T290005

Change 742958 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp1089 as cache::text_haproxy

https://gerrit.wikimedia.org/r/742958

Mentioned in SAL (#wikimedia-operations) [2021-12-01T17:54:12Z] <vgutierrez> depool cp1089 to be reimaged as cache::text_haproxy - T290005

Change 742958 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp1089 as cache::text_haproxy

https://gerrit.wikimedia.org/r/742958

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp1089.eqiad.wmnet with OS buster

Mentioned in SAL (#wikimedia-operations) [2021-12-01T18:39:54Z] <vgutierrez> pool cp1089 using HAProxy as TLS terminator - T290005

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp1089.eqiad.wmnet with OS buster completed:

  • cp1089 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202112011758_vgutierrez_3536_cp1089.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB