Page MenuHomePhabricator
Create Task
Maniphest T290005

Test haproxy as a WMF's CDN TLS terminator with real traffic
Closed, ResolvedPublic
Actions

Description

Haproxy is one of the candidates to replace ats-tls as the TLS terminator used in the WMF caching infrastructure. To fully validate its performance and stability a real traffic test is needed.
To be able to perform this test several requirements need to be fulfilled:

Details

SubjectRepoBranchLines +/-
cache::haproxy: Switch from UNIX sockets to TCP on cp4032operations/puppetproduction+25 -0
site: Reimage cp5002 as cache::upload_haproxyoperations/puppetproduction+1 -5
cache::varnish: Merge repeating host data to common dataoperations/puppetproduction+0 -930
site: Reimage cp6001 as cache::upload_haproxyoperations/puppetproduction+11 -4
site: Reimage cp6009 as cache::text_haproxyoperations/puppetproduction+11 -5
site: Reimage cp6001 as cache::upload_haproxyoperations/puppetproduction+11 -4
site: Reimage cp6009 as cache::text_haproxyoperations/puppetproduction+11 -5
site: Reimage cp6002 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6010 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6003 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6011 as cache::text_haproxyoperations/puppetproduction+13 -3
site: Reimage cp6004 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6012 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6005 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6013 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3051 as cache::upload_haproxyoperations/puppetproduction+11 -5
site: Reimage cp6006 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3053 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6014 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3050 as cache::text_haproxyoperations/puppetproduction+11 -5
site: Reimage cp4021 as cache::upload_haproxyoperations/puppetproduction+11 -6
site: Reimage cp4033 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp4027 as cache::text_haproxyoperations/puppetproduction+11 -5
traffic: Add HAProxyEdgeTrafficDropoperations/alertsmaster+16 -0
site: Reimage cp3052 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp4035 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp5001 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp5007 as cache::text_haproxyoperations/puppetproduction+11 -5
site: Reimage cp5013 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6007 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp5015 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6007 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp5002 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6015 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp5008 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp4022 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3055 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp4028 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3054 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6008 as cache::upload_haproxyoperations/puppetproduction+16 -2
site: Reimage cp5003 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp6016 as cache::text_haproxyoperations/puppetproduction+17 -1
site: Reimage cp5009 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp4023 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3057 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp4029 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3056 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp2028 as cache::upload_haproxyoperations/puppetproduction+11 -5
site: Reimage cp2030 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp2032 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp2034 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp2027 as cache::text_haproxyoperations/puppetproduction+11 -5
site: Reimage cp2029 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp2031 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp2033 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp1076 as cache::upload_haproxyoperations/puppetproduction+11 -5
site: Reimage cp1078 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp1080 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp1082 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp1081 as cache::text_haproxyoperations/puppetproduction+11 -5
site: Reimage cp1079 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp1077 as cache::text_haproxyoperations/puppetproduction+12 -17
site: Reimage cp1075 as cache::text_haproxyoperations/puppetproduction+12 -2
cache::haproxy: Ensure that old HAProxy instances die after 5moperations/puppetproduction+2 -0
site: Reimage cp1083 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp2035 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3058 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp5010 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp4030 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp1085 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp2037 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3060 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp5016 as cache::text_haproxyoperations/puppetproduction+12 -2
prometheus:rules_ops: Provide HAProxy total responses metricsoperations/puppetproduction+4 -0
site: Reimage cp4036 as cache::text_haproxyoperations/puppetproduction+12 -2
site: Reimage cp1084 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp2036 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp1086 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp2038 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3059 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp4024 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp5004 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp3061 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp5014 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp4034 as cache::upload_haproxyoperations/puppetproduction+12 -2
site: Reimage cp1087 as cache::text_haproxyoperations/puppetproduction+10 -8
site: Reimage cp3062 as cache::text_haproxyoperations/puppetproduction+10 -8
cache::haproxy: Add captured request headers to log-formatoperations/puppetproduction+7 -7
mtail::cache_haproxy: Provide haproxy_client_healthcheck_ttfb histogramoperations/puppetproduction+19 -5
haproxy::tls_terminator: Log Host headeroperations/puppetproduction+1 -0
site: Reimage cp2039 as cache::text_haproxyoperations/puppetproduction+11 -9
site: Reimage cp5011 as cache::text_haproxyoperations/puppetproduction+10 -8
site: Reimage cp1088 as cache::upload_haproxyoperations/puppetproduction+10 -8
site: Reimage cp3063 as cache::upload_haproxyoperations/puppetproduction+10 -8
site: Reimage cp5005 as cache::haproxy_uploadoperations/puppetproduction+10 -8
site: Reimage cp2040 as cache::upload_haproxyoperations/puppetproduction+10 -8
site: Reimage cp4025 as cache::upload_haproxyoperations/puppetproduction+10 -13
cache::haproxy: Provide a haproxy-restart scriptoperations/puppetproduction+37 -0
cache::haproxy: Use HAProxy 2.4operations/puppetproduction+5 -7
aptrepo: Add thirdparty/haproxy24 componentoperations/puppetproduction+100 -0
prometheus: Aggregation rules for HAProxy TTFBoperations/puppetproduction+13 -0
mtail::cache_haproxy: Split TTFB bucket by X-Cache-Statusoperations/puppetproduction+16 -14
cache::haproxy: Log X-Cache-Statusoperations/puppetproduction+4 -1
haproxy::tls_terminator: Fix cpu-mapoperations/puppetproduction+1 -1
cache::haproxy: Limit HAProxy to the same NUMA node as the main NICoperations/puppetproduction+24 -0
haproxy::tls_terminator: Support IPv[46] backendsoperations/puppetproduction+30 -13
varnish: match listen_depth and net.core.soxmaxconnoperations/puppetproduction+1 -1
site: Reimage cp1089 as cache::text_haproxyoperations/puppetproduction+16 -1
varnish: Listen on several Unix Domain Socketsoperations/puppetproduction+108 -43
cache::haproxy: Avoid using luaoperations/puppetproduction+62 -6
site: Reimage cp3064 as cache::text_haproxyoperations/puppetproduction+8 -1
site: Reimage cp2041 as cache::text_haproxyoperations/puppetproduction+8 -1
cache::haproxy: Relax HTTP parsing rulesoperations/puppetproduction+2 -0
cache::haproxy: Set stat socket privilege level to adminoperations/puppetproduction+1 -1
cache::haproxy: Disable PROXY protocoloperations/puppetproduction+1 -10
varnish: Fix UDS check cmdoperations/puppetproduction+5 -1
cp5006: Disable PROXY protocoloperations/puppetproduction+4 -2
cache:haproxy: Set http-reuse to alwaysoperations/puppetproduction+1 -0
haproxy:tls_teminator: Allow configuring http-useoperations/puppetproduction+7 -0
site: Reimage cp5012 as cache::text_haproxyoperations/puppetproduction+9 -1
cumin: Add cache::text_haproxy nodes on A:cp-textoperations/puppetproduction+1 -1
prometheus::ops: Gather varnish mtail metrics on text_haproxyoperations/puppetproduction+15 -0
prometheus::ops: Add cache::text_haproxy nodesoperations/puppetproduction+20 -0
site: Reimage cp4032 as cache::text_haproxyoperations/puppetproduction+9 -1
role:cache: Provide a text_haproxy roleoperations/puppetproduction+286 -0
cache::haproxy: Bypass systemd's journal for loggingoperations/puppetproduction+2 -1
cache::haproxy: Increase rsyslog config priorityoperations/puppetproduction+2 -1
prometheus:ops: Fix cache_haproxy_tls_mtail yaml filenameoperations/puppetproduction+1 -1
cache::haproxy: Fix unit_type for haproxy-mtail@tls.socketoperations/puppetproduction+2 -1
cache:haproxy: Gather TTFB metrics using mtailoperations/puppetproduction+120 -10
site: Reimage cp1090 as cache::upload_haproxyoperations/puppetproduction+9 -1
Remove digicert-2020 from upload/haproxy nodesoperations/puppetproduction+0 -48
site: Reimage cp2042 as cache::upload_haproxyoperations/puppetproduction+9 -1
cache:haproxy: Disable check_haproxy based checksoperations/puppetproduction+5 -4
haproxy: Allow disabling check_haproxy based monitoringoperations/puppetproduction+7 -0
cumin: Add cache::upload_haproxy to cp aliasesoperations/puppetproduction+2 -2
prometheus:ops: Take role cache::upload_haproxy into accountoperations/puppetproduction+10 -0
site: Reimage cp3065 as cache::upload_haproxyoperations/puppetproduction+9 -1
cache:haproxy: Restore stats socket permissionsoperations/puppetproduction+1 -1
cache:haproxy: Enable hitless reloadsoperations/puppetproduction+1 -1
varnish: Mimick XFF behaviour with UDS + PROXY protocoloperations/puppetproduction+53 -5
cache:haproxy: Monitor HTTPS portoperations/puppetproduction+64 -2
cache:haproxy: Fix OCSP symlinkoperations/puppetproduction+1 -1
site: Reimage cp5006 as cache::upload_haproxyoperations/puppetproduction+9 -1
varnish: Run nrpe UDS check as rootoperations/puppetproduction+7 -1
varnish: Check remote.ip for local tls terminator detectionoperations/puppetproduction+2 -2
cache:haproxy: Bump ulimit -n for haproxyoperations/puppetproduction+1 -0
cache:haproxy: Bump maxconn limitsoperations/puppetproduction+20 -0
prometheus: Avoid filename collitions between haproxy jobsoperations/puppetproduction+6 -6
cache:haproxy: Sort puppet dependenciesoperations/puppetproduction+2 -1
hieradata: Enable UDS for varnish-fe@cp4026operations/puppetproduction+4 -0
site: Use role cache::upload_haproxy for cp4026operations/puppetproduction+7 -1
prometheus::ops: Add haproxy-tls@cache_upload configoperations/puppetproduction+20 -1
upload_haproxy: Adopt ::profile::base::productionoperations/puppetproduction+1 -1
cache:haproxy: Require haproxy bpo pin to install itoperations/puppetproduction+1 -0
cache: Expose prometheus metrics for HAProxyoperations/puppetproduction+23 -0
cache: Provide a HAproxy upload roleoperations/puppetproduction+266 -0
haproxy: Add missing TLS configuration optionsoperations/puppetproduction+23 -1
haproxy: Enable TFO by default for the tls terminatoroperations/puppetproduction+3 -2
cache::haproxy: Bring systemd service unit up to dateoperations/puppetproduction+2 -0
cache::haproxy: Fix missing_xwd ACLoperations/puppetproduction+6 -1
haproxy: Allow setting variablesoperations/puppetproduction+16 -0
profile::haproxy: Fix typo on X-Analytics-TLS valueoperations/puppetproduction+1 -1
cache::haproxy: Manage request/response headersoperations/puppetproduction+121 -0
haproxy: Allow loading lua scriptsoperations/puppetproduction+10 -0
haproxy: Allow adding/removing HTTP headersoperations/puppetproduction+34 -0
haproxy: Add PROXY protocol supportoperations/puppetproduction+12 -0
haproxy: Add H2 performance tuning settingsoperations/puppetproduction+20 -0
haproxy: Allow configuring timeoutsoperations/puppetproduction+46 -0
cache::haproxy: Configure sslcert::ocspoperations/puppetproduction+56 -1
haproxy: STEK supportoperations/puppetproduction+150 -7
haproxy: Allow configuring TLS optionsoperations/puppetproduction+31 -1
haproxy: Basic TLS terminator based on HAProxyoperations/puppetproduction+96 -0
sslcert: Provide chained TLS cert with private keyoperations/puppetproduction+41 -5
sslcert: Provide chained TLS cert with private keyoperations/puppetproduction+41 -5
haproxy: Accept systemd unit content instead of template pathoperations/puppetproduction+16 -10
haproxy: Allow using a custom systemd::service templateoperations/puppetproduction+15 -6
haproxy: Use systemd::serviceoperations/puppetproduction+8 -23
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Apr 7 2022, 8:09 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6014.drmrs.wmnet with OS buster

ops-monitoring-bot added a comment.Apr 7 2022, 8:56 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp3050.esams.wmnet with OS buster completed:

  • cp3050 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204070755_mmandere_65012_cp3050.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 7 2022, 9:01 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T09:01:11Z] <mmandere> pool cp3050 with HAProxy as TLS termination layer - T290005

ops-monitoring-bot added a comment.Apr 7 2022, 9:16 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6014.drmrs.wmnet with OS buster completed:

  • cp6014 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204070809_mmandere_67914_cp6014.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 7 2022, 9:20 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T09:20:12Z] <mmandere> pool cp6014 with HAProxy as TLS termination layer - T290005

Stashbot added a comment.Apr 7 2022, 9:25 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T09:25:26Z] <mmandere> depool cp3053 for reimage - T290005

gerritbot added a comment.Apr 7 2022, 9:30 AM

Change 777848 merged by MMandere:

[operations/puppet@production] site: Reimage cp3053 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/777848

ops-monitoring-bot added a comment.Apr 7 2022, 9:33 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp3053.esams.wmnet with OS buster

Stashbot added a comment.Apr 7 2022, 9:34 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T09:34:22Z] <mmandere> depool cp6006 for reimage - T290005

gerritbot added a comment.Apr 7 2022, 9:38 AM

Change 777849 merged by MMandere:

[operations/puppet@production] site: Reimage cp6006 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/777849

ops-monitoring-bot added a comment.Apr 7 2022, 9:39 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6006.drmrs.wmnet with OS buster

ops-monitoring-bot added a comment.Apr 7 2022, 10:37 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6006.drmrs.wmnet with OS buster completed:

  • cp6006 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204070939_mmandere_121256_cp6006.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 7 2022, 10:40 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T10:40:26Z] <mmandere> pool cp6006 with HAProxy as TLS termination layer - T290005

ops-monitoring-bot added a comment.Apr 7 2022, 10:55 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp3053.esams.wmnet with OS buster completed:

  • cp3053 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204070933_mmandere_114086_cp3053.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 7 2022, 10:59 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T10:59:34Z] <mmandere> pool cp3053 with HAProxy as TLS termination layer - T290005

Stashbot added a comment.Apr 7 2022, 11:23 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T11:23:30Z] <mmandere> depool cp3051 for reimage - T290005

gerritbot added a comment.Apr 7 2022, 11:32 AM

Change 777850 merged by MMandere:

[operations/puppet@production] site: Reimage cp3051 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/777850

ops-monitoring-bot added a comment.Apr 7 2022, 11:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp3051.esams.wmnet with OS buster

Stashbot added a comment.Apr 7 2022, 11:35 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T11:35:25Z] <mmandere> depool cp6013 for reimage - T290005

gerritbot added a comment.Apr 7 2022, 11:44 AM

Change 777851 merged by MMandere:

[operations/puppet@production] site: Reimage cp6013 as cache::text_haproxy

https://gerrit.wikimedia.org/r/777851

ops-monitoring-bot added a comment.Apr 7 2022, 11:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6013.drmrs.wmnet with OS buster

ops-monitoring-bot added a comment.Apr 7 2022, 12:30 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp3051.esams.wmnet with OS buster completed:

  • cp3051 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071133_mmandere_230902_cp3051.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 7 2022, 12:32 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T12:32:22Z] <mmandere> pool cp3051 with HAProxy as TLS termination layer - T290005

ops-monitoring-bot added a comment.Apr 7 2022, 12:52 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6013.drmrs.wmnet with OS buster completed:

  • cp6013 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071145_mmandere_243882_cp6013.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 7 2022, 12:55 PM

Mentioned in SAL (#wikimedia-traffic) [2022-04-07T12:54:59Z] <mmandere> pool cp6013 with HAProxy as TLS termination layer - T290005

Stashbot added a comment.Apr 7 2022, 12:58 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T12:58:56Z] <mmandere> depool cp6005 for reimage - T290005

gerritbot added a comment.Apr 7 2022, 1:09 PM

Change 777852 merged by MMandere:

[operations/puppet@production] site: Reimage cp6005 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/777852

ops-monitoring-bot added a comment.Apr 7 2022, 1:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6005.drmrs.wmnet with OS buster

Stashbot added a comment.Apr 7 2022, 1:13 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T13:13:46Z] <mmandere> depool cp6012 for reimage - T290005

gerritbot added a comment.Apr 7 2022, 1:19 PM

Change 777853 merged by MMandere:

[operations/puppet@production] site: Reimage cp6012 as cache::text_haproxy

https://gerrit.wikimedia.org/r/777853

ops-monitoring-bot added a comment.Apr 7 2022, 1:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6012.drmrs.wmnet with OS buster

ops-monitoring-bot added a comment.Apr 7 2022, 2:06 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6005.drmrs.wmnet with OS buster completed:

  • cp6005 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071311_mmandere_328202_cp6005.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 7 2022, 2:08 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T14:08:43Z] <mmandere> pool cp6005 with HAProxy as TLS termination layer - T290005

ops-monitoring-bot added a comment.Apr 7 2022, 2:10 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6012.drmrs.wmnet with OS buster completed:

  • cp6012 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071320_mmandere_335449_cp6012.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 7 2022, 2:13 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T14:13:02Z] <mmandere> pool cp6012 with HAProxy as TLS termination layer - T290005

Stashbot added a comment.Apr 7 2022, 2:19 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T14:19:22Z] <mmandere> depool cp6004 for reimage - T290005

gerritbot added a comment.Apr 7 2022, 2:22 PM

Change 777854 merged by MMandere:

[operations/puppet@production] site: Reimage cp6004 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/777854

ops-monitoring-bot added a comment.Apr 7 2022, 2:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6004.drmrs.wmnet with OS buster

gerritbot added a comment.Apr 7 2022, 3:07 PM

Change 778300 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Reimage cp6011 as cache::text_haproxy

https://gerrit.wikimedia.org/r/778300

gerritbot added a comment.Apr 7 2022, 3:07 PM

Change 778301 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Reimage cp6003 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/778301

gerritbot added a comment.Apr 7 2022, 3:07 PM

Change 778302 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Reimage cp6010 as cache::text_haproxy

https://gerrit.wikimedia.org/r/778302

gerritbot added a comment.Apr 7 2022, 3:07 PM

Change 778303 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Reimage cp6002 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/778303

gerritbot added a comment.Apr 7 2022, 3:08 PM

Change 778304 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Reimage cp6009 as cache::text_haproxy

https://gerrit.wikimedia.org/r/778304

gerritbot added a comment.Apr 7 2022, 3:08 PM

Change 778305 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Reimage cp6001 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/778305

ops-monitoring-bot added a comment.Apr 7 2022, 3:12 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6004.drmrs.wmnet with OS buster completed:

  • cp6004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204071424_mmandere_400225_cp6004.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 7 2022, 3:21 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-07T15:21:32Z] <mmandere> pool cp6004 with HAProxy as TLS termination layer - T290005

Stashbot added a comment.Apr 8 2022, 7:12 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T07:12:40Z] <mmandere> depool cp6011 for reimage - T290005

gerritbot added a comment.Apr 8 2022, 7:15 AM

Change 778300 merged by MMandere:

[operations/puppet@production] site: Reimage cp6011 as cache::text_haproxy

https://gerrit.wikimedia.org/r/778300

ops-monitoring-bot added a comment.Apr 8 2022, 7:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6011.drmrs.wmnet with OS buster

Stashbot added a comment.Apr 8 2022, 7:21 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T07:21:47Z] <mmandere> depool cp6003 for reimage - T290005

gerritbot added a comment.Apr 8 2022, 7:24 AM

Change 778301 merged by MMandere:

[operations/puppet@production] site: Reimage cp6003 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/778301

ops-monitoring-bot added a comment.Apr 8 2022, 7:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6003.drmrs.wmnet with OS buster

ops-monitoring-bot added a comment.Apr 8 2022, 8:24 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6011.drmrs.wmnet with OS buster completed:

  • cp6011 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080720_mmandere_671714_cp6011.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 8 2022, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T08:26:34Z] <mmandere> pool cp6011 with HAProxy as TLS termination layer - T290005

ops-monitoring-bot added a comment.Apr 8 2022, 8:37 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6003.drmrs.wmnet with OS buster completed:

  • cp6003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080728_mmandere_672261_cp6003.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 8 2022, 8:41 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T08:41:55Z] <mmandere> pool cp6003 with HAProxy as TLS termination layer - T290005

Stashbot added a comment.Apr 8 2022, 8:48 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T08:48:03Z] <mmandere> depool cp6010 for reimage - T290005

gerritbot added a comment.Apr 8 2022, 8:51 AM

Change 778302 merged by MMandere:

[operations/puppet@production] site: Reimage cp6010 as cache::text_haproxy

https://gerrit.wikimedia.org/r/778302

ops-monitoring-bot added a comment.Apr 8 2022, 8:57 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6010.drmrs.wmnet with OS buster

Stashbot added a comment.Apr 8 2022, 9:02 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T09:02:10Z] <mmandere> depool cp6002 for reimage - T290005

gerritbot added a comment.Apr 8 2022, 9:08 AM

Change 778303 merged by MMandere:

[operations/puppet@production] site: Reimage cp6002 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/778303

ops-monitoring-bot added a comment.Apr 8 2022, 9:13 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6002.drmrs.wmnet with OS buster

ops-monitoring-bot added a comment.Apr 8 2022, 10:05 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6002.drmrs.wmnet with OS buster completed:

  • cp6002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080913_mmandere_692372_cp6002.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 8 2022, 10:07 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6010.drmrs.wmnet with OS buster completed:

  • cp6010 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204080856_mmandere_689142_cp6010.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 8 2022, 10:11 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T10:11:20Z] <mmandere> pool cp6010 with HAProxy as TLS termination layer - T290005

Stashbot added a comment.Apr 8 2022, 10:18 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T10:18:40Z] <mmandere> pool cp6002 with HAProxy as TLS termination layer - T290005

Stashbot added a comment.Apr 8 2022, 11:11 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T11:11:21Z] <mmandere> depool cp6009 for reimage - T290005

gerritbot added a comment.Apr 8 2022, 11:32 AM

Change 778490 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Reimage cp6009 as cache::text_haproxy

https://gerrit.wikimedia.org/r/778490

gerritbot added a comment.Apr 8 2022, 11:32 AM

Change 778491 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Reimage cp6001 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/778491

gerritbot added a comment.Apr 8 2022, 12:08 PM

Change 778304 merged by MMandere:

[operations/puppet@production] site: Reimage cp6009 as cache::text_haproxy

https://gerrit.wikimedia.org/r/778304

ops-monitoring-bot added a comment.Apr 8 2022, 12:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6009.drmrs.wmnet with OS buster

Stashbot added a comment.Apr 8 2022, 12:15 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T12:15:45Z] <mmandere> depool cp6001 for reimage - T290005

gerritbot added a comment.Apr 8 2022, 12:19 PM

Change 778305 merged by MMandere:

[operations/puppet@production] site: Reimage cp6001 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/778305

ops-monitoring-bot added a comment.Apr 8 2022, 12:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp6001.drmrs.wmnet with OS buster

gerritbot added a comment.Apr 8 2022, 12:43 PM

Change 778490 abandoned by MMandere:

[operations/puppet@production] site: Reimage cp6009 as cache::text_haproxy

Reason:

Conflicts with already merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/778304

https://gerrit.wikimedia.org/r/778490

gerritbot added a comment.Apr 8 2022, 12:44 PM

Change 778491 abandoned by MMandere:

[operations/puppet@production] site: Reimage cp6001 as cache::upload_haproxy

Reason:

Conflicts with already merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/778305

https://gerrit.wikimedia.org/r/778491

ops-monitoring-bot added a comment.Apr 8 2022, 1:11 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6009.drmrs.wmnet with OS buster completed:

  • cp6009 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204081211_mmandere_720511_cp6009.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 8 2022, 1:13 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp6001.drmrs.wmnet with OS buster completed:

  • cp6001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204081222_mmandere_721438_cp6001.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 8 2022, 1:16 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T13:16:01Z] <mmandere> pool cp6009 with HAProxy as TLS termination layer - T290005

Stashbot added a comment.Apr 8 2022, 1:20 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-08T13:20:54Z] <mmandere> pool cp6001 with HAProxy as TLS termination layer - T290005

gerritbot added a comment.Apr 11 2022, 9:23 AM

Change 778989 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] cache::varnish: Merge repeating host data to site data

https://gerrit.wikimedia.org/r/778989

gerritbot added a comment.Apr 11 2022, 11:09 AM

Change 778989 merged by MMandere:

[operations/puppet@production] cache::varnish: Merge repeating host data to common data

https://gerrit.wikimedia.org/r/778989

• MMandere changed the status of subtask T304153: Clean up Traffic Grafana dashboards to reflect HA-Proxy metrics from Open to In Progress.Apr 19 2022, 1:52 PM
gerritbot added a comment.May 4 2022, 1:54 AM

Change 788893 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site: Reimage cp5002 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/788893

gerritbot added a comment.May 4 2022, 1:55 AM

Change 788893 merged by Ssingh:

[operations/puppet@production] site: Reimage cp5002 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/788893

ops-monitoring-bot added a comment.May 4 2022, 1:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5002.eqsin.wmnet with OS buster

ops-monitoring-bot added a comment.May 4 2022, 2:32 AM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5002.eqsin.wmnet with OS buster executed with errors:

  • cp5002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"cp5002.eqsin.wmnet": {"weight": 100, "pooled": "no"}, "tags": "dc=eqsin,cluster=cache_upload,service=ats-be"}
{"cp5002.eqsin.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=eqsin,cluster=cache_upload,service=ats-tls"}
{"cp5002.eqsin.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=eqsin,cluster=cache_upload,service=varnish-fe"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=eqsin,cluster=cache_upload,service=ats-be' set/pooled=no
sudo confctl select 'dc=eqsin,cluster=cache_upload,service=ats-tls' set/pooled=no
sudo confctl select 'dc=eqsin,cluster=cache_upload,service=varnish-fe' set/pooled=no

  • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 4 2022, 2:41 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5002.eqsin.wmnet with OS buster

ops-monitoring-bot added a comment.May 4 2022, 3:11 AM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5002.eqsin.wmnet with OS buster executed with errors:

  • cp5002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 4 2022, 1:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5002.eqsin.wmnet with OS buster

ops-monitoring-bot added a comment.May 4 2022, 2:17 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5002.eqsin.wmnet with OS buster executed with errors:

  • cp5002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 4 2022, 2:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5002.eqsin.wmnet with OS buster

ops-monitoring-bot added a comment.May 4 2022, 4:03 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5002.eqsin.wmnet with OS buster completed:

  • cp5002 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205041444_sukhe_1735192_cp5002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
BCornwall mentioned this in T249335: Memory leak on ats-tls 8.0.6.Sep 19 2022, 8:54 PM
Vgutierrez closed this task as Resolved.Oct 24 2022, 8:28 AM
Vgutierrez claimed this task.
Vgutierrez mentioned this in T323365: Rename role::cache::(text|upload)_haproxy to role::cache::(text|upload).Nov 18 2022, 11:18 AM
Vgutierrez closed subtask T323365: Rename role::cache::(text|upload)_haproxy to role::cache::(text|upload) as Resolved.Nov 21 2022, 4:05 PM
BCornwall closed subtask T304153: Clean up Traffic Grafana dashboards to reflect HA-Proxy metrics as Invalid.May 1 2023, 9:01 PM
BTullis mentioned this in T351117: Move analytics log from Varnish to HAProxy.Nov 17 2023, 1:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits