Page MenuHomePhabricator
Create Task
Maniphest T321309

Upgrade Traffic hosts to bullseye
Closed, ResolvedPublic
Actions

Description

This task tracks the upgrade of the Traffic hosts to bullseye, affecting the services below (identified by their cumin aliases). There is no particular order but we will be doing the cp hosts first to resolve T319067.

Progress (order of execution):

HostDebian PackagesReimaging
cp✓✓
acmechief✓✓
ncredir✓✓
dns-rec✓✓
dns-auth✓✓
durum✓✓
Wikidough✓✓
lvs✓✓
lvs experimental✓✓
pybal-test✓✓
  • A:cp

Debian packages upgraded:

  • varnish_6.0.10-1wm2
  • trafficserver_9.1.3-1wm3
  • fifo-log-demux_0.6.3
  • file-read-backwards_2.0.0-3
  • prometheus-rdkafka-exporter_0.3
  • python-logstash_0.4.6-3
  • prometheus-varnishkafka-exporter_0.1-2
  • varnishkafka_1.1.0-2
  • libvmod-netmapper_1.9-2
  • libvmod-querysort_0.3
  • purged_0.19
  • libvmod-re2_1.5.3-3
  • varnish-modules_0.15.0-2
  • A:dns-auth
  • gdnsd_3.8.0-1~wmf2

eqiad

text

  • cp1075.eqiad.wmnet
  • cp1077.eqiad.wmnet
  • cp1079.eqiad.wmnet
  • cp1081.eqiad.wmnet
  • cp1083.eqiad.wmnet
  • cp1085.eqiad.wmnet
  • cp1087.eqiad.wmnet
  • cp1089.eqiad.wmnet

eqiad

upload

  • cp1076.eqiad.wmnet
  • cp1078.eqiad.wmnet
  • cp1080.eqiad.wmnet
  • cp1082.eqiad.wmnet
  • cp1084.eqiad.wmnet
  • cp1086.eqiad.wmnet
  • cp1088.eqiad.wmnet
  • cp1090.eqiad.wmnet

eqsin

text

  • cp5017.eqsin.wmnet
  • cp5018.eqsin.wmnet
  • cp5019.eqsin.wmnet
  • cp5020.eqsin.wmnet
  • cp5021.eqsin.wmnet
  • cp5022.eqsin.wmnet
  • cp5023.eqsin.wmnet
  • cp5024.eqsin.wmnet

eqsin

upload

  • cp5025.eqsin.wmnet
  • cp5026.eqsin.wmnet
  • cp5027.eqsin.wmnet
  • cp5028.eqsin.wmnet
  • cp5029.eqsin.wmnet
  • cp5030.eqsin.wmnet
  • cp5031.eqsin.wmnet

codfw

text

  • cp2027.codfw.wmnet
  • cp2029.codfw.wmnet
  • cp2031.codfw.wmnet
  • cp2033.codfw.wmnet
  • cp2035.codfw.wmnet
  • cp2037.codfw.wmnet
  • cp2039.codfw.wmnet

codfw

upload

  • cp2028.codfw.wmnet
  • cp2030.codfw.wmnet
  • cp2032.codfw.wmnet
  • cp2034.codfw.wmnet
  • cp2036.codfw.wmnet
  • cp2038.codfw.wmnet
  • cp2040.codfw.wmnet

esams

text

  • cp3050.esams.wmnet
  • cp3052.esams.wmnet
  • cp3054.esams.wmnet
  • cp3056.esams.wmnet
  • cp3058.esams.wmnet
  • cp3060.esams.wmnet
  • cp3062.esams.wmnet
  • cp3064.esams.wmnet

esams

upload

  • cp3051.esams.wmnet
  • cp3053.esams.wmnet
  • cp3055.esams.wmnet
  • cp3057.esams.wmnet
  • cp3059.esams.wmnet
  • cp3061.esams.wmnet
  • cp3063.esams.wmnet
  • cp3065.esams.wmnet

ulsfo

text

  • cp4037.ulsfo.wmnet
  • cp4038.ulsfo.wmnet
  • cp4039.ulsfo.wmnet
  • cp4040.ulsfo.wmnet
  • cp4041.ulsfo.wmnet
  • cp4042.ulsfo.wmnet
  • cp4043.ulsfo.wmnet
  • cp4044.ulsfo.wmnet

ulsfo

upload

  • cp4045.ulsfo.wmnet
  • cp4046.ulsfo.wmnet
  • cp4047.ulsfo.wmnet
  • cp4048.ulsfo.wmnet
  • cp4049.ulsfo.wmnet
  • cp4050.ulsfo.wmnet
  • cp4051.ulsfo.wmnet
  • cp4052.ulsfo.wmnet

drmrs

text

  • cp6009.drmrs.wmnet
  • cp6010.drmrs.wmnet
  • cp6011.drmrs.wmnet
  • cp6012.drmrs.wmnet
  • cp6013.drmrs.wmnet
  • cp6014.drmrs.wmnet
  • cp6015.drmrs.wmnet
  • cp6016.drmrs.wmnet

drmrs

upload

  • cp6001.drmrs.wmnet
  • cp6002.drmrs.wmnet
  • cp6003.drmrs.wmnet
  • cp6004.drmrs.wmnet
  • cp6005.drmrs.wmnet
  • cp6006.drmrs.wmnet
  • cp6007.drmrs.wmnet
  • cp6008.drmrs.wmnet
  • A:dns-auth
  • A:dns-rec
  • dns1001.wikimedia.org
  • dns1002.wikimedia.org
  • dns2001.wikimedia.org
  • dns2002.wikimedia.org
  • dns3001.wikimedia.org
  • dns3002.wikimedia.org
  • dns4003.wikimedia.org
  • dns4004.wikimedia.org
  • dns5003.wikimedia.org
  • dns5004.wikimedia.org
  • dns6001.wikimedia.org
  • dns6002.wikimedia.org
  • A:acmechief
  • acmechief1001.eqiad.wmnet
  • acmechief2001.codfw.wmnet
  • acmechief-test1001.eqiad.wmnet
  • acmechief-test2001.codfw.wmnet

Debian packages upgraded:

  • acme-chief_0.36-1
  • A:ncredir
  • ncredir1001.eqiad.wmnet
  • ncredir1002.eqiad.wmnet
  • ncredir2001.codfw.wmnet
  • ncredir2002.codfw.wmnet
  • ncredir3001.esams.wmnet
  • ncredir3002.esams.wmnet
  • ncredir4001.ulsfo.wmnet
  • ncredir4002.ulsfo.wmnet
  • ncredir5001.eqsin.wmnet
  • ncredir5002.eqsin.wmnet
  • ncredir6001.drmrs.wmnet
  • ncredir6002.drmrs.wmnet
  • A:lvs
  • A:durum
  • durum1001.eqiad.wmnet
  • durum1002.eqiad.wmnet
  • durum2001.codfw.wmnet
  • durum2002.codfw.wmnet
  • durum3001.esams.wmnet
  • durum3002.esams.wmnet
  • durum4001.ulsfo.wmnet
  • durum4002.ulsfo.wmnet
  • durum5001.eqsin.wmnet
  • durum5002.eqsin.wmnet
  • durum6001.drmrs.wmnet
  • durum6002.drmrs.wmnet
  • A:wikidough
  • doh1001.wikimedia.org
  • doh1002.wikimedia.org
  • doh2001.wikimedia.org
  • doh2002.wikimedia.org
  • doh3001.wikimedia.org
  • doh3002.wikimedia.org
  • doh4001.wikimedia.org
  • doh4002.wikimedia.org
  • doh5001.wikimedia.org
  • doh5002.wikimedia.org
  • doh6001.wikimedia.org
  • doh6002.wikimedia.org
  • A:lvs
  • pybal_1.15.10+deb11u1

iDRAC firmware should be 6.10.00.00 [for bullseye installer]
NIC firmware should be 21.85.21.92 [for bullseye installer]

We should upgrade the iDRAC as well in this round.

HostiDRAC FirmwareNIC Firmware
lvs1017.eqiad.wmnet6.10.00.0021.85.21.91
lvs1018.eqiad.wmnet6.10.00.0021.85.21.91
lvs1019.eqiad.wmnet6.10.00.0021.85.21.91
lvs1020.eqiad.wmnet6.10.00.0021.85.21.91
lvs2007.codfw.wmnet6.10.00.0021.40.16.60
lvs2008.codfw.wmnet6.10.00.0021.40.16.60
lvs2009.codfw.wmnet6.10.00.0021.40.25.31
lvs2010.codfw.wmnet6.10.00.0021.85.21.92
lvs3005.esams.wmnet6.10.00.0021.40.22.20
lvs3006.esams.wmnet6.10.00.0021.40.22.20
lvs3007.esams.wmnet6.10.00.0021.40.22.20
lvs4008.ulsfo.wmnet6.10.00.0021.85.21.92
lvs4009.ulsfo.wmnet6.10.00.0021.85.21.92
lvs4010.ulsfo.wmnet6.10.00.0021.85.21.92
lvs5004.eqsin.wmnet5.10.30.0021.85.21.92
lvs5005.eqsin.wmnet5.10.30.0021.85.21.92
lvs5006.eqsin.wmnet5.10.30.0021.85.21.92
lvs6001.drmrs.wmnet6.10.00.0021.85.21.92
lvs6002.drmrs.wmnet6.10.00.0021.85.21.92
lvs6003.drmrs.wmnet6.10.00.0021.85.21.92
  • lvs1017.eqiad.wmnet
  • lvs1018.eqiad.wmnet
  • lvs1019.eqiad.wmnet
  • lvs1020.eqiad.wmnet
  • lvs2007.codfw.wmnet [definitely needs NIC firmware upgrade, firmware was upgraded]
  • lvs2008.codfw.wmnet [definitely needs NIC firmware upgrade, firmware was upgraded]
  • lvs2009.codfw.wmnet [definitely needs NIC firmware upgrade, firmware was upgraded]
  • lvs2010.codfw.wmnet [definitely needs NIC firmware upgrade, firmware was upgraded]
  • lvs3005.esams.wmnet [definitely needs NIC firmware upgrade, firmware was upgraded]
  • lvs3006.esams.wmnet [definitely needs NIC firmware upgrade, firmware was upgraded]
  • lvs3007.esams.wmnet [definitely needs NIC firmware upgrade, firmware was upgraded]
  • lvs4008.ulsfo.wmnet
  • lvs4009.ulsfo.wmnet
  • lvs4010.ulsfo.wmnet
  • lvs5004.eqsin.wmnet
  • lvs5005.eqsin.wmnet
  • lvs5006.eqsin.wmnet
  • lvs6001.drmrs.wmnet
  • lvs6002.drmrs.wmnet
  • lvs6003.drmrs.wmnet

lvs (experimental, L4LB):

  • lvs1013.eqiad.wmnet
  • lvs1014.eqiad.wmnet
  • lvs1015.eqiad.wmnet
  • lvs1016.eqiad.wmnet

pybal-test:

  • pybal-test2003.codfw.wmnet

This is meant to be an umbrella task for all changes that will be part of this upgrade, such as the Debian packaging, Puppet changes, and the related testing include reimaging.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -1acme-chief: Fix PASSIVE_FQDN syntax
operations/puppetproduction+6 -31pybal/lvs: remove backward compatibility for buster
operations/puppetproduction+10 -48hiera: lvs/balancer: unify hiera post bullseye upgrade (eqiad)
operations/dnsmaster+2 -0depool eqiad (emergency patch, do not merge)
operations/puppetproduction+0 -2hiera: remove lvs1017's bgp-med override
operations/puppetproduction+15 -13hiera: lvs1017: update iface names for bullseye (eqiad)
operations/puppetproduction+0 -2hiera: remove lvs1018's bgp-med override
operations/puppetproduction+16 -14hiera: lvs1018: update iface names for bullseye (eqiad)
operations/puppetproduction+0 -2hiera: remove lvs1019's bgp-med override
operations/puppetproduction+15 -13hiera: lvs1019: update iface names for bullseye (eqiad)
operations/puppetproduction+4 -40hiera: lvs/balancer: unify hiera post bullseye upgrade (codfw)
operations/puppetproduction+14 -14hiera: lvs1020: update iface names for bullseye (eqiad)
operations/puppetproduction+0 -1hierdata: Remove bgp-med for lvs2009
operations/puppetproduction+0 -1hiera: remove bgp-med override for lvs2009
operations/puppetproduction+12 -11hiera: lvs2009: update iface names for bullseye
operations/puppetproduction+0 -1hierdata: Remove bgp-med for lvs2008
operations/puppetproduction+12 -11hiera: lvs2008: update iface names for bullseye
operations/puppetproduction+0 -1hiera: remove bgp-med override for lvs2007
operations/puppetproduction+12 -11hiera: lvs2007: update iface names for bullseye
operations/puppetproduction+11 -11hiera: lvs2010: update iface names for bullseye (codfw)
operations/puppetproduction+2 -11hiera: lvs/balancer: unify hiera post bullseye upgrade (esams)
operations/puppetproduction+2 -2hiera: lvs/interfaces: update lvs3006 iface name
operations/puppetproduction+2 -18hiera: lvs/balancer: unify hiera post bullseye upgrade (drmrs)
operations/puppetproduction+1 -11hiera: lvs/balancer: unify hiera post bullseye upgrade (eqsin)
operations/puppetproduction+2 -2hiera: lvs/interfaces: update lvs3005 iface name
operations/puppetproduction+8 -3hiera: lvs/interfaces: update lvs6002 iface name
operations/puppetproduction+8 -3hiera: lvs/interfaces: update 6001 iface name
operations/puppetproduction+1 -11hiera: lvs/balancer: unify hiera post bullseye upgrade
operations/puppetproduction+2 -2hiera: lvs3007: update iface names for bullseye (esams)
operations/puppetproduction+3 -3hiera: update lvs6003 interfaces in common/interfaces.yaml
operations/puppetproduction+1 -1hiera: lvs6003: update interface name
operations/puppetproduction+6 -0hiera: lvs6003: update iface names for bullseye
operations/puppetproduction+4 -1hiera: lvs/interfaces: update 5005 iface name
operations/puppetproduction+4 -1hiera: lvs/interfaces: update 5004 iface name
operations/puppetproduction+4 -1hiera: lvs/interfaces: update lvs4009 iface name
operations/puppetproduction+4 -1hiera: lvs/interfaces: update lvs4008 iface name
operations/puppetproduction+2 -2fixup! hiera: lvs/interfaces: update lvs5006 iface name
operations/puppetproduction+5 -1hiera: lvs/interfaces: update lvs5006 iface name
operations/puppetproduction+1 -0hiera: set bpg-med to 101 for lvs4008 (100 for lvs4010)
operations/puppetproduction+14 -13pybal: port check_pybal_ipvs_diff.py to urllib2
operations/puppetproduction+3 -7pybal: don't install python3-requests on bullseye hosts
operations/puppetproduction+8 -4pybal: install prometheus-client from component
operations/puppetproduction+1 -1hiera: lvs/interfaces: update interface name for lvs4010
operations/puppetproduction+15 -6P:lvs: do not enable legacy vlan names for bullseye
operations/puppetproduction+14 -2pybal: add conditional for Python{2,3} packages
operations/puppetproduction+12 -2pybal: install pybal from component in bullseye
operations/puppetproduction+4 -0hiera: update interface names for lvs4010 (bullseye)
operations/puppetproduction+1 -0Set profile::base::remove_python2_on_bullseye for the LVSes
operations/puppetproduction+1 -7auditd: remove obsolete buster code
operations/puppetproduction+7 -1auditd: update location of audisp syslog.conf
operations/puppetproduction+1 -1hieradata/common: acmechief_host as acmechief2001
operations/puppetproduction+2 -2acmechief: Set acmechief2001 as active
operations/puppetproduction+4 -4acmechief-test: Set acmechief-test2001 as active
operations/puppetproduction+4 -2acmechief: Ensure rsync is installed
operations/puppetproduction+1 -0prometheus: update pdnsrec job for bullseye upgrade of dnsrec hosts
operations/puppetproduction+0 -2hieradata/common: Remove dns1001/2001 from authdns
operations/puppetproduction+4 -6acme_chief: Enforce passive_hosts as a list of FQDN
operations/puppetproduction+15 -11acme_chief: support several passive hosts
operations/homer/publicmaster+6 -1config: Add brett for router configuration
operations/dnsmaster+1 -1ntp/ulsfo: set to dns4004
operations/puppetproduction+2 -2prometheus: update ensure_packages for node_gdnsd (bullseye)
operations/puppetproduction+1 -4prometheus: refactor prometheus-gdnsd-stats.py to Python 3
operations/puppetproduction+4 -1P:dns::recursor: set webserver port to 9199
operations/puppetproduction+3 -2dnsrecursor: enable webserver for bullseye installation of pdns-rec
operations/puppetproduction+1 -1P:dns::auth::update: remove trailing slash for git safe.directory
operations/puppetproduction+21 -0P:dns::auth::update: add safe.directory for gdnsd bootstrapping
operations/puppetproduction+4 -1P:dns::recursor: skip installation of prometheus-pdns-rec-exporter
operations/puppetproduction+0 -2hiera: temporarily remove references to dns4004
operations/puppetproduction+9 -1dnsrecursor: update template to prepare for bullseye upgrade
operations/debs/gdnsdmaster+613 -0Release 3.8.0-1~wmf2
integration/configmaster+1 -0Zuul: [operations/debs/gdnsd] Add debian-glue CI
operations/puppetproduction+0 -1esitest: remove deprecated nbproc config option
operations/software/varnish/libvmod-querysortmain+10 -1Release 0.3
operations/software/acme-chiefdebian+9 -16debian: Add release 0.36 to changelog
operations/software/acme-chiefdebian+2 -2Release 0.36
operations/software/acme-chiefdebian+24 -79setup.py: update dependencies for bullseye
operations/software/acme-chiefmaster+2 -2Release 0.36
operations/software/acme-chiefmaster+24 -79setup.py: update dependencies for bullseye
integration/configmaster+1 -1jjb: Use tox-acme-chief:0.7.0 on acme-chief job
integration/configmaster+8 -2dockerfiles: Use tox-buster on tox-acme-chief
operations/software/acme-chiefdebian+12 -17Release 0.35-2
integration/configmaster+1 -0zuul: configure CI for operations/debs/varnish-modules
operations/puppetproduction+5 -5monitoring: Update check_fresh_files_in_dir for python3
operations/puppetproduction+2 -0Pull in the fdisk-udeb in d-i
operations/debs/varnish-modulesmaster+466 -0Release 0.15.0-2
operations/puppetproduction+1 -1sslcert::update-ocsp: Stop using SafeConfigParser
operations/puppetproduction+2 -1sslcert: add text=True to Popen for update-ocsp.py
operations/puppetproduction+4 -24varnish::common: set Python version for bullseye
operations/puppetproduction+6 -6sslcert: refactor update-ocsp.py to Python 3
operations/software/varnish/libvmod-re2debian-6.0+93 -3Release 1.5.3-3
operations/software/purgedmaster+468 -30Release 0.19
operations/software/varnish/libvmod-re2debian-6.0+15 -5Release 1.5.3-2
operations/software/varnish/libvmod-netmapperdebian+18 -5Release 1.9-2
operations/software/varnish/varnishkafkadebian+13 -4Release 1.1.0-2
operations/puppetproduction+0 -25package_builder: remove deprecated Varnish6 hooks
operations/debs/varnish4debian-wmf+12 -3Release 6.0.10-1wm2
operations/puppetproduction+13 -0package_builder: add hook for varnish6 (bullseye)
operations/debs/prometheus-varnishkafka-exportermaster+13 -3Release 0.1-2
operations/debs/file-read-backwardsdebian+13 -9Release 2.0.0-3
operations/software/prometheus-rdkafka-exportermaster+11 -3Release 0.3
operations/debs/python-logstashmaster+13 -16Release 0.4.6-3
operations/software/fifo-log-demuxmaster+13 -3Release 0.6.3
operations/puppetproduction+1 -0aptrepo: add trafficserver9 to bullseye-wikimedia
operations/debs/trafficservermaster+13 -2Release 9.1.3-1wm3
operations/puppetproduction+1 -1package_builder: add deb-src for buster and bookworm
operations/puppetproduction+11 -0aptrepo: add thirdparty/haproxy24 for bullseye
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Apr 12 2023, 8:58 PM

Change 908322 merged by BCornwall:

[operations/puppet@production] hiera: lvs2007: update iface names for bullseye

https://gerrit.wikimedia.org/r/908322

ops-monitoring-bot added a comment.Apr 12 2023, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host lvs2007.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 12 2023, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host lvs2007.codfw.wmnet with OS bullseye executed with errors:

  • lvs2007 (FAIL)
    • The reimage failed, see the cookbook logs for the details
Maintenance_bot removed a project: Patch-For-Review.Apr 12 2023, 9:10 PM
ops-monitoring-bot added a comment.Apr 12 2023, 9:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host lvs2007.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 12 2023, 9:56 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host lvs2007.codfw.wmnet with OS bullseye completed:

  • lvs2007 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304122116_brett_479794_lvs2007.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 13 2023, 1:51 PM

Change 908552 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: remove bgp-med override for lvs2007

https://gerrit.wikimedia.org/r/908552

gerritbot added a project: Patch-For-Review.Apr 13 2023, 1:51 PM
gerritbot added a comment.Apr 13 2023, 2:25 PM

Change 908552 merged by Ssingh:

[operations/puppet@production] hiera: remove bgp-med override for lvs2007

https://gerrit.wikimedia.org/r/908552

Maintenance_bot removed a project: Patch-For-Review.Apr 13 2023, 2:31 PM
BCornwall updated the task description. (Show Details)Apr 13 2023, 3:36 PM
gerritbot added a comment.Apr 13 2023, 3:38 PM

Change 908585 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: lvs2008: update iface names for bullseye

https://gerrit.wikimedia.org/r/908585

gerritbot added a project: Patch-For-Review.Apr 13 2023, 3:38 PM
Stashbot added a comment.Apr 13 2023, 3:46 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-13T15:46:40Z] <brett> Disable Puppet/PyBal on lvs2008 in preparation for reimaging - T321309

BCornwall updated the task description. (Show Details)Apr 13 2023, 3:50 PM
gerritbot added a comment.Apr 13 2023, 4:48 PM

Change 908585 merged by BCornwall:

[operations/puppet@production] hiera: lvs2008: update iface names for bullseye

https://gerrit.wikimedia.org/r/908585

ops-monitoring-bot added a comment.Apr 13 2023, 4:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host lvs2008.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 5:30 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host lvs2008.codfw.wmnet with OS bullseye completed:

  • lvs2008 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304131649_brett_1334141_lvs2008.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Apr 13 2023, 5:30 PM
gerritbot added a comment.Apr 13 2023, 5:47 PM

Change 908605 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hierdata: Remove bgp-med for lvs2008

https://gerrit.wikimedia.org/r/908605

gerritbot added a project: Patch-For-Review.Apr 13 2023, 5:47 PM
gerritbot added a comment.Apr 13 2023, 5:51 PM

Change 908605 merged by BCornwall:

[operations/puppet@production] hierdata: Remove bgp-med for lvs2008

https://gerrit.wikimedia.org/r/908605

Stashbot added a comment.Apr 13 2023, 5:57 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-13T17:57:30Z] <brett> Disable Puppet/PyBal on lvs2009 in preparation for reimaging - T321309

gerritbot added a comment.Apr 13 2023, 6:00 PM

Change 908609 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: lvs2009: update iface names for bullseye

https://gerrit.wikimedia.org/r/908609

BCornwall updated the task description. (Show Details)Apr 13 2023, 6:06 PM
gerritbot added a comment.Apr 13 2023, 6:15 PM

Change 908609 merged by BCornwall:

[operations/puppet@production] hiera: lvs2009: update iface names for bullseye

https://gerrit.wikimedia.org/r/908609

ops-monitoring-bot added a comment.Apr 13 2023, 6:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host lvs2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 6:55 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host lvs2009.codfw.wmnet with OS bullseye completed:

  • lvs2009 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304131816_brett_1395867_lvs2009.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 13 2023, 7:27 PM

Change 908619 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: remove bgp-med override for lvs2009

https://gerrit.wikimedia.org/r/908619

gerritbot added a comment.Apr 13 2023, 7:27 PM

Change 908619 merged by Ssingh:

[operations/puppet@production] hiera: remove bgp-med override for lvs2009

https://gerrit.wikimedia.org/r/908619

gerritbot added a comment.Apr 13 2023, 7:28 PM

Change 908620 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hierdata: Remove bgp-med for lvs2009

https://gerrit.wikimedia.org/r/908620

gerritbot added a comment.Apr 13 2023, 7:29 PM

Change 908620 abandoned by BCornwall:

[operations/puppet@production] hierdata: Remove bgp-med for lvs2009

Reason:

Duplicate I3187d937ffb2a1af9eea2f81b8167c9e67c62530

https://gerrit.wikimedia.org/r/908620

BCornwall updated the task description. (Show Details)Apr 13 2023, 7:58 PM
gerritbot added a comment.Apr 14 2023, 2:41 PM

Change 908860 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] Remove outdated references to pybal-test200[12]

https://gerrit.wikimedia.org/r/908860

ssingh updated the task description. (Show Details)Apr 14 2023, 3:13 PM
ssingh updated the task description. (Show Details)Apr 14 2023, 3:24 PM
ops-monitoring-bot added a comment.Apr 14 2023, 3:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 4:06 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs1013.eqiad.wmnet with OS bullseye completed:

  • lvs1013 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304141536_sukhe_2281014_lvs1013.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ssingh updated the task description. (Show Details)Apr 14 2023, 4:11 PM
ops-monitoring-bot added a comment.Apr 14 2023, 4:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs1015.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 5:15 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs1015.eqiad.wmnet with OS bullseye completed:

  • lvs1015 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304141647_sukhe_2330891_lvs1015.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ssingh updated the task description. (Show Details)Apr 14 2023, 5:16 PM
ops-monitoring-bot added a comment.Apr 14 2023, 5:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host lvs1014.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 5:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host lvs1016.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 5:53 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host lvs1014.eqiad.wmnet with OS bullseye completed:

  • lvs1014 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304141717_brett_2353586_lvs1014.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 14 2023, 5:57 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host lvs1016.eqiad.wmnet with OS bullseye completed:

  • lvs1016 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304141729_brett_2363309_lvs1016.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 14 2023, 6:03 PM

Change 908909 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: lvs/balancer: unify hiera post bullseye upgrade (codfw)

https://gerrit.wikimedia.org/r/908909

BCornwall updated the task description. (Show Details)Apr 14 2023, 6:26 PM
ssingh updated the task description. (Show Details)Apr 14 2023, 6:51 PM
ops-monitoring-bot added a comment.Apr 17 2023, 2:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs1020.eqiad.wmnet with OS bullseye

gerritbot added a comment.Apr 17 2023, 2:34 PM

Change 909294 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: lvs1020: update iface names for bullseye (eqiad)

https://gerrit.wikimedia.org/r/909294

gerritbot added a comment.Apr 17 2023, 2:49 PM

Change 909294 merged by Ssingh:

[operations/puppet@production] hiera: lvs1020: update iface names for bullseye (eqiad)

https://gerrit.wikimedia.org/r/909294

ops-monitoring-bot added a comment.Apr 17 2023, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs1020.eqiad.wmnet with OS bullseye completed:

  • lvs1020 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304171407_sukhe_1028637_lvs1020.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 17 2023, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs1020.eqiad.wmnet with OS bullseye executed with errors:

  • lvs1020 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304171407_sukhe_1028637_lvs1020.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Apr 17 2023, 6:11 PM

Change 909325 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: lvs1019: update iface names for bullseye (eqiad)

https://gerrit.wikimedia.org/r/909325

ssingh updated the task description. (Show Details)Apr 18 2023, 1:40 PM
gerritbot added a comment.Apr 18 2023, 3:38 PM

Change 908909 merged by Ssingh:

[operations/puppet@production] hiera: lvs/balancer: unify hiera post bullseye upgrade (codfw)

https://gerrit.wikimedia.org/r/908909

gerritbot added a comment.Apr 19 2023, 11:19 AM

Change 909985 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] depool eqiad (emergency patch, do not merge)

https://gerrit.wikimedia.org/r/909985

Stashbot added a comment.Apr 19 2023, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-19T13:41:23Z] <sukhe@deploy2002> Locking from deployment [ALL REPOSITORIES]: LVS reimaging in eqiad, blocking deploys T321309

Stashbot added a comment.Apr 19 2023, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-19T13:41:39Z] <sukhe@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: LVS reimaging in eqiad, blocking deploys T321309 (duration: 00m 16s)

Stashbot added a comment.Apr 19 2023, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-19T13:41:46Z] <sukhe@deploy2002> Locking from deployment [ALL REPOSITORIES]: LVS reimaging in eqiad, blocking deploys T321309

ops-monitoring-bot added a comment.Apr 19 2023, 2:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs1019.eqiad.wmnet with OS bullseye

gerritbot added a comment.Apr 19 2023, 2:12 PM

Change 909325 merged by Ssingh:

[operations/puppet@production] hiera: lvs1019: update iface names for bullseye (eqiad)

https://gerrit.wikimedia.org/r/909325

gerritbot added a comment.Apr 19 2023, 2:17 PM

Change 910004 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: remove lvs1019's bgp-med override

https://gerrit.wikimedia.org/r/910004

ops-monitoring-bot added a comment.Apr 19 2023, 2:38 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs1019.eqiad.wmnet with OS bullseye completed:

  • lvs1019 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304191404_sukhe_2973122_lvs1019.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 19 2023, 2:53 PM

Change 910004 merged by Ssingh:

[operations/puppet@production] hiera: remove lvs1019's bgp-med override

https://gerrit.wikimedia.org/r/910004

gerritbot added a comment.Apr 19 2023, 2:56 PM

Change 910027 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: lvs1018: update iface names for bullseye (eqiad)

https://gerrit.wikimedia.org/r/910027

ssingh updated the task description. (Show Details)Apr 19 2023, 3:01 PM
Stashbot added a comment.Apr 19 2023, 3:20 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-19T15:20:36Z] <sukhe> stop pybal on lvs1018 for reimaging: T321309

ops-monitoring-bot added a comment.Apr 19 2023, 3:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs1018.eqiad.wmnet with OS bullseye

gerritbot added a comment.Apr 19 2023, 3:51 PM

Change 910027 merged by Ssingh:

[operations/puppet@production] hiera: lvs1018: update iface names for bullseye (eqiad)

https://gerrit.wikimedia.org/r/910027

ops-monitoring-bot added a comment.Apr 19 2023, 4:23 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs1018.eqiad.wmnet with OS bullseye completed:

  • lvs1018 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304191548_sukhe_3045130_lvs1018.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 19 2023, 4:35 PM

Change 910047 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: remove lvs1018's bgp-med override

https://gerrit.wikimedia.org/r/910047

gerritbot added a comment.Apr 19 2023, 4:38 PM

Change 910047 merged by Ssingh:

[operations/puppet@production] hiera: remove lvs1018's bgp-med override

https://gerrit.wikimedia.org/r/910047

Stashbot added a comment.Apr 19 2023, 4:39 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-19T16:39:15Z] <sukhe> restart pybal on lvs1018 to remove bgp-med change: T321309

ssingh updated the task description. (Show Details)Apr 19 2023, 4:44 PM
gerritbot added a comment.Apr 19 2023, 4:58 PM

Change 910050 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: lvs1017: update iface names for bullseye (eqiad)

https://gerrit.wikimedia.org/r/910050

ops-monitoring-bot added a comment.Apr 19 2023, 5:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs1017.eqiad.wmnet with OS bullseye

gerritbot added a comment.Apr 19 2023, 5:48 PM

Change 910050 merged by Ssingh:

[operations/puppet@production] hiera: lvs1017: update iface names for bullseye (eqiad)

https://gerrit.wikimedia.org/r/910050

gerritbot added a comment.Apr 19 2023, 5:50 PM

Change 910058 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: remove lvs1017's bgp-med override

https://gerrit.wikimedia.org/r/910058

ops-monitoring-bot added a comment.Apr 19 2023, 6:22 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs1017.eqiad.wmnet with OS bullseye completed:

  • lvs1017 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304191746_sukhe_3128361_lvs1017.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 19 2023, 6:23 PM

Change 910058 merged by Ssingh:

[operations/puppet@production] hiera: remove lvs1017's bgp-med override

https://gerrit.wikimedia.org/r/910058

Stashbot added a comment.Apr 19 2023, 6:25 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-19T18:25:40Z] <sukhe> restart pybal on lvs1017 to pick up bgp-med change: T321309

ssingh updated the task description. (Show Details)Apr 19 2023, 6:26 PM
Stashbot added a comment.Apr 19 2023, 6:28 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-19T18:28:26Z] <sukhe@deploy2002> Unlocked for deployment [ALL REPOSITORIES]: LVS reimaging in eqiad, blocking deploys T321309 (duration: 286m 39s)

gerritbot added a comment.Apr 19 2023, 6:28 PM

Change 909985 abandoned by Ssingh:

[operations/dns@master] depool eqiad (emergency patch, do not merge)

Reason:

no longer required

https://gerrit.wikimedia.org/r/909985

ssingh added subscribers: SLyngshede-WMF, cmooney, Volans.EditedApr 19 2023, 6:41 PM

This is now complete and we have upgraded all 176 Traffic hosts to bullseye. We would like to thank @MoritzMuehlenhoff for helping with the Pybal backport that made the LVS reimaging possible, @SLyngshede-WMF and @Volans for the Ganeti reimaging cookbook, and @cmooney for all his help in configuring Netbox. Thank you all!

ssingh added a comment.EditedApr 19 2023, 6:46 PM

@jbond for the firmware reimaging cookbook that saved us a lot of time by automating the iDRAC and NIC firmwares and having the defer reboot option.

MoritzMuehlenhoff added a comment.Apr 20 2023, 5:57 AM

Great work Traffic team (and you're the first SRE sub team to have completed their migration off Buster)!

BCornwall closed this task as Resolved.Apr 20 2023, 2:56 PM

synm.jpg (503×331 px, 35 KB)

gerritbot added a comment.Apr 20 2023, 5:51 PM

Change 910563 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: lvs/balancer: unify hiera post bullseye upgrade (eqiad)

https://gerrit.wikimedia.org/r/910563

gerritbot added a comment.Apr 20 2023, 5:56 PM

Change 910563 merged by Ssingh:

[operations/puppet@production] hiera: lvs/balancer: unify hiera post bullseye upgrade (eqiad)

https://gerrit.wikimedia.org/r/910563

gerritbot added a comment.Apr 20 2023, 6:27 PM

Change 910566 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] pybal/lvs: remove backward compatibility for buster

https://gerrit.wikimedia.org/r/910566

gerritbot added a comment.May 19 2023, 2:24 PM

Change 910566 merged by Ssingh:

[operations/puppet@production] pybal/lvs: remove backward compatibility for buster

https://gerrit.wikimedia.org/r/910566

gerritbot added a comment.Jun 16 2023, 10:29 AM

Change 930761 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] acme-chief: Fix PASSIVE_FQDN syntax

https://gerrit.wikimedia.org/r/930761

gerritbot added a comment.Jun 16 2023, 10:41 AM

Change 930761 merged by Vgutierrez:

[operations/puppet@production] acme-chief: Fix PASSIVE_FQDN syntax

https://gerrit.wikimedia.org/r/930761

gerritbot added a comment.Jul 25 2023, 8:59 AM

Change 941367 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/debs/varnish4@debian-wmf] Version 6.0.11-1wm2 for Debian Bookworm

https://gerrit.wikimedia.org/r/941367

Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL