Page MenuHomePhabricator
Create Task
Maniphest T332604

Upgrade the druid-analytics cluster to bullseye
Closed, ResolvedPublic
Actions

Description

This ticket tracks the work required to upgrade the five-node druid-analytics cluster to Debian bullseye.

It should be noted that, unlike the druid-public cluster, LVS isn't available for druid-analytics.

This means that we do not have load-balancing available and therefore many ingestion jobs are configured to use an-druid1001.eqiad.wmnet as their service host.

Details

SubjectRepoBranchLines +/-
Replace an-druid1001 by an-druid1002 in druid connection stringsanalytics/refinerymaster+21 -21
Replace an-druid1001 by an-druid1002 in druid connection stringsoperations/puppetproduction+1 -1
Format both LVM volumes of an-druid1005 at next reimageoperations/puppetproduction+2 -1
Fix typo in the an-druid netboot partman caseoperations/puppetproduction+1 -1
Setup partman reuse recipe for an-druid hostsoperations/puppetproduction+11 -1
Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
Replace an-druid1001 by an-druid1002 in druid connection stringsrepos/data-engineering/airflow-dags!543brouberolan-druid-connection-stringmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

BTullis created this task.Mar 20 2023, 2:48 PM
BTullis added a project: Data-Platform-SRE.Jun 5 2023, 1:46 PM
JArguello-WMF removed a project: Shared-Data-Infrastructure.Jun 29 2023, 1:44 PM
JArguello-WMF removed a project: Data-Engineering-Planning.Jun 29 2023, 9:41 PM
BTullis moved this task from Incoming to Misc on the Data-Platform-SRE board.Aug 22 2023, 3:55 PM
BTullis moved this task from Misc to Ready for Work on the Data-Platform-SRE board.Oct 10 2023, 9:05 AM
gerritbot added a comment.Nov 8 2023, 3:50 PM

Change 972851 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Setup partman reuse recipe for an-druid hosts

https://gerrit.wikimedia.org/r/972851

gerritbot added a project: Patch-For-Review.Nov 8 2023, 3:50 PM
gerritbot added a comment.Nov 9 2023, 8:19 AM

Change 972851 merged by Brouberol:

[operations/puppet@production] Setup partman reuse recipe for an-druid hosts

https://gerrit.wikimedia.org/r/972851

Maintenance_bot removed a project: Patch-For-Review.Nov 9 2023, 8:30 AM
ops-monitoring-bot added a comment.Nov 9 2023, 2:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1001 for host an-druid1005.eqiad.wmnet with OS bullseye

gerritbot added a comment.Nov 9 2023, 3:03 PM

Change 973178 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Fix typo in the an-druit netboot partman case

https://gerrit.wikimedia.org/r/973178

gerritbot added a project: Patch-For-Review.Nov 9 2023, 3:03 PM
gerritbot added a comment.Nov 9 2023, 3:11 PM

Change 973178 merged by Brouberol:

[operations/puppet@production] Fix typo in the an-druid netboot partman case

https://gerrit.wikimedia.org/r/973178

Maintenance_bot removed a project: Patch-For-Review.Nov 9 2023, 3:30 PM
gerritbot added a comment.Nov 9 2023, 4:56 PM

Change 973204 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Format both LVM volumes of an-druid1005 at next reimage

https://gerrit.wikimedia.org/r/973204

gerritbot added a project: Patch-For-Review.Nov 9 2023, 4:56 PM
gerritbot added a comment.Nov 9 2023, 5:08 PM

Change 973204 merged by Brouberol:

[operations/puppet@production] Format both LVM volumes of an-druid1005 at next reimage

https://gerrit.wikimedia.org/r/973204

Maintenance_bot removed a project: Patch-For-Review.Nov 9 2023, 5:10 PM
ops-monitoring-bot added a comment.Nov 9 2023, 8:54 PM

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1001 for host an-druid1005.eqiad.wmnet with OS bullseye completed:

  • an-druid1005 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311092032_brouberol_1662988_an-druid1005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
brouberol changed the task status from Open to In Progress.Nov 13 2023, 8:30 AM
brouberol claimed this task.
brouberol moved this task from Ready for Work to In Progress on the Data-Platform-SRE board.
ops-monitoring-bot added a comment.Nov 14 2023, 2:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1001 for host an-druid1004.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 14 2023, 2:42 PM

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1001 for host an-druid1004.eqiad.wmnet with OS bullseye completed:

  • an-druid1004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311141424_brouberol_574730_an-druid1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 15 2023, 2:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1001 for host an-druid1003.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 15 2023, 2:47 PM

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1001 for host an-druid1003.eqiad.wmnet with OS bullseye completed:

  • an-druid1003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311151423_brouberol_1225348_an-druid1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 16 2023, 3:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1001 for host an-druid1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 16 2023, 3:55 PM

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1001 for host an-druid1002.eqiad.wmnet with OS bullseye completed:

  • an-druid1002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311161538_brouberol_1889527_an-druid1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Nov 17 2023, 9:15 AM

Change 975206 had a related patch set uploaded (by Brouberol; author: Brouberol):

[analytics/refinery@master] Replace an-druid1001 by an-druid1001 in druid connection strings

https://gerrit.wikimedia.org/r/975206

gerritbot added a project: Patch-For-Review.Nov 17 2023, 9:15 AM
CodeReviewBot added a comment.Nov 17 2023, 9:16 AM

brouberol updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/543

Replace an-druid1001 by an-druid1001 in druid connection strings

gerritbot added a comment.Nov 17 2023, 9:18 AM

Change 975207 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Replace an-druid1001 by an-druid1001 in druid connection strings

https://gerrit.wikimedia.org/r/975207

CodeReviewBot added a comment.Nov 17 2023, 9:34 AM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/543

Replace an-druid1001 by an-druid1002 in druid connection strings

gerritbot added a comment.Nov 17 2023, 9:39 AM

Change 975207 merged by Brouberol:

[operations/puppet@production] Replace an-druid1001 by an-druid1002 in druid connection strings

https://gerrit.wikimedia.org/r/975207

gerritbot added a comment.Nov 22 2023, 8:27 AM

Change 975206 abandoned by Brouberol:

[analytics/refinery@master] Replace an-druid1001 by an-druid1002 in druid connection strings

Reason:

🎵 Dead codes don't need no patches 🎵

https://gerrit.wikimedia.org/r/975206

ops-monitoring-bot added a comment.Nov 22 2023, 12:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1001 for host an-druid1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 22 2023, 12:53 PM

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1001 for host an-druid1001.eqiad.wmnet with OS bullseye completed:

  • an-druid1001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311221235_brouberol_1388795_an-druid1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
brouberol closed this task as Resolved.Nov 22 2023, 1:06 PM
BTullis mentioned this in T288804: Upgrade the Data Engineering infrastructure to Debian Bullseye.Nov 22 2023, 5:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL