Page MenuHomePhabricator
Create Task
Maniphest T299889

Upgrade es2 to Bullseye
Closed, ResolvedPublic
Actions

Description

Let's upgrade es2 to Bullseye.

  • es2033
  • es2031
  • es2026
  • es1033
  • es1030
  • es1026

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -0es1026: Disable notifications
operations/puppetproduction+1 -0es1030: Disable notifications
operations/puppetproduction+1 -0es1033: Disable notifications
operations/puppetproduction+3 -0es2026,2031,2033: Disable notifications
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jan 24 2022, 10:31 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2022, 10:31 AM
Marostegui triaged this task as Medium priority.Jan 24 2022, 10:31 AM
Marostegui moved this task from Triage to In progress on the DBA board.
gerritbot added a comment.Jan 24 2022, 10:32 AM

Change 756537 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2026,2031,2033: Disable notifications

https://gerrit.wikimedia.org/r/756537

gerritbot added a project: Patch-For-Review.Jan 24 2022, 10:32 AM

Change 756537 merged by Marostegui:

[operations/puppet@production] es2026,2031,2033: Disable notifications

https://gerrit.wikimedia.org/r/756537

ops-monitoring-bot added a comment.Jan 24 2022, 10:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es2031.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2022, 10:35 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es2033.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2022, 10:51 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es2031.codfw.wmnet with OS bullseye executed with errors:

  • es2031 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jan 24 2022, 10:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es2031.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2022, 11:06 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es2033.codfw.wmnet with OS bullseye completed:

  • es2033 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201241035_marostegui_27897_es2033.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Jan 24 2022, 11:11 AM
ops-monitoring-bot added a comment.Jan 24 2022, 11:27 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es2031.codfw.wmnet with OS bullseye completed:

  • es2031 (WARN)
    • Downtimed on Icinga
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201241053_marostegui_10654_es2031.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 24 2022, 11:36 AM
ops-monitoring-bot added a comment.Jan 24 2022, 11:37 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es2026.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2022, 12:08 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es2026.codfw.wmnet with OS bullseye completed:

  • es2026 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201241137_marostegui_23902_es2026.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 24 2022, 12:08 PM
Stashbot added a comment.Jan 24 2022, 12:10 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-24T12:10:34Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1033 T299889', diff saved to https://phabricator.wikimedia.org/P19042 and previous config saved to /var/cache/conftool/dbconfig/20220124-121029-marostegui.json

gerritbot added a comment.Jan 24 2022, 12:11 PM

Change 756550 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1033: Disable notifications

https://gerrit.wikimedia.org/r/756550

gerritbot added a project: Patch-For-Review.Jan 24 2022, 12:11 PM

Change 756550 merged by Marostegui:

[operations/puppet@production] es1033: Disable notifications

https://gerrit.wikimedia.org/r/756550

ops-monitoring-bot added a comment.Jan 24 2022, 12:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1033.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2022, 12:40 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1033.eqiad.wmnet with OS bullseye completed:

  • es1033 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201241213_marostegui_25705_es1033.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 24 2022, 1:06 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 24 2022, 1:11 PM
Stashbot added a comment.Jan 25 2022, 6:02 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T06:02:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1030 T299889', diff saved to https://phabricator.wikimedia.org/P19079 and previous config saved to /var/cache/conftool/dbconfig/20220125-060241-marostegui.json

gerritbot added a comment.Jan 25 2022, 6:03 AM

Change 756871 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1030: Disable notifications

https://gerrit.wikimedia.org/r/756871

gerritbot added a project: Patch-For-Review.Jan 25 2022, 6:03 AM

Change 756871 merged by Marostegui:

[operations/puppet@production] es1030: Disable notifications

https://gerrit.wikimedia.org/r/756871

ops-monitoring-bot added a comment.Jan 25 2022, 6:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1030.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jan 25 2022, 6:10 AM
ops-monitoring-bot added a comment.Jan 25 2022, 6:36 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1030.eqiad.wmnet with OS bullseye completed:

  • es1030 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201250607_marostegui_3851_es1030.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 25 2022, 6:36 AM
Stashbot added a comment.Jan 25 2022, 1:16 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T13:16:22Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote es1030 to es2 master T299889', diff saved to https://phabricator.wikimedia.org/P19146 and previous config saved to /var/cache/conftool/dbconfig/20220125-131622-marostegui.json

Stashbot added a comment.Jan 25 2022, 1:17 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T13:17:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1026 T299889', diff saved to https://phabricator.wikimedia.org/P19147 and previous config saved to /var/cache/conftool/dbconfig/20220125-131727-marostegui.json

gerritbot added a comment.Jan 25 2022, 1:18 PM

Change 756976 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1026: Disable notifications

https://gerrit.wikimedia.org/r/756976

gerritbot added a project: Patch-For-Review.Jan 25 2022, 1:18 PM

Change 756976 merged by Marostegui:

[operations/puppet@production] es1026: Disable notifications

https://gerrit.wikimedia.org/r/756976

ops-monitoring-bot added a comment.Jan 25 2022, 1:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1026.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jan 25 2022, 2:10 PM
ops-monitoring-bot added a comment.Jan 25 2022, 2:13 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1026.eqiad.wmnet with OS bullseye completed:

  • es1026 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201251343_marostegui_23973_es1026.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui closed this task as Resolved.Jan 25 2022, 2:21 PM
Marostegui updated the task description. (Show Details)

es2 is now running Bullseye

===== NODE GROUP =====
(6) es[2026,2031,2033].codfw.wmnet,es[1026,1030,1033].eqiad.wmnet
----- OUTPUT of 'lsb_release -a' -----
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye
No LSB modules are available.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL