Page MenuHomePhabricator
Create Task
Maniphest T299741

Upgrade es1 to Bullseye
Closed, ResolvedPublic
Actions

Description

Let's upgrade es1 to Bullseye.

  • es2032
  • es2030
  • es2028 (master)
  • es1032
  • es1029
  • es1027 (master)

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -0es1027: Disable notifications
operations/puppetproduction+1 -0es1029: Disable notifications
operations/puppetproduction+1 -0es1032: Disable notifications
operations/puppetproduction+1 -0es2028: Disable notifications
operations/puppetproduction+2 -0es2030,es2032: Disable notifications
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jan 21 2022, 5:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 21 2022, 5:47 AM
Marostegui triaged this task as Medium priority.Jan 21 2022, 5:47 AM
Marostegui moved this task from Triage to In progress on the DBA board.
gerritbot added a comment.Jan 21 2022, 5:47 AM

Change 755850 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2030,es2032: Disable notifications

https://gerrit.wikimedia.org/r/755850

gerritbot added a project: Patch-For-Review.Jan 21 2022, 5:47 AM

Change 755850 merged by Marostegui:

[operations/puppet@production] es2030,es2032: Disable notifications

https://gerrit.wikimedia.org/r/755850

ops-monitoring-bot added a comment.Jan 21 2022, 5:49 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es2030.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2022, 6:10 AM
ops-monitoring-bot added a comment.Jan 21 2022, 6:19 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es2030.codfw.wmnet with OS bullseye completed:

  • es2030 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201210549_marostegui_6123_es2030.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 21 2022, 6:21 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-21T06:21:16Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote es2032 to es1 master T299741', diff saved to https://phabricator.wikimedia.org/P18953 and previous config saved to /var/cache/conftool/dbconfig/20220121-062116-marostegui.json

gerritbot added a comment.Jan 21 2022, 6:22 AM

Change 755853 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2028: Disable notifications

https://gerrit.wikimedia.org/r/755853

gerritbot added a project: Patch-For-Review.Jan 21 2022, 6:22 AM

Change 755853 merged by Marostegui:

[operations/puppet@production] es2028: Disable notifications

https://gerrit.wikimedia.org/r/755853

ops-monitoring-bot added a comment.Jan 21 2022, 6:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es2028.codfw.wmnet with OS bullseye

Marostegui updated the task description. (Show Details)Jan 21 2022, 6:24 AM
ops-monitoring-bot added a comment.Jan 21 2022, 6:54 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es2028.codfw.wmnet with OS bullseye completed:

  • es2028 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201210623_marostegui_12867_es2028.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 21 2022, 6:57 AM
Stashbot added a comment.Jan 21 2022, 6:58 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-21T06:58:54Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1032 for reimage T299741', diff saved to https://phabricator.wikimedia.org/P18957 and previous config saved to /var/cache/conftool/dbconfig/20220121-065854-marostegui.json

gerritbot added a comment.Jan 21 2022, 7:00 AM

Change 755856 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1032: Disable notifications

https://gerrit.wikimedia.org/r/755856

gerritbot added a comment.Jan 21 2022, 7:01 AM

Change 755856 merged by Marostegui:

[operations/puppet@production] es1032: Disable notifications

https://gerrit.wikimedia.org/r/755856

ops-monitoring-bot added a comment.Jan 21 2022, 7:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1032.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2022, 7:10 AM
ops-monitoring-bot added a comment.Jan 21 2022, 7:30 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1032.eqiad.wmnet with OS bullseye completed:

  • es1032 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201210703_marostegui_19319_es1032.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 21 2022, 7:31 AM
Stashbot added a comment.Jan 24 2022, 5:43 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-24T05:43:50Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1029 for reimage T299741', diff saved to https://phabricator.wikimedia.org/P18978 and previous config saved to /var/cache/conftool/dbconfig/20220124-054349-marostegui.json

gerritbot added a comment.Jan 24 2022, 5:44 AM

Change 756192 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1029: Disable notifications

https://gerrit.wikimedia.org/r/756192

gerritbot added a project: Patch-For-Review.Jan 24 2022, 5:44 AM

Change 756192 merged by Marostegui:

[operations/puppet@production] es1029: Disable notifications

https://gerrit.wikimedia.org/r/756192

ops-monitoring-bot added a comment.Jan 24 2022, 5:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1029.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jan 24 2022, 6:10 AM
ops-monitoring-bot added a comment.Jan 24 2022, 6:23 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1029.eqiad.wmnet with OS bullseye completed:

  • es1029 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201240552_marostegui_2989_es1029.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 24 2022, 7:22 AM
Stashbot added a comment.Jan 24 2022, 9:43 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-24T09:43:00Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set es1029 as es1 master T299741', diff saved to https://phabricator.wikimedia.org/P19016 and previous config saved to /var/cache/conftool/dbconfig/20220124-094300-marostegui.json

gerritbot added a comment.Jan 24 2022, 9:43 AM

Change 756532 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1027: Disable notifications

https://gerrit.wikimedia.org/r/756532

gerritbot added a project: Patch-For-Review.Jan 24 2022, 9:44 AM
Stashbot added a comment.Jan 24 2022, 9:45 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-24T09:45:05Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1027 T299741', diff saved to https://phabricator.wikimedia.org/P19017 and previous config saved to /var/cache/conftool/dbconfig/20220124-094504-marostegui.json

gerritbot added a comment.Jan 24 2022, 9:46 AM

Change 756532 merged by Marostegui:

[operations/puppet@production] es1027: Disable notifications

https://gerrit.wikimedia.org/r/756532

ops-monitoring-bot added a comment.Jan 24 2022, 9:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1027.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2022, 10:18 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1027.eqiad.wmnet with OS bullseye completed:

  • es1027 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201240950_marostegui_13423_es1027.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui closed this task as Resolved.Jan 24 2022, 10:27 AM
Marostegui updated the task description. (Show Details)

es1 is now running Bullseye

$ sudo cumin "P{R:profile::mariadb::section = es1}" "lsb_release -a"
6 hosts will be targeted:
es[2028,2030,2032].codfw.wmnet,es[1027,1029,1032].eqiad.wmnet
Ok to proceed on 6 hosts? Enter the number of affected hosts to confirm or "q" to quit 6
===== NODE GROUP =====
(6) es[2028,2030,2032].codfw.wmnet,es[1027,1029,1032].eqiad.wmnet
----- OUTPUT of 'lsb_release -a' -----
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye
No LSB modules are available.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL