Page MenuHomePhabricator
Create Task
Maniphest T299911

Upgrade es3 to Bullseye
Closed, ResolvedPublic
Actions

Description

  • es2034
  • es2029
  • es2027
  • es1034
  • es1031
  • es1028

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -0es1028: Disable notifications
operations/puppetproduction+1 -0es1034: Disable notifications
operations/puppetproduction+1 -0es1031: Disable notifications
operations/puppetproduction+1 -0es2027: Disable notifications
operations/puppetproduction+2 -0es2029: Disable notifications
operations/puppetproduction+1 -0es2034: Disable notifications
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jan 24 2022, 12:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2022, 12:47 PM
Marostegui assigned this task to • Ladsgroup.Jan 24 2022, 12:47 PM
Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to In progress on the DBA board.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJan 24 2022, 12:47 PM
gerritbot added a comment.Jan 24 2022, 12:55 PM

Change 756565 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] es2034: Disable notifications

https://gerrit.wikimedia.org/r/756565

gerritbot added a project: Patch-For-Review.Jan 24 2022, 12:55 PM
gerritbot added a comment.Jan 24 2022, 1:00 PM

Change 756586 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] es2029: Disable notifications

https://gerrit.wikimedia.org/r/756586

gerritbot added a comment.Jan 24 2022, 1:00 PM

Change 756565 merged by Ladsgroup:

[operations/puppet@production] es2034: Disable notifications

https://gerrit.wikimedia.org/r/756565

Stashbot added a comment.Jan 24 2022, 1:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-24T13:06:00Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2034.codfw.wmnet with reason: reimage for upgrade - T299911

ops-monitoring-bot added a comment.Jan 24 2022, 1:06 PM

Icinga downtime set by ladsgroup@cumin1001 for 1 day, 0:00:00 1 host(s) and their services with reason: reimage for upgrade - T299911

es2034.codfw.wmnet
Stashbot added a comment.Jan 24 2022, 1:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-24T13:06:03Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2034.codfw.wmnet with reason: reimage for upgrade - T299911

ops-monitoring-bot added a comment.Jan 24 2022, 1:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es2034.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2022, 1:42 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es2034.codfw.wmnet with OS bullseye executed with errors:

  • es2034 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jan 24 2022, 2:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es2034.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2022, 2:34 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es2034.codfw.wmnet with OS bullseye completed:

  • es2034 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201241400_ladsgroup_1151_es2034.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 25 2022, 9:58 AM

Change 756586 merged by Ladsgroup:

[operations/puppet@production] es2029: Disable notifications

https://gerrit.wikimedia.org/r/756586

Stashbot added a comment.Jan 25 2022, 10:00 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T10:00:36Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2029.codfw.wmnet with reason: reimage for upgrade - T299911

ops-monitoring-bot added a comment.Jan 25 2022, 10:00 AM

Icinga downtime set by ladsgroup@cumin1001 for 1 day, 0:00:00 1 host(s) and their services with reason: reimage for upgrade - T299911

es2029.codfw.wmnet
Stashbot added a comment.Jan 25 2022, 10:00 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T10:00:45Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2029.codfw.wmnet with reason: reimage for upgrade - T299911

ops-monitoring-bot added a comment.Jan 25 2022, 10:02 AM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es2029.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jan 25 2022, 10:10 AM
ops-monitoring-bot added a comment.Jan 25 2022, 10:37 AM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es2029.codfw.wmnet with OS bullseye completed:

  • es2029 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201251001_ladsgroup_9377_es2029.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• Ladsgroup updated the task description. (Show Details)Jan 25 2022, 10:42 AM
gerritbot added a comment.Jan 25 2022, 10:48 AM

Change 756960 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] es2027: Disable notifications

https://gerrit.wikimedia.org/r/756960

gerritbot added a project: Patch-For-Review.Jan 25 2022, 10:48 AM

Change 756960 merged by Ladsgroup:

[operations/puppet@production] es2027: Disable notifications

https://gerrit.wikimedia.org/r/756960

Stashbot added a comment.Jan 25 2022, 10:52 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T10:52:55Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2027.codfw.wmnet with reason: reimage for upgrade - T299911

ops-monitoring-bot added a comment.Jan 25 2022, 10:52 AM

Icinga downtime set by ladsgroup@cumin1001 for 1 day, 0:00:00 1 host(s) and their services with reason: reimage for upgrade - T299911

es2027.codfw.wmnet
Stashbot added a comment.Jan 25 2022, 10:52 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T10:52:59Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2027.codfw.wmnet with reason: reimage for upgrade - T299911

ops-monitoring-bot added a comment.Jan 25 2022, 10:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es2027.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jan 25 2022, 11:10 AM
ops-monitoring-bot added a comment.Jan 25 2022, 11:27 AM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es2027.codfw.wmnet with OS bullseye completed:

  • es2027 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201251053_ladsgroup_26115_es2027.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• Ladsgroup updated the task description. (Show Details)Jan 25 2022, 11:46 AM
Stashbot added a comment.Jan 25 2022, 12:33 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T12:33:03Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool es1031 (T299911)', diff saved to https://phabricator.wikimedia.org/P19136 and previous config saved to /var/cache/conftool/dbconfig/20220125-123303-ladsgroup.json

gerritbot added a comment.Jan 25 2022, 12:42 PM

Change 756971 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] es1031: Disable notifications

https://gerrit.wikimedia.org/r/756971

gerritbot added a project: Patch-For-Review.Jan 25 2022, 12:42 PM

Change 756971 merged by Ladsgroup:

[operations/puppet@production] es1031: Disable notifications

https://gerrit.wikimedia.org/r/756971

Stashbot added a comment.Jan 25 2022, 1:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T13:06:33Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es1031.eqiad.wmnet with reason: reimage for upgrade - T299911

ops-monitoring-bot added a comment.Jan 25 2022, 1:06 PM

Icinga downtime set by ladsgroup@cumin1001 for 1 day, 0:00:00 1 host(s) and their services with reason: reimage for upgrade - T299911

es1031.eqiad.wmnet
Stashbot added a comment.Jan 25 2022, 1:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T13:06:37Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1031.eqiad.wmnet with reason: reimage for upgrade - T299911

Maintenance_bot removed a project: Patch-For-Review.Jan 25 2022, 1:10 PM
ops-monitoring-bot added a comment.Jan 25 2022, 1:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es1031.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2022, 1:56 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es1031.eqiad.wmnet with OS bullseye completed:

  • es1031 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201251324_ladsgroup_5946_es1031.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 25 2022, 2:15 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T14:15:39Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1031 (T299911)', diff saved to https://phabricator.wikimedia.org/P19163 and previous config saved to /var/cache/conftool/dbconfig/20220125-141538-ladsgroup.json

gerritbot added a comment.Jan 25 2022, 2:30 PM

Change 757008 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] es1034: Disable notifications

https://gerrit.wikimedia.org/r/757008

gerritbot added a project: Patch-For-Review.Jan 25 2022, 2:30 PM
gerritbot added a comment.Jan 25 2022, 2:33 PM

Change 757008 merged by Ladsgroup:

[operations/puppet@production] es1034: Disable notifications

https://gerrit.wikimedia.org/r/757008

Stashbot added a comment.Jan 25 2022, 3:00 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T15:00:53Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1031 (T299911)', diff saved to https://phabricator.wikimedia.org/P19175 and previous config saved to /var/cache/conftool/dbconfig/20220125-150052-ladsgroup.json

Stashbot added a comment.Jan 25 2022, 3:02 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T15:02:57Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling es1034 (T299911)', diff saved to https://phabricator.wikimedia.org/P19176 and previous config saved to /var/cache/conftool/dbconfig/20220125-150256-ladsgroup.json

Maintenance_bot removed a project: Patch-For-Review.Jan 25 2022, 3:10 PM
ops-monitoring-bot added a comment.Jan 25 2022, 3:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es1034.eqiad.wmnet with OS bullseye

• Ladsgroup updated the task description. (Show Details)Jan 25 2022, 3:46 PM
ops-monitoring-bot added a comment.Jan 25 2022, 3:52 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es1034.eqiad.wmnet with OS bullseye completed:

  • es1034 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201251520_ladsgroup_28989_es1034.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 25 2022, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T15:56:04Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1034 (T299911)', diff saved to https://phabricator.wikimedia.org/P19193 and previous config saved to /var/cache/conftool/dbconfig/20220125-155604-ladsgroup.json

Stashbot added a comment.Jan 25 2022, 4:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T16:41:19Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1034 (T299911)', diff saved to https://phabricator.wikimedia.org/P19204 and previous config saved to /var/cache/conftool/dbconfig/20220125-164118-ladsgroup.json

Stashbot added a comment.Jan 25 2022, 4:43 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T16:43:25Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Make es1031 master of es3 T299911', diff saved to https://phabricator.wikimedia.org/P19206 and previous config saved to /var/cache/conftool/dbconfig/20220125-164324-ladsgroup.json

gerritbot added a comment.Jan 25 2022, 4:47 PM

Change 757032 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] es1028: Disable notifications

https://gerrit.wikimedia.org/r/757032

gerritbot added a project: Patch-For-Review.Jan 25 2022, 4:47 PM

Change 757032 merged by Ladsgroup:

[operations/puppet@production] es1028: Disable notifications

https://gerrit.wikimedia.org/r/757032

Stashbot added a comment.Jan 25 2022, 4:49 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T16:49:00Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling es1028 (T299911)', diff saved to https://phabricator.wikimedia.org/P19208 and previous config saved to /var/cache/conftool/dbconfig/20220125-164900-ladsgroup.json

Maintenance_bot removed a project: Patch-For-Review.Jan 25 2022, 5:10 PM
ops-monitoring-bot added a comment.Jan 25 2022, 5:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es1028.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2022, 6:17 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es1028.eqiad.wmnet with OS bullseye completed:

  • es1028 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201251744_ladsgroup_24763_es1028.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 25 2022, 6:24 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T18:24:36Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1028 (T299911)', diff saved to https://phabricator.wikimedia.org/P19215 and previous config saved to /var/cache/conftool/dbconfig/20220125-182435-ladsgroup.json

• Ladsgroup updated the task description. (Show Details)Jan 25 2022, 6:46 PM
Stashbot added a comment.Jan 25 2022, 7:09 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T19:09:50Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1028 (T299911)', diff saved to https://phabricator.wikimedia.org/P19220 and previous config saved to /var/cache/conftool/dbconfig/20220125-190949-ladsgroup.json

Stashbot added a comment.Jan 25 2022, 7:12 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T19:12:39Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Make es1028 master of es3 T299911', diff saved to https://phabricator.wikimedia.org/P19221 and previous config saved to /var/cache/conftool/dbconfig/20220125-191238-ladsgroup.json

• Ladsgroup closed this task as Resolved.Jan 25 2022, 7:13 PM
• Ladsgroup updated the task description. (Show Details)
• Ladsgroup moved this task from Incoming to Done on the User-Ladsgroup board.
• Ladsgroup moved this task from In progress to Done on the DBA board.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL