Page MenuHomePhabricator
Create Task
Maniphest T299479

Upgrade s6 to Bullseye
Closed, ResolvedPublic
Actions

Description

Let's go ahead and upgrade s6 to Debian Bullseye + MariaDB 10.4

  • dbstore1005 to be handled by Analytics T299481
  • db2141 (backup host) T299876
  • db2129 (master)
  • db2124
  • db2117
  • db2114
  • db2095
  • db2089
  • db2087
  • db2076
  • db1180
  • db1173 (master)
  • db1168
  • db1165 (sanitarium master)
  • db1155
  • db1140 (backup host) T299876
  • db1131 (candidate master)
  • db1113
  • db1098
  • db1096
  • clouddb1021 to be handled by WMCS T299480
  • clouddb1019 to be handled by WMCS T299480
  • clouddb1015 to be handled by WMCS T299480

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Marostegui updated the task description. (Show Details)Jan 19 2022, 7:50 AM
ops-monitoring-bot added a comment.Jan 19 2022, 7:50 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2124.codfw.wmnet with OS bullseye completed:

  • db2124 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201190715_marostegui_14202_db2124.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 19 2022, 7:51 AM

Change 755309 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2114,db2076: Disable notifications

https://gerrit.wikimedia.org/r/755309

gerritbot added a project: Patch-For-Review.Jan 19 2022, 7:51 AM

Change 755309 merged by Marostegui:

[operations/puppet@production] db2114,db2076: Disable notifications

https://gerrit.wikimedia.org/r/755309

ops-monitoring-bot added a comment.Jan 19 2022, 7:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2076.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 19 2022, 7:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2114.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2022, 8:10 AM
• Marostegui updated the task description. (Show Details)Jan 19 2022, 8:22 AM
gerritbot added a comment.Jan 19 2022, 8:23 AM

Change 755311 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2129: Disable notifications

https://gerrit.wikimedia.org/r/755311

gerritbot added a project: Patch-For-Review.Jan 19 2022, 8:23 AM
ops-monitoring-bot added a comment.Jan 19 2022, 8:23 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2114.codfw.wmnet with OS bullseye completed:

  • db2114 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201190754_marostegui_24683_db2114.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 19 2022, 8:25 AM

Change 755311 merged by Marostegui:

[operations/puppet@production] db2129: Disable notifications

https://gerrit.wikimedia.org/r/755311

ops-monitoring-bot added a comment.Jan 19 2022, 8:26 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2129.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 19 2022, 8:27 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2076.codfw.wmnet with OS bullseye completed:

  • db2076 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201190753_marostegui_24525_db2076.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• Marostegui added a subscriber: jcrespo.Jan 19 2022, 8:56 AM

@jcrespo I have done all codfw s6 hosts except the backup host. I will leave that up to you in case you want to test specific things. If you are ok with the OS change and want me to reimage it myself, let me know, happy to offload that from you.
If you are ok with doing it yourself, feel free to do it anytime that works for you, there is no rush at all as the mariadb version doesn't change.
Further, if you prefer to handle this on a task for all backup sources, let me know and I can create such task for you too.

• Marostegui updated the task description. (Show Details)Jan 19 2022, 8:56 AM
Stashbot added a comment.Jan 19 2022, 8:59 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-19T08:59:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1098 (s6,s7) for Bullseye reimage T299479', diff saved to https://phabricator.wikimedia.org/P18809 and previous config saved to /var/cache/conftool/dbconfig/20220119-085927-marostegui.json

gerritbot added a comment.Jan 19 2022, 9:00 AM

Change 755315 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1098: Disable notifications

https://gerrit.wikimedia.org/r/755315

ops-monitoring-bot added a comment.Jan 19 2022, 9:01 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2129.codfw.wmnet with OS bullseye completed:

  • db2129 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201190826_marostegui_1112_db2129.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 19 2022, 9:01 AM

Change 755315 merged by Marostegui:

[operations/puppet@production] db1098: Disable notifications

https://gerrit.wikimedia.org/r/755315

ops-monitoring-bot added a comment.Jan 19 2022, 9:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1098.eqiad.wmnet with OS bullseye

jcrespo added a comment.Jan 19 2022, 9:07 AM

I need to talk to LSobanski, I mentioned the need for thinking about when upgrading the backup infrastructure, but I don't remember if we ended up scheduling it for this quarter or for a future one. It is certainly in the TODO, but because there should be no hard dependency this time against databases, I can take care this, sooner or later, as it may require some non-trivial upgrades and dependency upgrades/building new packages for backup software that may not be yet ready or need some planing (e.g. bacula upgrade, while technically not a dependency, it will be a relatively large project), plus it needs weighting against the several backup's priorities (if there are more urgent things, we may delay it a bit).

I will give you a more concrete answer/estimation next week.

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2022, 9:10 AM
LSobanski subscribed.Jan 19 2022, 9:10 AM
• Marostegui added a comment.Jan 19 2022, 9:10 AM

Sounds good to me. Again, no rush on this. I will leave it as it is now until you've decided how/when to proceed with them.

• Marostegui updated the task description. (Show Details)Jan 19 2022, 9:24 AM
ops-monitoring-bot added a comment.Jan 19 2022, 9:28 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1098.eqiad.wmnet with OS bullseye completed:

  • db1098 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201190903_marostegui_8331_db1098.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 19 2022, 12:16 PM

Change 755348 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1155: Disable notifications

https://gerrit.wikimedia.org/r/755348

gerritbot added a project: Patch-For-Review.Jan 19 2022, 12:16 PM

Change 755348 merged by Marostegui:

[operations/puppet@production] db1155: Disable notifications

https://gerrit.wikimedia.org/r/755348

ops-monitoring-bot added a comment.Jan 19 2022, 12:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1155.eqiad.wmnet with OS bullseye

• Marostegui updated the task description. (Show Details)Jan 19 2022, 12:47 PM
ops-monitoring-bot added a comment.Jan 19 2022, 12:49 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1155.eqiad.wmnet with OS bullseye completed:

  • db1155 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201191219_marostegui_12839_db1155.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Jan 19 2022, 1:10 PM
Stashbot added a comment.Jan 20 2022, 6:15 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-20T06:15:29Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1180 T299479', diff saved to https://phabricator.wikimedia.org/P18897 and previous config saved to /var/cache/conftool/dbconfig/20220120-061529-marostegui.json

gerritbot added a comment.Jan 20 2022, 6:16 AM

Change 755525 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1180: Disable notifications

https://gerrit.wikimedia.org/r/755525

gerritbot added a project: Patch-For-Review.Jan 20 2022, 6:16 AM
ops-monitoring-bot added a comment.Jan 20 2022, 6:17 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1180.eqiad.wmnet with OS bullseye

gerritbot added a comment.Jan 20 2022, 6:17 AM

Change 755525 merged by Marostegui:

[operations/puppet@production] db1180: Disable notifications

https://gerrit.wikimedia.org/r/755525

ops-monitoring-bot added a comment.Jan 20 2022, 6:47 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1180.eqiad.wmnet with OS bullseye completed:

  • db1180 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201200617_marostegui_31860_db1180.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• Marostegui updated the task description. (Show Details)Jan 20 2022, 6:50 AM
• Marostegui updated the task description. (Show Details)Jan 24 2022, 7:48 AM
Stashbot added a comment.Jan 27 2022, 7:08 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-27T07:08:21Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1131 T299479', diff saved to https://phabricator.wikimedia.org/P19381 and previous config saved to /var/cache/conftool/dbconfig/20220127-070821-marostegui.json

gerritbot added a comment.Jan 27 2022, 7:08 AM

Change 757558 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1131: Disable notifications

https://gerrit.wikimedia.org/r/757558

gerritbot added a comment.Jan 27 2022, 7:10 AM

Change 757558 merged by Marostegui:

[operations/puppet@production] db1131: Disable notifications

https://gerrit.wikimedia.org/r/757558

ops-monitoring-bot added a comment.Jan 27 2022, 7:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1131.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 27 2022, 7:17 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1131.eqiad.wmnet with OS bullseye executed with errors:

  • db1131 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jan 27 2022, 7:17 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1131.eqiad.wmnet with OS bullseye

• Marostegui updated the task description. (Show Details)Jan 27 2022, 7:42 AM
ops-monitoring-bot added a comment.Jan 27 2022, 7:46 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1131.eqiad.wmnet with OS bullseye completed:

  • db1131 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201270717_marostegui_25533_db1131.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 27 2022, 10:52 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-27T10:52:24Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1165 T299479', diff saved to https://phabricator.wikimedia.org/P19427 and previous config saved to /var/cache/conftool/dbconfig/20220127-105223-marostegui.json

gerritbot added a comment.Jan 27 2022, 10:54 AM

Change 757629 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1165: Disable notifications

https://gerrit.wikimedia.org/r/757629

ops-monitoring-bot added a comment.Jan 27 2022, 10:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1165.eqiad.wmnet with OS bullseye

gerritbot added a comment.Jan 27 2022, 10:55 AM

Change 757629 merged by Marostegui:

[operations/puppet@production] db1165: Disable notifications

https://gerrit.wikimedia.org/r/757629

• Marostegui updated the task description. (Show Details)Jan 27 2022, 11:17 AM
ops-monitoring-bot added a comment.Jan 27 2022, 11:24 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1165.eqiad.wmnet with OS bullseye completed:

  • db1165 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201271054_marostegui_17934_db1165.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 28 2022, 7:01 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-28T07:01:12Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1096 (s5,s6) T299479', diff saved to https://phabricator.wikimedia.org/P19531 and previous config saved to /var/cache/conftool/dbconfig/20220128-070112-marostegui.json

gerritbot added a comment.Jan 28 2022, 7:02 AM

Change 757799 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1096: Disable notifications

https://gerrit.wikimedia.org/r/757799

ops-monitoring-bot added a comment.Jan 28 2022, 7:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1096.eqiad.wmnet with OS bullseye

gerritbot added a comment.Jan 28 2022, 7:03 AM

Change 757799 merged by Marostegui:

[operations/puppet@production] db1096: Disable notifications

https://gerrit.wikimedia.org/r/757799

• Marostegui updated the task description. (Show Details)Jan 28 2022, 7:25 AM
ops-monitoring-bot added a comment.Jan 28 2022, 7:29 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1096.eqiad.wmnet with OS bullseye completed:

  • db1096 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201280703_marostegui_8191_db1096.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 28 2022, 9:46 AM

Change 757870 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1168: Disable notifications

https://gerrit.wikimedia.org/r/757870

Stashbot added a comment.Jan 28 2022, 9:46 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-28T09:46:37Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1168 T299479', diff saved to https://phabricator.wikimedia.org/P19552 and previous config saved to /var/cache/conftool/dbconfig/20220128-094636-marostegui.json

gerritbot added a comment.Jan 28 2022, 9:47 AM

Change 757870 merged by Marostegui:

[operations/puppet@production] db1168: Disable notifications

https://gerrit.wikimedia.org/r/757870

ops-monitoring-bot added a comment.Jan 28 2022, 9:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1168.eqiad.wmnet with OS bullseye

• Marostegui updated the task description. (Show Details)Jan 28 2022, 10:13 AM
ops-monitoring-bot added a comment.Jan 28 2022, 10:18 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1168.eqiad.wmnet with OS bullseye completed:

  • db1168 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201280950_marostegui_7209_db1168.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 31 2022, 5:59 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-31T05:59:47Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1113 (s5,s6) T299479', diff saved to https://phabricator.wikimedia.org/P19570 and previous config saved to /var/cache/conftool/dbconfig/20220131-055947-marostegui.json

gerritbot added a comment.Jan 31 2022, 6:00 AM

Change 758271 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1113: Disable notifications

https://gerrit.wikimedia.org/r/758271

gerritbot added a comment.Jan 31 2022, 6:02 AM

Change 758271 merged by Marostegui:

[operations/puppet@production] db1113: Disable notifications

https://gerrit.wikimedia.org/r/758271

ops-monitoring-bot added a comment.Jan 31 2022, 6:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1113.eqiad.wmnet with OS bullseye

• Marostegui updated the task description. (Show Details)Jan 31 2022, 6:24 AM
ops-monitoring-bot added a comment.Jan 31 2022, 6:29 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1113.eqiad.wmnet with OS bullseye completed:

  • db1113 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201310604_marostegui_18489_db1113.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• Marostegui added a comment.Jan 31 2022, 6:35 AM

s6 core hosts are done and we are now pending a primary master switch: T300471

• Marostegui moved this task from In progress to Blocked on the DBA board.Jan 31 2022, 8:54 AM
• Marostegui moved this task from Blocked to In progress on the DBA board.Jan 31 2022, 9:36 AM
• Marostegui moved this task from In progress to Blocked on the DBA board.Feb 14 2022, 6:37 AM
• Marostegui closed subtask T300471: Switchover s6 master (db1173 -> db1131) as Resolved.Jun 14 2022, 6:19 AM
ops-monitoring-bot added a comment.Jun 14 2022, 10:22 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1173.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jun 14 2022, 11:10 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1173.eqiad.wmnet with OS bullseye executed with errors:

  • db1173 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
• Marostegui added a comment.Jun 14 2022, 11:12 AM

db1173 isn't booting up, so I have opened: T310595

ops-monitoring-bot added a comment.Jun 15 2022, 4:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1173.eqiad.wmnet with OS bullseye

• Marostegui closed this task as Resolved.Jun 15 2022, 5:21 AM
• Marostegui updated the task description. (Show Details)

This is all done

ops-monitoring-bot added a comment.Jun 15 2022, 5:23 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1173.eqiad.wmnet with OS bullseye completed:

  • db1173 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202206150453_marostegui_1680707_db1173.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• Cmjohnson closed subtask T310595: db1173 won't boot up as Resolved.Jun 29 2022, 10:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL