Page MenuHomePhabricator
Create Task
Maniphest T339835

Install Debian Bookworm on a DB
Closed, ResolvedPublic
Actions

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
d-i: Fix retrieval of reuse-parts-test.sh for bookwormoperations/puppetproduction+3 -1
mariadb: Move db1119 to s1operations/puppetproduction+2 -9
install_server: Reimage db1124operations/puppetproduction+1 -1
d-i: Fix retrieval of reuse-parts.sh for bookwormoperations/puppetproduction+3 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jun 19 2023, 7:09 AM
Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to Ready on the DBA board.
ops-monitoring-bot added a comment.Jun 19 2023, 7:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1124.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jun 19 2023, 7:39 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1124.eqiad.wmnet with OS bookworm executed with errors:

  • db1124 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 19 2023, 7:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1124.eqiad.wmnet with OS bullseye

Marostegui added a comment.Jun 19 2023, 7:50 AM

There seem to be issues with the partitioning

ops-monitoring-bot added a comment.Jun 19 2023, 8:03 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1124.eqiad.wmnet with OS bullseye executed with errors:

  • db1124 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202306190753_marostegui_2592974_db1124.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 19 2023, 8:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1124.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jun 19 2023, 8:49 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1124.eqiad.wmnet with OS bookworm executed with errors:

  • db1124 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Jun 19 2023, 9:35 AM

Change 931232 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] d-i: Fix retrieval of reuse-parts.sh for bookworm

https://gerrit.wikimedia.org/r/931232

gerritbot added a project: Patch-For-Review.Jun 19 2023, 9:35 AM
gerritbot added a comment.Jun 19 2023, 9:43 AM

Change 931232 merged by Muehlenhoff:

[operations/puppet@production] d-i: Fix retrieval of reuse-parts.sh for bookworm

https://gerrit.wikimedia.org/r/931232

ops-monitoring-bot added a comment.Jun 19 2023, 9:51 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1124.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Jun 19 2023, 10:10 AM
ops-monitoring-bot added a comment.Jun 19 2023, 10:55 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1124.eqiad.wmnet with OS bookworm completed:

  • db1124 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202306191004_marostegui_2619288_db1124.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jun 19 2023, 12:21 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-19T12:21:00Z] <moritzm> uploaded wmfmariadbpy 0.10+deb12u1 T339835

MoritzMuehlenhoff subscribed.Jun 19 2023, 12:22 PM
In T339835#8945852, @Marostegui wrote:

There seem to be issues with the partitioning

For posterity, this got fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/931232

gerritbot added a comment.Jun 20 2023, 6:26 AM

Change 931493 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db1124

https://gerrit.wikimedia.org/r/931493

gerritbot added a project: Patch-For-Review.Jun 20 2023, 6:26 AM

Change 931493 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db1124

https://gerrit.wikimedia.org/r/931493

ops-monitoring-bot added a comment.Jun 20 2023, 6:29 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1119.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Jun 20 2023, 6:30 AM
ops-monitoring-bot added a comment.Jun 20 2023, 6:34 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1119.eqiad.wmnet with OS bookworm executed with errors:

  • db1119 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 20 2023, 7:18 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1119.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jun 20 2023, 8:39 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1119.eqiad.wmnet with OS bookworm completed:

  • db1119 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202306200729_marostegui_2855360_db1119.out, asking the operator what to do
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202306200814_marostegui_2855360_db1119.out, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202306200820_marostegui_2855360_db1119.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui moved this task from Ready to In progress on the DBA board.Jun 22 2023, 6:51 AM
Marostegui added a comment.Aug 30 2023, 7:37 AM

I need to recloned db1119 to s1 so it can serve some traffic.

Marostegui mentioned this in T337206: decommission db1119.eqiad.wmnet.Aug 30 2023, 7:41 AM
gerritbot added a comment.Aug 30 2023, 7:41 AM

Change 953555 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move db1119 to s1

https://gerrit.wikimedia.org/r/953555

gerritbot added a project: Patch-For-Review.Aug 30 2023, 7:41 AM
gerritbot added a comment.Aug 31 2023, 11:35 AM

Change 953555 merged by Marostegui:

[operations/puppet@production] mariadb: Move db1119 to s1

https://gerrit.wikimedia.org/r/953555

Maintenance_bot removed a project: Patch-For-Review.Aug 31 2023, 12:10 PM
Marostegui added a comment.Aug 31 2023, 1:42 PM

db1119 has been recloned and it is now on s1 (not serving traffic)

Marostegui added a comment.Sep 1 2023, 5:54 AM

This has been done, the testing can be followed up on the parent task.

Marostegui closed this task as Resolved.Sep 11 2023, 5:38 AM
Maintenance_bot moved this task from In progress to Done on the DBA board.Sep 11 2023, 6:15 AM
gerritbot added a comment.Oct 17 2023, 12:31 PM

Change 966524 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] d-i: Fix retrieval of reuse-parts-test.sh for bookworm

https://gerrit.wikimedia.org/r/966524

gerritbot added a project: Patch-For-Review.Oct 17 2023, 12:31 PM
gerritbot added a comment.Oct 17 2023, 3:16 PM

Change 966524 merged by Elukey:

[operations/puppet@production] d-i: Fix retrieval of reuse-parts-test.sh for bookworm

https://gerrit.wikimedia.org/r/966524

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2023, 3:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits