Page MenuHomePhabricator
Create Task
Maniphest T364552

Fully format and reclone db1246
Closed, ResolvedPublic
Actions

Description

db1246 crashed and got its filesystem corrupted it needs the following.

  • Reimage + full format of / and /srv
  • Reclone
  • Repool

There might be still some HW related work as part of T363119, but can be done at a later time.

Details

SubjectRepoBranchLines +/-
db1246: Enable notifications.operations/puppetproduction+0 -1
Revert "db1246: Disable notifications"operations/puppetproduction+0 -1
Revert "installserver: Allowing formatting db1246"operations/puppetproduction+1 -1
db1246: Switch to puppet 7operations/puppetproduction+3 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJclark-ctrT363119 db1246 crashed
 Resolved• KormatT364552 Fully format and reclone db1246

Event Timeline

Marostegui created this task.May 9 2024, 2:41 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 9 2024, 2:41 PM
Marostegui assigned this task to • Kormat.May 9 2024, 2:42 PM
Marostegui moved this task from Triage to In progress on the DBA board.
MoritzMuehlenhoff subscribed.May 15 2024, 2:58 PM
gerritbot added a comment.May 21 2024, 12:53 PM

Change #1034489 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] db1226: Switch to puppet 7

https://gerrit.wikimedia.org/r/1034489

gerritbot added a project: Patch-For-Review.May 21 2024, 12:53 PM
gerritbot added a comment.May 21 2024, 12:59 PM

Change #1034489 merged by Kormat:

[operations/puppet@production] db1246: Switch to puppet 7

https://gerrit.wikimedia.org/r/1034489

ops-monitoring-bot added a comment.May 21 2024, 1:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by kormat@cumin1002 for host db1246.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.May 21 2024, 1:30 PM
ops-monitoring-bot added a comment.May 21 2024, 1:39 PM

Cookbook cookbooks.sre.hosts.reimage started by kormat@cumin1002 for host db1246.eqiad.wmnet with OS bookworm completed:

  • db1246 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405211318_kormat_1510993_db1246.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.May 21 2024, 1:40 PM

Change #1034178 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] Revert "installserver: Allowing formatting db1246"

https://gerrit.wikimedia.org/r/1034178

gerritbot added a project: Patch-For-Review.May 21 2024, 1:40 PM
gerritbot added a comment.May 21 2024, 2:43 PM

Change #1034178 merged by Kormat:

[operations/puppet@production] Revert "installserver: Allowing formatting db1246"

https://gerrit.wikimedia.org/r/1034178

Stashbot added a comment.May 21 2024, 2:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-21T14:59:25Z] <kormat@cumin1002> dbctl commit (dc=all): 'Depooling db1182 as cloning source T364552', diff saved to https://phabricator.wikimedia.org/P62785 and previous config saved to /var/cache/conftool/dbconfig/20240521-145924-kormat.json

Maintenance_bot removed a project: Patch-For-Review.May 21 2024, 3:31 PM
• Kormat added a comment.May 22 2024, 10:17 AM

Cloning from db1182 completed successfully.

Stashbot added a comment.May 22 2024, 10:24 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T10:24:18Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1182 (re)pooling @ 15%: repool clone source T364552', diff saved to https://phabricator.wikimedia.org/P62858 and previous config saved to /var/cache/conftool/dbconfig/20240522-102418-kormat.json

gerritbot added a comment.May 22 2024, 10:27 AM

Change #1034871 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] Revert "db1246: Disable notifications"

https://gerrit.wikimedia.org/r/1034871

gerritbot added a project: Patch-For-Review.May 22 2024, 10:27 AM

Change #1034871 abandoned by Kormat:

[operations/puppet@production] Revert "db1246: Disable notifications"

Reason:

Simpler to just make the change directly instead of using revert

https://gerrit.wikimedia.org/r/1034871

gerritbot added a comment.May 22 2024, 10:30 AM

Change #1034859 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] db1246: Enable notifications.

https://gerrit.wikimedia.org/r/1034859

• Kormat changed the task status from Open to In Progress.May 22 2024, 10:35 AM
• Kormat triaged this task as Medium priority.
Stashbot added a comment.May 22 2024, 10:39 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T10:39:27Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1182 (re)pooling @ 30%: repool clone source T364552', diff saved to https://phabricator.wikimedia.org/P62860 and previous config saved to /var/cache/conftool/dbconfig/20240522-103924-kormat.json

Stashbot added a comment.May 22 2024, 10:54 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T10:54:33Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1182 (re)pooling @ 45%: repool clone source T364552', diff saved to https://phabricator.wikimedia.org/P62862 and previous config saved to /var/cache/conftool/dbconfig/20240522-105432-kormat.json

gerritbot added a comment.May 22 2024, 10:57 AM

Change #1034859 merged by Kormat:

[operations/puppet@production] db1246: Enable notifications.

https://gerrit.wikimedia.org/r/1034859

Stashbot added a comment.May 22 2024, 11:09 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T11:09:39Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1182 (re)pooling @ 60%: repool clone source T364552', diff saved to https://phabricator.wikimedia.org/P62864 and previous config saved to /var/cache/conftool/dbconfig/20240522-110938-kormat.json

Stashbot added a comment.May 22 2024, 11:24 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T11:24:46Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1182 (re)pooling @ 75%: repool clone source T364552', diff saved to https://phabricator.wikimedia.org/P62866 and previous config saved to /var/cache/conftool/dbconfig/20240522-112444-kormat.json

Maintenance_bot removed a project: Patch-For-Review.May 22 2024, 11:31 AM
Stashbot added a comment.May 22 2024, 11:39 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T11:39:52Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1182 (re)pooling @ 90%: repool clone source T364552', diff saved to https://phabricator.wikimedia.org/P62869 and previous config saved to /var/cache/conftool/dbconfig/20240522-113952-kormat.json

Stashbot added a comment.May 22 2024, 11:54 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T11:54:58Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1182 (re)pooling @ 100%: repool clone source T364552', diff saved to https://phabricator.wikimedia.org/P62872 and previous config saved to /var/cache/conftool/dbconfig/20240522-115458-kormat.json

Stashbot added a comment.May 22 2024, 11:56 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T11:56:34Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 15%: Repool db1246 T364552', diff saved to https://phabricator.wikimedia.org/P62873 and previous config saved to /var/cache/conftool/dbconfig/20240522-115633-kormat.json

Stashbot added a comment.May 22 2024, 12:11 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T12:11:41Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 30%: Repool db1246 T364552', diff saved to https://phabricator.wikimedia.org/P62877 and previous config saved to /var/cache/conftool/dbconfig/20240522-121139-kormat.json

Stashbot added a comment.May 22 2024, 12:26 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T12:26:48Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 45%: Repool db1246 T364552', diff saved to https://phabricator.wikimedia.org/P62881 and previous config saved to /var/cache/conftool/dbconfig/20240522-122647-kormat.json

Stashbot added a comment.May 22 2024, 12:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T12:57:01Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 75%: Repool db1246 T364552', diff saved to https://phabricator.wikimedia.org/P62889 and previous config saved to /var/cache/conftool/dbconfig/20240522-125659-kormat.json

Stashbot added a comment.May 22 2024, 1:12 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T13:12:07Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 90%: Repool db1246 T364552', diff saved to https://phabricator.wikimedia.org/P62893 and previous config saved to /var/cache/conftool/dbconfig/20240522-131206-kormat.json

Stashbot added a comment.May 22 2024, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T13:27:13Z] <kormat@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 100%: Repool db1246 T364552', diff saved to https://phabricator.wikimedia.org/P62898 and previous config saved to /var/cache/conftool/dbconfig/20240522-132712-kormat.json

• Kormat closed this task as Resolved.May 22 2024, 1:39 PM

db1246 fully repooled.

Maintenance_bot moved this task from In progress to Done on the DBA board.May 22 2024, 2:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL