Page MenuHomePhabricator
Create Task
Maniphest T284128

Re-image (rename) dbstore1006 into db1125
Closed, ResolvedPublic
Actions

Description

As part of T283125, db1125 was converted into dbstore1006 (https://gerrit.wikimedia.org/r/c/operations/puppet/+/692984) but due to a mistake, it wasn't detected that the disk space wasn't enough, so dbstore1006 needs to be converted back to db1125.

dbstore1006 is not and won't be in use so its data isn't needed so this can be done anytime.

The process to rename a host is: https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Rename_while_reimaging

db1125 needs to go into test-s4 section as replica of db1124.

Details

SubjectRepoBranchLines +/-
install_server: Temporarily set db1125 to destructive install.operations/puppetproduction+1 -1
db1125: Rename back from dbstore1006operations/puppetproduction+11 -14
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• razziT283125 dbstore1004 85% disk space used.
 ResolvedKormatT284128 Re-image (rename) dbstore1006 into db1125

Event Timeline

• Marostegui created this task.Jun 2 2021, 6:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 2 2021, 6:59 AM
• Marostegui triaged this task as Medium priority.Jun 2 2021, 6:59 AM
• Marostegui moved this task from Triage to Ready on the DBA board.
LSobanski subscribed.Jun 2 2021, 8:21 AM
Kormat claimed this task.Jun 2 2021, 8:39 AM
ops-monitoring-bot added a comment.Jun 2 2021, 10:01 AM

cookbooks.sre.hosts.decommission executed by kormat@cumin1001 for hosts: dbstore1006.eqiad.wmnet

  • dbstore1006.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
• Marostegui added a comment.Jun 2 2021, 12:03 PM

Dropped dbstore1006 from tendril and zarcillo

gerritbot added a comment.Jun 2 2021, 12:05 PM

Change 697736 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] db1125: Rename back from dbstore1006

https://gerrit.wikimedia.org/r/697736

gerritbot added a project: Patch-For-Review.Jun 2 2021, 12:05 PM
gerritbot added a comment.Jun 2 2021, 12:14 PM

Change 697736 merged by Kormat:

[operations/puppet@production] db1125: Rename back from dbstore1006

https://gerrit.wikimedia.org/r/697736

ops-monitoring-bot added a comment.Jun 2 2021, 12:18 PM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['db1125.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106021218_kormat_13166.log.

gerritbot added a comment.Jun 2 2021, 12:37 PM

Change 697779 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] install_server: Temporarily set db1125 to destructive install.

https://gerrit.wikimedia.org/r/697779

gerritbot added a comment.Jun 2 2021, 12:39 PM

Change 697779 merged by Kormat:

[operations/puppet@production] install_server: Temporarily set db1125 to destructive install.

https://gerrit.wikimedia.org/r/697779

ops-monitoring-bot added a comment.Jun 2 2021, 12:43 PM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['db1125.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106021242_kormat_2843.log.

ops-monitoring-bot added a comment.Jun 2 2021, 1:00 PM

Completed auto-reimage of hosts:

['db1125.eqiad.wmnet']

Of which those FAILED:

['db1125.eqiad.wmnet']
ops-monitoring-bot added a comment.Jun 2 2021, 1:03 PM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['db1125.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106021303_kormat_19444.log.

Maintenance_bot removed a project: Patch-For-Review.Jun 2 2021, 1:10 PM
ops-monitoring-bot added a comment.Jun 2 2021, 1:24 PM

Completed auto-reimage of hosts:

['db1125.eqiad.wmnet']

and were ALL successful.

Kormat added a comment.Jun 2 2021, 1:36 PM

Current status:

  • db1125 has been renamed, wiped, and reimaged
  • It still needs to be re-added to tendril/zarcillo, and have an s4 snapshot deployed on it.
• Marostegui added a comment.Jun 2 2021, 2:29 PM

@Kormat no need to add s4 data to it, just make it a replica of db1124 :)

Kormat closed this task as Resolved.Jun 3 2021, 12:44 PM

"just" done :)

It's back in tendril+zarcillo, and is a replica of db1124.

• Marostegui added a comment.Jun 3 2021, 12:46 PM

Thank you!
PS: Orchestrator detected it automatically too! <3

• Marostegui added a comment.Jun 4 2021, 12:37 PM

Updated: https://wikitech.wikimedia.org/w/index.php?title=MariaDB&type=revision&diff=1914322&oldid=1912301

Kormat added a comment.Jun 7 2021, 9:47 AM

Re-labelling not necessary, as it wasn't re-labelled away from db1125 in the first place: T283300

Machine state set to 'active' in netbox.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL