Page MenuHomePhabricator
Create Task
Maniphest T411498

Reclone db1169 (s1)
Closed, ResolvedPublic
Actions

Description

After fixing T410400 we can now reclone db1169 and get it back to production

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
db1169: Enable notificationsoperations/puppetproduction+0 -1
installserver: Add db1169 to preseedoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Tue, Dec 2, 1:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Dec 2, 1:54 PM
• Marostegui renamed this task from Reclone db1169 to Reclone db1169 (s1).Tue, Dec 2, 1:55 PM
• Marostegui triaged this task as Medium priority.
ops-monitoring-bot added a comment.Tue, Dec 2, 1:58 PM

Started cloning db1251.eqiad.wmnet to db1169.eqiad.wmnet - marostegui@cumin1003

ops-monitoring-bot added a comment.Tue, Dec 2, 1:58 PM

Completed depool of db1251 - Depool db1251.eqiad.wmnet to then clone it to db1169.eqiad.wmnet - marostegui@cumin1003 - marostegui@cumin1003

• Marostegui moved this task from Triage to In progress on the DBA board.Tue, Dec 2, 2:25 PM

Recloning

ops-monitoring-bot added a comment.Tue, Dec 2, 3:50 PM

Start pool of db1251 gradually with 4 steps - Pool db1251.eqiad.wmnet in after cloning - marostegui@cumin1003

ops-monitoring-bot added a comment.Tue, Dec 2, 4:36 PM

Completed pool of db1251 gradually with 4 steps - Pool db1251.eqiad.wmnet in after cloning - marostegui@cumin1003

ops-monitoring-bot added a comment.Tue, Dec 2, 5:44 PM

Finished cloning db1251.eqiad.wmnet to db1169.eqiad.wmnet - marostegui@cumin1003

gerritbot added a comment.Wed, Dec 3, 4:54 AM

Change #1214220 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Add db1169 to preseed

https://gerrit.wikimedia.org/r/1214220

gerritbot added a project: Patch-For-Review.Wed, Dec 3, 4:54 AM
gerritbot added a comment.Wed, Dec 3, 4:57 AM

Change #1214220 merged by Marostegui:

[operations/puppet@production] installserver: Add db1169 to preseed

https://gerrit.wikimedia.org/r/1214220

Maintenance_bot removed a project: Patch-For-Review.Wed, Dec 3, 5:30 AM
ops-monitoring-bot added a comment.Wed, Dec 3, 5:41 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1169.eqiad.wmnet with OS trixie

• Marostegui added a comment.Wed, Dec 3, 6:02 AM

I am reimagining this host with Trixie, which was the original point of the task, which ended up with all the nokia/uefi things

ops-monitoring-bot added a comment.Wed, Dec 3, 6:26 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1169.eqiad.wmnet with OS trixie completed:

  • db1169 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512030605_marostegui_501408_db1169.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Dec 3, 6:29 AM

Completed depool of db1169 - Depooling db1169 - marostegui@cumin1003

gerritbot added a comment.Wed, Dec 3, 6:31 AM

Change #1214243 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1169: Enable notifications

https://gerrit.wikimedia.org/r/1214243

gerritbot added a project: Patch-For-Review.Wed, Dec 3, 6:31 AM

Change #1214243 merged by Marostegui:

[operations/puppet@production] db1169: Enable notifications

https://gerrit.wikimedia.org/r/1214243

ops-monitoring-bot added a comment.Wed, Dec 3, 6:40 AM

Start pool of db1169 gradually with 4 steps - Repooling db1169 - marostegui@cumin1003

ops-monitoring-bot added a comment.Wed, Dec 3, 7:26 AM

Completed pool of db1169 gradually with 4 steps - Repooling db1169 - marostegui@cumin1003

• Marostegui closed this task as Resolved.Wed, Dec 3, 7:26 AM

Done

Maintenance_bot moved this task from In progress to Done on the DBA board.Wed, Dec 3, 7:29 AM
Maintenance_bot removed a project: Patch-For-Review.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits