Page MenuHomePhabricator
Create Task
Maniphest T351620

Upgrade pc2 to Debian Bookworm and MariaDB 10.6
Closed, ResolvedPublic
Actions

Description

  • pc2012
  • pc1012

Details

SubjectRepoBranchLines +/-
pc1012: Disable notificationsoperations/puppetproduction+2 -1
ProductionServices.php: Promote pc1014 to pc2operations/mediawiki-configmaster+2 -2
pc2012: Disable notificationsoperations/puppetproduction+1 -0
ProductionServices.php: Promote pc2014 to pc2 masteroperations/mediawiki-configmaster+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Mon, Nov 20, 8:02 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Nov 20, 8:02 AM
Marostegui triaged this task as Medium priority.Mon, Nov 20, 8:02 AM
Marostegui added a subtask: T351285: Move pc3 spares to pc2.
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui closed subtask T351285: Move pc3 spares to pc2 as Resolved.Tue, Nov 21, 6:31 AM
gerritbot added a comment.Wed, Nov 22, 6:13 AM

Change 976383 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/mediawiki-config@master] ProductionServices.php: Promote pc2014 to pc2 master

https://gerrit.wikimedia.org/r/976383

gerritbot added a project: Patch-For-Review.Wed, Nov 22, 6:13 AM

Change 976383 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices.php: Promote pc2014 to pc2 master

https://gerrit.wikimedia.org/r/976383

gerritbot added a comment.Wed, Nov 22, 6:15 AM

Change 976384 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc2012: Disable notifications

https://gerrit.wikimedia.org/r/976384

Stashbot added a comment.Wed, Nov 22, 6:15 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-22T06:15:27Z] <marostegui@deploy2002> Started scap: Backport for [[gerrit:976383|ProductionServices.php: Promote pc2014 to pc2 master (T351620)]]

gerritbot added a comment.Wed, Nov 22, 6:15 AM

Change 976384 merged by Marostegui:

[operations/puppet@production] pc2012: Disable notifications

https://gerrit.wikimedia.org/r/976384

ops-monitoring-bot added a comment.Wed, Nov 22, 6:16 AM

Icinga downtime and Alertmanager silence (ID=58af808d-b84d-4626-a401-d36ece63d937) set by marostegui@cumin1001 for 2:00:00 on 4 host(s) and their services with reason: Switch

pc[2012,2014].codfw.wmnet,pc[1012,1014].eqiad.wmnet
Stashbot added a comment.Wed, Nov 22, 6:16 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-22T06:16:50Z] <marostegui@deploy2002> marostegui: Backport for [[gerrit:976383|ProductionServices.php: Promote pc2014 to pc2 master (T351620)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Wed, Nov 22, 6:22 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-22T06:22:56Z] <marostegui@deploy2002> Finished scap: Backport for [[gerrit:976383|ProductionServices.php: Promote pc2014 to pc2 master (T351620)]] (duration: 07m 28s)

ops-monitoring-bot added a comment.Wed, Nov 22, 6:23 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host pc2012.codfw.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Wed, Nov 22, 6:30 AM
ops-monitoring-bot added a comment.Wed, Nov 22, 6:57 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host pc2012.codfw.wmnet with OS bookworm completed:

  • pc2012 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311220644_marostegui_1222028_pc2012.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Wed, Nov 22, 6:58 AM
gerritbot added a comment.Wed, Nov 29, 7:12 AM

Change 978365 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/mediawiki-config@master] ProductionServices.php: Promote pc1014 to pc2

https://gerrit.wikimedia.org/r/978365

gerritbot added a project: Patch-For-Review.Wed, Nov 29, 7:12 AM

Change 978366 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc1012: Disable notifications

https://gerrit.wikimedia.org/r/978366

ops-monitoring-bot added a comment.Wed, Nov 29, 7:14 AM

Icinga downtime and Alertmanager silence (ID=e9c495e7-c71c-488c-84a5-80cbeeab64f2) set by marostegui@cumin1001 for 2:00:00 on 3 host(s) and their services with reason: Switch

pc2012.codfw.wmnet,pc[1012,1014].eqiad.wmnet
gerritbot added a comment.Wed, Nov 29, 7:14 AM

Change 978366 merged by Marostegui:

[operations/puppet@production] pc1012: Disable notifications

https://gerrit.wikimedia.org/r/978366

gerritbot added a comment.Wed, Nov 29, 7:14 AM

Change 978365 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices.php: Promote pc1014 to pc2

https://gerrit.wikimedia.org/r/978365

Stashbot added a comment.Wed, Nov 29, 7:15 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-29T07:15:52Z] <marostegui@deploy2002> Started scap: Backport for [[gerrit:978365|ProductionServices.php: Promote pc1014 to pc2 (T351620)]]

Stashbot added a comment.Wed, Nov 29, 7:17 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-29T07:17:17Z] <marostegui@deploy2002> marostegui: Backport for [[gerrit:978365|ProductionServices.php: Promote pc1014 to pc2 (T351620)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Wed, Nov 29, 7:25 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-29T07:25:18Z] <marostegui@deploy2002> Finished scap: Backport for [[gerrit:978365|ProductionServices.php: Promote pc1014 to pc2 (T351620)]] (duration: 09m 25s)

ops-monitoring-bot added a comment.Wed, Nov 29, 7:26 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host pc1012.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Wed, Nov 29, 7:30 AM
ops-monitoring-bot added a comment.Wed, Nov 29, 8:01 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host pc1012.eqiad.wmnet with OS bookworm completed:

  • pc1012 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290743_marostegui_1468539_pc1012.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Wed, Nov 29, 8:01 AM
Marostegui closed this task as Resolved.Wed, Nov 29, 8:11 AM

This is done

Maintenance_bot moved this task from In progress to Done on the DBA board.Wed, Nov 29, 8:15 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL