Page MenuHomePhabricator
Create Task
Maniphest T351787

Upgrade pc1 to Debian Bookworm and MariaDB 10.6
Closed, ResolvedPublic
Actions

Description

  • pc2011
  • pc1011

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
ProductionServices.php: Promote pc2014 as master of pc1operations/mediawiki-configmaster+2 -2
pc2011: Disable notificationsoperations/puppetproduction+1 -0
ProductionServices.php: Promote pc1014 to pc1operations/mediawiki-configmaster+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Nov 22 2023, 7:46 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 22 2023, 7:46 AM
Marostegui triaged this task as Medium priority.Nov 22 2023, 7:47 AM
Marostegui added a subtask: T351786: Move parsercache spares to pc1.
Marostegui moved this task from Triage to In progress on the DBA board.
gerritbot added a comment.Dec 11 2023, 5:34 AM

Change 981710 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/mediawiki-config@master] ProductionServices.php: Promote pc1014 to pc1

https://gerrit.wikimedia.org/r/981710

gerritbot added a project: Patch-For-Review.Dec 11 2023, 5:34 AM
Stashbot added a comment.Dec 11 2023, 5:34 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-11T05:34:46Z] <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: pc1 master switch T351787

Stashbot added a comment.Dec 11 2023, 5:35 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-11T05:35:13Z] <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: pc1 master switch T351787

gerritbot added a comment.Dec 11 2023, 5:36 AM

Change 981710 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices.php: Promote pc1014 to pc1

https://gerrit.wikimedia.org/r/981710

Stashbot added a comment.Dec 11 2023, 5:37 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-11T05:37:14Z] <marostegui@deploy2002> Started scap: Backport for [[gerrit:981710|ProductionServices.php: Promote pc1014 to pc1 (T351787)]]

Stashbot added a comment.Dec 11 2023, 5:46 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-11T05:46:43Z] <marostegui@deploy2002> marostegui: Backport for [[gerrit:981710|ProductionServices.php: Promote pc1014 to pc1 (T351787)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Dec 11 2023, 5:54 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-11T05:54:09Z] <marostegui@deploy2002> Finished scap: Backport for [[gerrit:981710|ProductionServices.php: Promote pc1014 to pc1 (T351787)]] (duration: 16m 54s)

ops-monitoring-bot added a comment.Dec 11 2023, 5:55 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host pc1011.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Dec 11 2023, 6:10 AM
Marostegui updated the task description. (Show Details)Dec 11 2023, 6:28 AM
ops-monitoring-bot added a comment.Dec 11 2023, 6:29 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host pc1011.eqiad.wmnet with OS bookworm completed:

  • pc1011 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312110610_marostegui_649898_pc1011.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Dec 12 2023, 5:47 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-12T05:47:09Z] <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: pc1 master switch T351787

gerritbot added a comment.Dec 12 2023, 5:47 AM

Change 982206 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/mediawiki-config@master] ProductionServices.php: Promote pc2014 as master of pc1

https://gerrit.wikimedia.org/r/982206

gerritbot added a project: Patch-For-Review.Dec 12 2023, 5:47 AM
Stashbot added a comment.Dec 12 2023, 5:47 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-12T05:47:27Z] <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: pc1 master switch T351787

gerritbot added a comment.Dec 12 2023, 5:49 AM

Change 982207 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc2011: Disable notifications

https://gerrit.wikimedia.org/r/982207

gerritbot added a comment.Dec 12 2023, 5:49 AM

Change 982206 merged by jenkins-bot:

[operations/mediawiki-config@master] ProductionServices.php: Promote pc2014 as master of pc1

https://gerrit.wikimedia.org/r/982206

gerritbot added a comment.Dec 12 2023, 5:49 AM

Change 982207 merged by Marostegui:

[operations/puppet@production] pc2011: Disable notifications

https://gerrit.wikimedia.org/r/982207

Stashbot added a comment.Dec 12 2023, 5:50 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-12T05:50:59Z] <marostegui@deploy2002> Started scap: Backport for [[gerrit:982206|ProductionServices.php: Promote pc2014 as master of pc1 (T351787)]]

Stashbot added a comment.Dec 12 2023, 5:52 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-12T05:52:24Z] <marostegui@deploy2002> marostegui: Backport for [[gerrit:982206|ProductionServices.php: Promote pc2014 as master of pc1 (T351787)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Dec 12 2023, 5:59 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-12T05:59:34Z] <marostegui@deploy2002> Finished scap: Backport for [[gerrit:982206|ProductionServices.php: Promote pc2014 as master of pc1 (T351787)]] (duration: 08m 35s)

ops-monitoring-bot added a comment.Dec 12 2023, 6:00 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host pc2011.codfw.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Dec 12 2023, 6:10 AM
ops-monitoring-bot added a comment.Dec 12 2023, 6:35 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host pc2011.codfw.wmnet with OS bookworm completed:

  • pc2011 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312120621_marostegui_1465917_pc2011.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui closed this task as Resolved.Dec 12 2023, 7:31 AM
Marostegui updated the task description. (Show Details)

This is all done

Maintenance_bot moved this task from In progress to Done on the DBA board.Dec 12 2023, 8:15 AM
Marostegui merged a task: T352243: Upgrade pc1 to Debian Bookworm and MariaDB 10.6.Dec 13 2023, 9:32 AM
Marostegui closed subtask T351786: Move parsercache spares to pc1 as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits