Page MenuHomePhabricator
Create Task
Maniphest T423874

Migrate msX to Debian Trixie
Closed, ResolvedPublic
Actions

Description

ms1 was migrated/tested at T418561

It was all fine, so let's migrate ms2 and ms3 too

  • ms2
    • db1151
    • db2144
  • ms3
    • db1153
    • db2143

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
db2143: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db1153: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db2144: Disable notificationsMarosteguioperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Apr 20 2026, 1:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 20 2026, 1:57 PM
Marostegui triaged this task as Medium priority.Apr 20 2026, 1:57 PM
Marostegui added a subtask: T418561: Test Debian Trixie on ms1.
Marostegui moved this task from Triage to In progress on the DBA board.
ops-monitoring-bot added a comment.Apr 20 2026, 2:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1151.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Apr 20 2026, 2:42 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1151.eqiad.wmnet with OS trixie completed:

  • db1151 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604201419_marostegui_3275201_db1151.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Apr 20 2026, 3:09 PM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Apr 21 2026, 6:11 AM

Change #1275749 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2144: Disable notifications

https://gerrit.wikimedia.org/r/1275749

gerritbot added a project: Patch-For-Review.Apr 21 2026, 6:11 AM

Change #1275749 merged by Marostegui:

[operations/puppet@production] db2144: Disable notifications

https://gerrit.wikimedia.org/r/1275749

ops-monitoring-bot added a comment.Apr 21 2026, 6:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2144.codfw.wmnet with OS trixie

Maintenance_bot removed a project: Patch-For-Review.Apr 21 2026, 6:31 AM
ops-monitoring-bot added a comment.Apr 21 2026, 7:02 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2144.codfw.wmnet with OS trixie completed:

  • db2144 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604210638_marostegui_3650275_db2144.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Apr 21 2026, 7:04 AM
gerritbot added a comment.Apr 21 2026, 8:44 AM

Change #1275811 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1153: Disable notifications

https://gerrit.wikimedia.org/r/1275811

gerritbot added a project: Patch-For-Review.Apr 21 2026, 8:44 AM

Change #1275811 merged by Marostegui:

[operations/puppet@production] db1153: Disable notifications

https://gerrit.wikimedia.org/r/1275811

ops-monitoring-bot added a comment.Apr 21 2026, 8:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1153.eqiad.wmnet with OS trixie

Maintenance_bot removed a project: Patch-For-Review.Apr 21 2026, 9:32 AM
ops-monitoring-bot added a comment.Apr 21 2026, 9:34 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1153.eqiad.wmnet with OS trixie completed:

  • db1153 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604210911_marostegui_3698142_db1153.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Apr 21 2026, 9:37 AM
gerritbot added a comment.Apr 21 2026, 10:36 AM

Change #1275842 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2143: Disable notifications

https://gerrit.wikimedia.org/r/1275842

gerritbot added a project: Patch-For-Review.Apr 21 2026, 10:36 AM

Change #1275842 merged by Marostegui:

[operations/puppet@production] db2143: Disable notifications

https://gerrit.wikimedia.org/r/1275842

ops-monitoring-bot added a comment.Apr 21 2026, 10:38 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2143.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Apr 21 2026, 11:23 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2143.codfw.wmnet with OS trixie completed:

  • db2143 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604211100_marostegui_3780090_db2143.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Apr 21 2026, 11:24 AM
Marostegui closed this task as Resolved.Apr 21 2026, 11:27 AM

All done

Maintenance_bot moved this task from In progress to Done on the DBA board.Apr 21 2026, 11:29 AM
Marostegui mentioned this in T422365: Migration to Debian Trixie of production database-related hosts.May 22 2026, 5:34 AM
Maintenance_bot removed a project: Patch-For-Review.May 22 2026, 6:31 AM
CWilliams-WMF mentioned this in T432321: msX codfw needs reimaging.Jul 16 2026, 9:13 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits