Page MenuHomePhabricator
Create Task
Maniphest T427032

Migrate es1 section to Debian Trixie
Closed, ResolvedPublic
Actions

Description

  • es2055
  • es2053
  • es2051
  • es1055
  • es1052
  • es1050

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wmnet: Update es1-master CNAMEoperations/dnsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Fri, May 22, 5:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, May 22, 5:40 AM
Marostegui triaged this task as Medium priority.Fri, May 22, 5:40 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui mentioned this in T422365: Migration to Debian Trixie of production database-related hosts.
ops-monitoring-bot added a comment.Wed, May 27, 5:40 AM

Upgrading es2055.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 27, 5:40 AM

Completed depooling of es2055 by marostegui@cumin1003: Upgrading es2055.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 27, 5:41 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2055.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, May 27, 6:21 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2055.codfw.wmnet with OS trixie completed:

  • es2055 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605270604_marostegui_972444_es2055.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Wed, May 27, 6:22 AM
ops-monitoring-bot added a comment.Wed, May 27, 7:41 AM

Upgrading es2051.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 27, 7:42 AM

Completed depooling of es2051 by marostegui@cumin1003: Upgrading es2051.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 27, 7:59 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2051.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, May 27, 8:35 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2051.codfw.wmnet with OS trixie completed:

  • es2051 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605270818_marostegui_992610_es2051.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Wed, May 27, 8:44 AM
ops-monitoring-bot added a comment.Wed, May 27, 9:25 AM

Upgrading es1050.eqiad.wmnet

ops-monitoring-bot added a comment.Wed, May 27, 9:26 AM

Completed depooling of es1050 by marostegui@cumin1003: Upgrading es1050.eqiad.wmnet

ops-monitoring-bot added a comment.Wed, May 27, 9:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, May 27, 10:02 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS trixie completed:

  • es1050 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605270946_marostegui_1071117_es1050.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Wed, May 27, 10:05 AM
ops-monitoring-bot added a comment.Mon, Jun 1, 9:38 AM

Upgrading es1055.eqiad.wmnet

ops-monitoring-bot added a comment.Mon, Jun 1, 9:38 AM

Completed depooling of es1055 by marostegui@cumin1003: Upgrading es1055.eqiad.wmnet

ops-monitoring-bot added a comment.Mon, Jun 1, 9:39 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1055.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Mon, Jun 1, 10:15 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1055.eqiad.wmnet with OS trixie completed:

  • es1055 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606010958_marostegui_3573007_es1055.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Mon, Jun 1, 10:16 AM
Stashbot added a comment.Mon, Jun 1, 10:47 AM

Mentioned in SAL (#wikimedia-operations) [2026-06-01T10:47:39Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Promote es2055 to es1 codfw primary T427032', diff saved to https://phabricator.wikimedia.org/P93424 and previous config saved to /var/cache/conftool/dbconfig/20260601-104739-marostegui.json

Stashbot added a comment.Mon, Jun 1, 10:48 AM

Mentioned in SAL (#wikimedia-operations) [2026-06-01T10:48:38Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Promote es1050 to es1 eqiad primary T427032', diff saved to https://phabricator.wikimedia.org/P93425 and previous config saved to /var/cache/conftool/dbconfig/20260601-104837-marostegui.json

gerritbot added a comment.Mon, Jun 1, 10:49 AM

Change #1295872 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Update es1-master CNAME

https://gerrit.wikimedia.org/r/1295872

gerritbot added a project: Patch-For-Review.Mon, Jun 1, 10:49 AM
gerritbot added a comment.Mon, Jun 1, 10:52 AM

Change #1295872 merged by Marostegui:

[operations/dns@master] wmnet: Update es1-master CNAME

https://gerrit.wikimedia.org/r/1295872

Maintenance_bot removed a project: Patch-For-Review.Mon, Jun 1, 11:31 AM
ops-monitoring-bot added a comment.Tue, Jun 2, 5:06 AM

Upgrading es1052.eqiad.wmnet

ops-monitoring-bot added a comment.Tue, Jun 2, 5:06 AM

Completed depooling of es1052 by marostegui@cumin1003: Upgrading es1052.eqiad.wmnet

ops-monitoring-bot added a comment.Tue, Jun 2, 5:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1052.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Tue, Jun 2, 5:45 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1052.eqiad.wmnet with OS trixie completed:

  • es1052 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606020529_marostegui_3861657_es1052.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Tue, Jun 2, 5:51 AM
ops-monitoring-bot added a comment.Tue, Jun 2, 6:41 AM

Upgrading es2053.codfw.wmnet

ops-monitoring-bot added a comment.Tue, Jun 2, 6:42 AM

Completed depooling of es2053 by marostegui@cumin1003: Upgrading es2053.codfw.wmnet

ops-monitoring-bot added a comment.Tue, Jun 2, 6:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2053.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Tue, Jun 2, 7:20 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2053.codfw.wmnet with OS trixie completed:

  • es2053 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606020704_marostegui_3876624_es2053.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui closed this task as Resolved.Tue, Jun 2, 7:31 AM
Marostegui updated the task description. (Show Details)

All done

Maintenance_bot moved this task from In progress to Done on the DBA board.Tue, Jun 2, 8:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits