Page MenuHomePhabricator
Create Task
Maniphest T428050

Migrate es3 section to Debian Trixie
Closed, ResolvedPublic
Actions

Description

  • es2057
  • es2052
  • es2050
  • es1057
  • es1054
  • es1051

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wmnet: Update es3-master CNAMEoperations/dnsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Wed, Jun 3, 12:24 PM
Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui mentioned this in T422365: Migration to Debian Trixie of production database-related hosts.
ops-monitoring-bot added a comment.Thu, Jun 4, 9:32 AM

Upgrading es2057.codfw.wmnet

ops-monitoring-bot added a comment.Thu, Jun 4, 9:32 AM

Completed depooling of es2057 by marostegui@cumin1003: Upgrading es2057.codfw.wmnet

ops-monitoring-bot added a comment.Thu, Jun 4, 9:33 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2057.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Thu, Jun 4, 10:11 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2057.codfw.wmnet with OS trixie completed:

  • es2057 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606040954_marostegui_1078552_es2057.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Thu, Jun 4, 10:12 AM
ops-monitoring-bot added a comment.Thu, Jun 4, 10:59 AM

Upgrading es2050.codfw.wmnet

ops-monitoring-bot added a comment.Thu, Jun 4, 11:00 AM

Completed depooling of es2050 by marostegui@cumin1003: Upgrading es2050.codfw.wmnet

ops-monitoring-bot added a comment.Thu, Jun 4, 11:00 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2050.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Thu, Jun 4, 11:40 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2050.codfw.wmnet with OS trixie completed:

  • es2050 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606041123_marostegui_1091846_es2050.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Thu, Jun 4, 12:04 PM
ops-monitoring-bot added a comment.Thu, Jun 4, 12:59 PM

Upgrading es1057.eqiad.wmnet

ops-monitoring-bot added a comment.Thu, Jun 4, 1:00 PM

Completed depooling of es1057 by marostegui@cumin1003: Upgrading es1057.eqiad.wmnet

ops-monitoring-bot added a comment.Thu, Jun 4, 1:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1057.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Thu, Jun 4, 1:41 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1057.eqiad.wmnet with OS trixie completed:

  • es1057 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606041325_marostegui_1118960_es1057.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Thu, Jun 4, 1:43 PM
Stashbot added a comment.Thu, Jun 4, 1:56 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-04T13:56:32Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Promote es2050 to es3 codfw primary T428050', diff saved to https://phabricator.wikimedia.org/P93878 and previous config saved to /var/cache/conftool/dbconfig/20260604-135631-marostegui.json

ops-monitoring-bot added a comment.Fri, Jun 5, 5:21 AM

Upgrading es1054.eqiad.wmnet

ops-monitoring-bot added a comment.Fri, Jun 5, 5:21 AM

Completed depooling of es1054 by marostegui@cumin1003: Upgrading es1054.eqiad.wmnet

ops-monitoring-bot added a comment.Fri, Jun 5, 5:22 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1054.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Fri, Jun 5, 6:01 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1054.eqiad.wmnet with OS trixie completed:

  • es1054 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606050545_marostegui_1236596_es1054.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Fri, Jun 5, 7:39 AM
Marostegui updated the task description. (Show Details)Fri, Jun 5, 7:49 AM
ops-monitoring-bot added a comment.Mon, Jun 8, 5:18 AM

Upgrading es2052.codfw.wmnet

ops-monitoring-bot added a comment.Mon, Jun 8, 5:18 AM

Completed depooling of es2052 by marostegui@cumin1003: Upgrading es2052.codfw.wmnet

ops-monitoring-bot added a comment.Mon, Jun 8, 5:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2052.codfw.wmnet with OS trixie

Stashbot added a comment.Mon, Jun 8, 5:31 AM

Mentioned in SAL (#wikimedia-operations) [2026-06-08T05:31:56Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Promote es1054 to es3 eqiad primary T428050', diff saved to https://phabricator.wikimedia.org/P93895 and previous config saved to /var/cache/conftool/dbconfig/20260608-053156-marostegui.json

gerritbot added a comment.Mon, Jun 8, 5:32 AM

Change #1298421 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Update es3-master CNAME

https://gerrit.wikimedia.org/r/1298421

gerritbot added a project: Patch-For-Review.Mon, Jun 8, 5:32 AM
Marostegui updated the task description. (Show Details)Mon, Jun 8, 5:33 AM
ops-monitoring-bot added a comment.Mon, Jun 8, 5:33 AM

Upgrading es1051.eqiad.wmnet

gerritbot added a comment.Mon, Jun 8, 5:33 AM

Change #1298421 merged by Marostegui:

[operations/dns@master] wmnet: Update es3-master CNAME

https://gerrit.wikimedia.org/r/1298421

ops-monitoring-bot added a comment.Mon, Jun 8, 5:54 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2052.codfw.wmnet with OS trixie completed:

  • es2052 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606080539_marostegui_1786432_es2052.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Jun 8, 5:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1051.eqiad.wmnet with OS trixie

Maintenance_bot removed a project: Patch-For-Review.Mon, Jun 8, 6:30 AM
ops-monitoring-bot added a comment.Mon, Jun 8, 6:36 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1051.eqiad.wmnet with OS trixie completed:

  • es1051 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606080620_marostegui_1789103_es1051.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
Marostegui closed this task as Resolved.Mon, Jun 8, 7:18 AM
Marostegui updated the task description. (Show Details)

All done

Maintenance_bot moved this task from In progress to Done on the DBA board.Mon, Jun 8, 7:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits