Page MenuHomePhabricator
Create Task
Maniphest T407472

Install a testing db with Debian Trixie
Closed, ResolvedPublic
Actions

Description

Probably let's start with test-s4 cluster host to start testing the initial run.
Initially with MariaDB 10.11

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
isntallserver: Do not format /srv on es2028operations/puppetproduction+1 -1
es1033: Enable notificationsoperations/puppetproduction+0 -1
instances.yaml: Add es1033 to dbctloperations/puppetproduction+1 -0
installserver: Format /srv/ in es2028operations/puppetproduction+1 -1
aptrepo: enable wmfmariadbpy for Trixieoperations/puppetproduction+10 -0
packages_wmf,packages_client.pp: Add trixieoperations/puppetproduction+4 -0
mariadb: Define mariadb packages for trixieoperations/puppetproduction+4 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Oct 16 2025, 8:37 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 16 2025, 8:37 AM
Marostegui triaged this task as Medium priority.Oct 16 2025, 8:37 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Oct 16 2025, 8:42 AM

Change #1196628 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Define mariadb packages for trixie

https://gerrit.wikimedia.org/r/1196628

gerritbot added a project: Patch-For-Review.Oct 16 2025, 8:42 AM
gerritbot added a comment.Oct 17 2025, 5:05 AM

Change #1196628 merged by Marostegui:

[operations/puppet@production] mariadb: Define mariadb packages for trixie

https://gerrit.wikimedia.org/r/1196628

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2025, 5:30 AM
gerritbot added a comment.Oct 17 2025, 11:42 AM

Change #1196893 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] packages_wmf,packages_client.pp: Add trixie

https://gerrit.wikimedia.org/r/1196893

gerritbot added a project: Patch-For-Review.Oct 17 2025, 11:42 AM
gerritbot added a comment.Oct 17 2025, 11:46 AM

Change #1196893 merged by Marostegui:

[operations/puppet@production] packages_wmf,packages_client.pp: Add trixie

https://gerrit.wikimedia.org/r/1196893

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2025, 12:31 PM
Stashbot added a comment.Oct 20 2025, 7:28 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-20T07:28:33Z] <marostegui> Stop MariaDB on es2032 to clone sretest2003 T407472

Marostegui added a comment.Oct 20 2025, 7:28 AM
This comment was removed by Marostegui.
Marostegui added a comment.Oct 20 2025, 7:37 AM
In T407472#11288505, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2025-10-20T07:28:33Z] <marostegui> Stop MariaDB on es2032 to clone sretest2003 T407472

Ignore this - wrong task

ops-monitoring-bot added a comment.Oct 21 2025, 9:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db-test1003.eqiad.wmnet with OS trixie

Marostegui added a comment.Oct 21 2025, 9:41 AM

db-test1003 installed with trixie and mariadb 10.11
I will install a vanilla mariadb database just to do a few tests there with puppet, configuration etc.

Marostegui mentioned this in T400056: Core DB testbed on VMs.Oct 21 2025, 9:42 AM
Marostegui mentioned this in T407845: Build python3-wmfmariadbpy and wmfmariadbpy-common for Debian Trixie.Oct 21 2025, 9:53 AM
Marostegui added a subtask: T407845: Build python3-wmfmariadbpy and wmfmariadbpy-common for Debian Trixie.

Some puppet issues tracked at T407845

FCeratto-WMF changed the status of subtask T407845: Build python3-wmfmariadbpy and wmfmariadbpy-common for Debian Trixie from Open to In Progress.Oct 21 2025, 10:03 AM
gerritbot added a comment.Oct 21 2025, 12:22 PM

Change #1197629 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] aptrepo: enable wmfmariadbpy for Trixie

https://gerrit.wikimedia.org/r/1197629

gerritbot added a project: Patch-For-Review.Oct 21 2025, 12:22 PM
gerritbot added a comment.Oct 21 2025, 2:52 PM

Change #1197629 merged by Federico Ceratto:

[operations/puppet@production] aptrepo: enable wmfmariadbpy for Trixie

https://gerrit.wikimedia.org/r/1197629

Maintenance_bot removed a project: Patch-For-Review.Oct 21 2025, 3:31 PM
ops-monitoring-bot added a comment.Oct 22 2025, 12:22 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db-test1003.eqiad.wmnet with OS trixie executed with errors:

  • db-test1003 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510210923_marostegui_1835076_db-test1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console db-test1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Marostegui added a comment.Oct 22 2025, 5:38 AM
In T407472#11296707, @ops-monitoring-bot wrote:

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db-test1003.eqiad.wmnet with OS trixie executed with errors:

  • db-test1003 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510210923_marostegui_1835076_db-test1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console db-test1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

This was due to T407845 but the reimage in general went fine, it is just that puppet issue

FCeratto-WMF closed subtask T407845: Build python3-wmfmariadbpy and wmfmariadbpy-common for Debian Trixie as Resolved.Oct 28 2025, 9:27 AM
Marostegui added a comment.Oct 28 2025, 4:29 PM

Puppet works fine now as T407845 was resolved.

Marostegui added a subtask: T408772: decommission es1033.eqiad.wmnet.Oct 30 2025, 6:15 AM
Marostegui added a parent task: T408407: decommission es2028.Oct 30 2025, 11:39 AM
ops-monitoring-bot added a comment.Oct 30 2025, 11:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie

gerritbot added a comment.Oct 30 2025, 12:41 PM

Change #1200049 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Format /srv/ in es2028

https://gerrit.wikimedia.org/r/1200049

gerritbot added a project: Patch-For-Review.Oct 30 2025, 12:41 PM
gerritbot added a comment.Oct 30 2025, 12:45 PM

Change #1200049 merged by Marostegui:

[operations/puppet@production] installserver: Format /srv/ in es2028

https://gerrit.wikimedia.org/r/1200049

ops-monitoring-bot added a comment.Oct 30 2025, 12:52 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie executed with errors:

  • es2028 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510301203_marostegui_1349427_es2028.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es2028.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 30 2025, 12:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie

Maintenance_bot removed a project: Patch-For-Review.Oct 30 2025, 1:31 PM
MoritzMuehlenhoff closed subtask T408776: megacli issues on Debian Trixie as Resolved.Oct 30 2025, 5:37 PM
ops-monitoring-bot added a comment.Oct 30 2025, 6:05 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie completed:

  • es2028 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510301320_marostegui_1359934_es2028.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui added a comment.Oct 31 2025, 6:12 AM

mariadb started on es1033 and es2028 (running trixie)

Marostegui mentioned this in T409030: Pull a disk out from es1033.Nov 3 2025, 7:57 AM
Marostegui closed subtask T409030: Pull a disk out from es1033 as Resolved.Nov 4 2025, 6:13 AM
gerritbot added a comment.Nov 5 2025, 7:12 AM

Change #1201996 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es1033 to dbctl

https://gerrit.wikimedia.org/r/1201996

gerritbot added a project: Patch-For-Review.Nov 5 2025, 7:12 AM

Change #1201996 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es1033 to dbctl

https://gerrit.wikimedia.org/r/1201996

Stashbot added a comment.Nov 5 2025, 7:16 AM

Mentioned in SAL (#wikimedia-operations) [2025-11-05T07:16:06Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Add es1033 to es2 depooled T409257 T407472', diff saved to https://phabricator.wikimedia.org/P84834 and previous config saved to /var/cache/conftool/dbconfig/20251105-071605-marostegui.json

Stashbot mentioned this in T409257: Move es1033 (es2 Debian Trixie) to es7.Nov 5 2025, 7:16 AM
gerritbot added a comment.Nov 5 2025, 7:17 AM

Change #1202002 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1033: Enable notifications

https://gerrit.wikimedia.org/r/1202002

gerritbot added a comment.Nov 5 2025, 7:18 AM

Change #1202002 merged by Marostegui:

[operations/puppet@production] es1033: Enable notifications

https://gerrit.wikimedia.org/r/1202002

Marostegui added a comment.Nov 5 2025, 7:21 AM

es1033 (es) is starting to serve minimal traffic in es2

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2025, 7:30 AM
Marostegui mentioned this in T406690: Decommission es1026 - es1034.Nov 11 2025, 3:17 PM
Marostegui mentioned this in T409533: Compile and package MariaDB 10.11.15.Nov 11 2025, 4:22 PM
Marostegui closed subtask T409257: Move es1033 (es2 Debian Trixie) to es7 as Resolved.Nov 12 2025, 9:15 AM

es1033 now in es7

Marostegui closed subtask T408777: Swap partition issue when installing a DB with Debian Trixie as Resolved.Nov 17 2025, 2:27 PM
Marostegui added a comment.Dec 4 2025, 6:30 AM

db1169 is in s1

gerritbot added a comment.Dec 16 2025, 7:18 AM

Change #1218652 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] isntallserver: Do not format /srv on es2028

https://gerrit.wikimedia.org/r/1218652

gerritbot added a project: Patch-For-Review.Dec 16 2025, 7:18 AM
gerritbot added a comment.Dec 16 2025, 7:21 AM

Change #1218652 merged by Marostegui:

[operations/puppet@production] isntallserver: Do not format /srv on es2028

https://gerrit.wikimedia.org/r/1218652

Maintenance_bot removed a project: Patch-For-Review.Dec 16 2025, 7:30 AM
ops-monitoring-bot added a comment.Dec 16 2025, 10:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Dec 16 2025, 10:45 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie executed with errors:

  • es2028 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es2028.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Dec 16 2025, 10:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Dec 16 2025, 11:22 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie executed with errors:

  • es2028 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es2028.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Dec 16 2025, 12:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie

cmooney added a subtask: T412807: Dell R740xd reimage fails in debian-installer, configures IP on incorrect interface.Dec 16 2025, 1:03 PM
ops-monitoring-bot added a comment.Dec 16 2025, 1:55 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie executed with errors:

  • es2028 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es2028.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Dec 19 2025, 7:55 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Dec 19 2025, 1:39 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie executed with errors:

  • es2028 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es2028.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Dec 22 2025, 6:49 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Dec 22 2025, 7:32 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2028.codfw.wmnet with OS trixie completed:

  • es2028 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512220709_marostegui_3563158_es2028.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui closed subtask T412807: Dell R740xd reimage fails in debian-installer, configures IP on incorrect interface as Resolved.Dec 22 2025, 7:33 AM
Marostegui closed this task as Resolved.Dec 29 2025, 12:10 PM

db1169 is running Trixie.

Maintenance_bot moved this task from In progress to Done on the DBA board.Dec 29 2025, 12:29 PM
Marostegui created subtask T415165: Install a clouddb hosts with Debian Trixie.Wed, Jan 21, 6:30 AM
Marostegui added a subtask: T415358: Migrate 1P db* to Debian Trixie.Fri, Jan 23, 12:58 PM
Marostegui closed subtask T415358: Migrate 1P db* to Debian Trixie as Resolved.Mon, Feb 2, 10:17 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits