Page MenuHomePhabricator
Create Task
Maniphest T405978

Re-image remaining full graph hosts to post-graph-split roles
Closed, ResolvedPublic
Actions

Description

Per parent task, we have removed the WDQS full graph LVS (load balancer) pools. The hosts formerly belonging to these pools† need to be reimaged so they can serve the split graphs.

Creating this ticket to:

  • reimage the hosts to their new roles as described here
  • transfer the data to the new hosts
  • deploy the appropriate graph on the new hosts
  • Add the new hosts to load balancer rotation
  • Verify operation

† wdqs10[18-20].eqiad.wmnet, wdqs201[6-7].codfw.wmnet Check the progress of individual hosts in this Etherpad.

Details

Other Assignee
bking
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wdqs: add newly-reimaged hosts to nodes hieradataoperations/puppetproduction+5 -0
wdqs: bring internal-scholarly wdqs2017 into svcoperations/puppetproduction+1 -0
wdqs: bring wdqs101[8-9] into svcoperations/puppetproduction+2 -7
wdqs-internal-scholarly: add wdqs2017operations/puppetproduction+1 -2
wdqs: provision wdqs1018 for wdqs-mainoperations/puppetproduction+3 -2
wdqs1020: move back to production roleoperations/puppetproduction+2 -1
wdqs: Add soon-to-be-reimaged hosts to insetup roleoperations/puppetproduction+9 -2
wdqs-scholarly: Add wdqs2016 to load balancer pooloperations/puppetproduction+1 -0
wdqs: add newly-reimaged hosts as scap targetsoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
bking updated the task description. (Show Details)Sep 30 2025, 8:09 PM
gerritbot added a comment.Sep 30 2025, 8:12 PM

Change #1192626 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: add newly-reimaged hosts as scap targets

https://gerrit.wikimedia.org/r/1192626

gerritbot added a project: Patch-For-Review.Sep 30 2025, 8:12 PM
bking updated the task description. (Show Details)Sep 30 2025, 8:13 PM
gerritbot added a comment.Sep 30 2025, 8:22 PM

Change #1192626 merged by Bking:

[operations/puppet@production] wdqs: add newly-reimaged hosts as scap targets

https://gerrit.wikimedia.org/r/1192626

Stashbot added a comment.Sep 30 2025, 8:33 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-30T20:33:44Z] <bking@deploy2002> Started deploy [wdqs/wdqs@fea7794]: T405978

Stashbot added a comment.Sep 30 2025, 8:34 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-30T20:33:58Z] <bking@deploy2002> Finished deploy [wdqs/wdqs@fea7794]: T405978 (duration: 00m 20s)

Stashbot added a comment.Sep 30 2025, 8:35 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-30T20:35:40Z] <bking@deploy2002> Started deploy [wdqs/wdqs@fea7794]: T405978

Stashbot added a comment.Sep 30 2025, 8:35 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-30T20:35:45Z] <bking@deploy2002> Finished deploy [wdqs/wdqs@fea7794]: T405978 (duration: 00m 10s)

bking changed the task status from Open to In Progress.Sep 30 2025, 8:45 PM
bking triaged this task as Medium priority.
bking updated the task description. (Show Details)
gerritbot added a comment.Oct 1 2025, 1:35 PM

Change #1192890 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs-scholarly: Add wdqs2016 to load balancer pool

https://gerrit.wikimedia.org/r/1192890

gerritbot added a comment.Oct 1 2025, 1:49 PM

Change #1192890 merged by Bking:

[operations/puppet@production] wdqs-scholarly: Add wdqs2016 to load balancer pool

https://gerrit.wikimedia.org/r/1192890

bking updated the task description. (Show Details)Oct 1 2025, 1:57 PM
ops-monitoring-bot added a comment.Oct 1 2025, 9:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host wdqs1018.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 1 2025, 9:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs2017.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 1 2025, 11:14 PM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host wdqs1018.eqiad.wmnet with OS bullseye executed with errors:

  • wdqs1018 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs1018.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 1 2025, 11:16 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs2017.codfw.wmnet with OS bullseye executed with errors:

  • wdqs2017 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs2017.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Gehel edited projects, added Data-Platform-SRE (2025.09.26 - 2025.10.17); removed Data-Platform-SRE (2025.09.05 - 2025.09.26).Oct 2 2025, 1:50 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.
gerritbot added a comment.Oct 6 2025, 9:36 PM

Change #1193943 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: Add soon-to-be-reimaged hosts to insetup role

https://gerrit.wikimedia.org/r/1193943

gerritbot added a comment.Oct 6 2025, 9:39 PM

Change #1193943 merged by Bking:

[operations/puppet@production] wdqs: Add soon-to-be-reimaged hosts to insetup role

https://gerrit.wikimedia.org/r/1193943

ops-monitoring-bot added a comment.Oct 6 2025, 9:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs2017.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2025, 9:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host wdqs1018.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2025, 9:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs1020.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2025, 10:48 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs1020.eqiad.wmnet with OS bullseye executed with errors:

  • wdqs1020 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs1020.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 6 2025, 10:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs1020.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2025, 11:03 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs2017.codfw.wmnet with OS bullseye executed with errors:

  • wdqs2017 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs2017.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 7 2025, 5:02 AM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host wdqs1018.eqiad.wmnet with OS bullseye executed with errors:

  • wdqs1018 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs1018.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 7 2025, 1:49 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs1020.eqiad.wmnet with OS bullseye executed with errors:

  • wdqs1020 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs1020.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 7 2025, 3:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs1020.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 7 2025, 4:25 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs1020.eqiad.wmnet with OS bullseye executed with errors:

  • wdqs1020 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs1020.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 7 2025, 4:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs1020.eqiad.wmnet with OS bullseye

bking closed subtask T406617: wdqs1020: Hanging during partitioning step of installation, rack E2 as Invalid.Oct 7 2025, 7:41 PM
ops-monitoring-bot added a comment.Oct 7 2025, 8:04 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs1020.eqiad.wmnet with OS bullseye completed:

  • wdqs1020 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510071947_bking_554703_wdqs1020.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Oct 7 2025, 8:15 PM

Change #1194286 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs1020: move back to production role

https://gerrit.wikimedia.org/r/1194286

gerritbot added a comment.Oct 7 2025, 8:30 PM

Change #1194286 merged by Bking:

[operations/puppet@production] wdqs1020: move back to production role

https://gerrit.wikimedia.org/r/1194286

Stashbot added a comment.Oct 7 2025, 8:48 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T20:48:39Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T405978, transfer main graph to newly-reimaged host) xfer wikidata_main from wdqs1011.eqiad.wmnet -> wdqs1020.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.Oct 7 2025, 9:41 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T21:41:47Z] <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T405978, transfer main graph to newly-reimaged host) xfer wikidata_main from wdqs1011.eqiad.wmnet -> wdqs1020.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

ops-monitoring-bot added a comment.Oct 7 2025, 9:48 PM

Icinga downtime and Alertmanager silence (ID=6af48cfa-98de-4c84-a5e9-d78b15b819ee) set by bking@cumin2002 for 20:00:00 on 1 host(s) and their services with reason: finish getting host ready for production

wdqs1020.eqiad.wmnet
Stashbot added a comment.Oct 7 2025, 9:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T21:49:45Z] <bking@deploy2002> Started deploy [wdqs/wdqs@fea7794]: T405978

Stashbot added a comment.Oct 7 2025, 9:50 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T21:50:29Z] <bking@deploy2002> Finished deploy [wdqs/wdqs@fea7794]: T405978 (duration: 00m 45s)

ops-monitoring-bot added a comment.Oct 7 2025, 10:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host wdqs1018.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 8 2025, 2:12 AM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host wdqs1018.eqiad.wmnet with OS bullseye completed:

  • wdqs1018 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510080154_ryankemper_722343_wdqs1018.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Oct 8 2025, 3:37 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T03:37:01Z] <ryankemper@deploy2002> Started deploy [wdqs/wdqs@fea7794]: deploy to fresh wdqs-internal-main host T405978

RKemper added a comment.Oct 8 2025, 3:40 AM

wdqs1018 has been reimaged and scap-deployed. data-transfer in progress

gerritbot added a comment.Oct 8 2025, 3:47 AM

Change #1194336 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: provision wdqs1018 for wdqs-main

https://gerrit.wikimedia.org/r/1194336

gerritbot added a comment.Oct 8 2025, 3:50 AM

Change #1194336 merged by Ryan Kemper:

[operations/puppet@production] wdqs: provision wdqs1018 for wdqs-main

https://gerrit.wikimedia.org/r/1194336

Stashbot added a comment.Oct 8 2025, 3:53 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T03:53:12Z] <ryankemper@deploy2002> Finished deploy [wdqs/wdqs@fea7794]: deploy to fresh wdqs-internal-main host T405978 (duration: 16m 11s)

Stashbot added a comment.Oct 8 2025, 3:53 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T03:53:15Z] <ryankemper@deploy2002> Started deploy [wdqs/wdqs@fea7794]: deploy to fresh wdqs-internal-main host T405978

RKemper added a comment.Oct 8 2025, 3:53 AM

^ oops that should say wdqs-main host not wdqs-internal-main

Stashbot added a comment.Oct 8 2025, 3:55 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T03:55:17Z] <ryankemper@deploy2002> Finished deploy [wdqs/wdqs@fea7794]: deploy to fresh wdqs-internal-main host T405978 (duration: 02m 01s)

Stashbot added a comment.Oct 8 2025, 4:37 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T04:37:19Z] <ryankemper@deploy2002> Started deploy [wdqs/wdqs@fea7794]: deploy to fresh wdqs-main host T405978

Stashbot added a comment.Oct 8 2025, 4:37 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T04:37:33Z] <ryankemper@deploy2002> Finished deploy [wdqs/wdqs@fea7794]: deploy to fresh wdqs-main host T405978 (duration: 00m 14s)

ops-monitoring-bot added a comment.Oct 8 2025, 4:41 AM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host wdqs1018.eqiad.wmnet with OS bullseye

RKemper added a comment.Oct 8 2025, 4:57 AM

Sigh, had host all ready for the data-transfer and then ran the reimage by mistake. Probably my sign to log off for the night :) This host will need a scap deploy and data transfer when done

ops-monitoring-bot added a comment.Oct 8 2025, 6:57 AM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host wdqs1018.eqiad.wmnet with OS bullseye executed with errors:

  • wdqs1018 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs1018.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
bking changed the status of subtask T406609: wdqs2017: Apparent hardware issue, rack C2 from Open to In Progress.Oct 8 2025, 3:52 PM
ops-monitoring-bot added a comment.Oct 8 2025, 3:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs2017.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 8 2025, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs2017.codfw.wmnet with OS bullseye completed:

  • wdqs2017 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510081613_bking_1286123_wdqs2017.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Oct 8 2025, 4:53 PM

Change #1194696 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs-internal-scholarly: add wdqs2017

https://gerrit.wikimedia.org/r/1194696

gerritbot added a comment.Oct 8 2025, 6:23 PM

Change #1194696 merged by Bking:

[operations/puppet@production] wdqs-internal-scholarly: add wdqs2017

https://gerrit.wikimedia.org/r/1194696

ops-monitoring-bot added a comment.Oct 8 2025, 6:27 PM

Icinga downtime and Alertmanager silence (ID=c04b2c62-8b17-4f9a-b915-612722761c30) set by bking@cumin2002 for 20:00:00 on 1 host(s) and their services with reason: finish getting host ready for production

wdqs2017.codfw.wmnet
bking closed subtask T406609: wdqs2017: Apparent hardware issue, rack C2 as Resolved.Oct 8 2025, 6:27 PM
ops-monitoring-bot added a comment.Oct 8 2025, 9:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host wdqs1018.eqiad.wmnet with OS bullseye

Stashbot added a comment.Oct 8 2025, 9:13 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T21:13:10Z] <ryankemper@deploy2002> Started deploy [wdqs/wdqs@fea7794]: deploy to fresh internal-scholarly host T405978

Stashbot added a comment.Oct 8 2025, 9:13 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T21:13:22Z] <ryankemper@deploy2002> Finished deploy [wdqs/wdqs@fea7794]: deploy to fresh internal-scholarly host T405978 (duration: 00m 12s)

Stashbot added a comment.Oct 8 2025, 9:18 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T21:18:54Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T405978, transfer to freshly reimaged host) xfer scholarly_articles from wdqs2016.codfw.wmnet -> wdqs2017.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.Oct 8 2025, 9:19 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T21:19:21Z] <ryankemper@cumin2002> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T405978, transfer to freshly reimaged host) xfer scholarly_articles from wdqs2016.codfw.wmnet -> wdqs2017.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.Oct 8 2025, 9:19 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T21:19:28Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T405978, transfer to freshly reimaged host) xfer scholarly_articles from wdqs2016.codfw.wmnet -> wdqs2017.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

ops-monitoring-bot added a comment.Oct 8 2025, 9:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host wdqs1019.eqiad.wmnet with OS bullseye

Stashbot added a comment.Oct 8 2025, 10:09 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T22:09:54Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T405978, transfer to freshly reimaged host) xfer scholarly_articles from wdqs2016.codfw.wmnet -> wdqs2017.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

gerritbot added a comment.Oct 8 2025, 11:49 PM

Change #1194785 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: bring wdqs101[8-9] into svc

https://gerrit.wikimedia.org/r/1194785

gerritbot added a comment.Oct 8 2025, 11:50 PM

Change #1194785 merged by Ryan Kemper:

[operations/puppet@production] wdqs: bring wdqs101[8-9] into svc

https://gerrit.wikimedia.org/r/1194785

Stashbot added a comment.Oct 9 2025, 6:27 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T06:27:23Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1017.eqiad.wmnet -> wdqs1018.eqiad.wmnet w/ force delete existing files, repooling both afterwards

Stashbot added a comment.Oct 9 2025, 6:27 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T06:27:27Z] <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1017.eqiad.wmnet -> wdqs1018.eqiad.wmnet w/ force delete existing files, repooling both afterwards

Stashbot added a comment.Oct 9 2025, 6:27 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T06:27:35Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1017.eqiad.wmnet -> wdqs1018.eqiad.wmnet w/ force delete existing files, repooling both afterwards

Stashbot added a comment.Oct 9 2025, 6:28 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T06:28:36Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1020.eqiad.wmnet -> wdqs1019.eqiad.wmnet w/ force delete existing files, repooling both afterwards

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2025, 6:31 AM
ops-monitoring-bot added a comment.Oct 9 2025, 6:50 AM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host wdqs1019.eqiad.wmnet with OS bullseye completed:

  • wdqs1019 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510082358_ryankemper_1455130_wdqs1019.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Oct 9 2025, 7:20 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T07:20:10Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1017.eqiad.wmnet -> wdqs1018.eqiad.wmnet w/ force delete existing files, repooling both afterwards

Stashbot added a comment.Oct 9 2025, 7:20 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T07:20:54Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1020.eqiad.wmnet -> wdqs1019.eqiad.wmnet w/ force delete existing files, repooling both afterwards

ops-monitoring-bot added a comment.Oct 9 2025, 7:42 AM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host wdqs1018.eqiad.wmnet with OS bullseye completed:

  • wdqs1018 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510082354_ryankemper_1448924_wdqs1018.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Oct 9 2025, 9:34 PM

Change #1195063 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: bring internal-scholarly wdqs2017 into svc

https://gerrit.wikimedia.org/r/1195063

gerritbot added a project: Patch-For-Review.Oct 9 2025, 9:34 PM

Change #1195063 merged by Ryan Kemper:

[operations/puppet@production] wdqs: bring internal-scholarly wdqs2017 into svc

https://gerrit.wikimedia.org/r/1195063

Stashbot added a comment.Oct 9 2025, 10:11 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T22:11:22Z] <inflatador> bking@wdqs10(18|19|20) systemctl start load-categories-daily.service T405978

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2025, 10:30 PM
RKemper moved this task from In Progress to Done on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Oct 10 2025, 6:34 AM

This is done.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Oct 10 2025, 7:24 AM
Gehel closed this task as Resolved.Oct 10 2025, 8:33 AM
bking updated the task description. (Show Details)Oct 10 2025, 8:12 PM
bking closed subtask T406587: Repeated reimage failures on WDQS hosts as Invalid.Oct 17 2025, 7:39 PM
gerritbot added a comment.Oct 21 2025, 9:29 PM

Change #1197723 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: add newly-reimaged hosts to nodes hieradata

https://gerrit.wikimedia.org/r/1197723

gerritbot added a project: Patch-For-Review.Oct 21 2025, 9:29 PM
gerritbot added a comment.Oct 21 2025, 9:48 PM

Change #1197723 merged by Bking:

[operations/puppet@production] wdqs: add newly-reimaged hosts to nodes hieradata

https://gerrit.wikimedia.org/r/1197723

Maintenance_bot removed a project: Patch-For-Review.Oct 21 2025, 10:30 PM
bking mentioned this in T409769: Hardware requirements for WDQS backend migration..Nov 17 2025, 3:05 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits