Page MenuHomePhabricator
Create Task
Maniphest T430880

Migrate WDQS hosts to Bookworm or later
Closed, ResolvedPublic
Actions

Description

Per parent ticket, we need to have WDQS on a newer version of Debian by September of 2026. We'll start on WDQS once the WCQS upgrade in T430879 is finished.

Creating this ticket to:
Reimage/redeploy the following hosts (documented for WDQS here )

Hosts

36of 36 hosts complete

wdqs-test

  • wdqs2025

wdqs-internal_scholarly

  • wdqs1027
  • wdqs2017
  • wdqs2026
  • wdqs2027

wdqs-internal_main

  • wdqs1025
  • wdqs1026
  • wdqs2018 pybal issues after VLAN move
  • wdqs2019 pybal issues after VLAN move
  • wdqs2020

wdqs-scholarly

  • wdqs1023
  • wdqs1024 failed multiple NIC firmware updates; eventually got it to reimage with --use-http-for-dhcp flag.
  • wdqs2016
  • wdqs2023
  • wdqs2024

wdqs-main

  • wdqs1011
  • wdqs1012
  • wdqs1013 reimage, deploy, transfer, and postflight complete; required PyBal restart because the pre-reimage depool set pooled=no, which retained the cached address across the VLAN move
  • wdqs1014 reimage, deploy, transfer, and postflight complete; locally healthy but not serving due to the same pooled=no stale-address issue, pending a Traffic-coordinated rolling PyBal restart
  • wdqs1016
  • wdqs1017
  • wdqs1018
  • wdqs1019
  • wdqs1020
  • wdqs1021
  • wdqs1022
  • wdqs2007
  • wdqs2008
  • wdqs2010
  • wdqs2011
  • wdqs2012
  • wdqs2013
  • wdqs2014
  • wdqs2015
  • wdqs2021
  • wdqs2022
  • Make config management changes if necessary
  • Verify operation

Details

Other Assignee
RKemper
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
sre.wdqs.data-transfer: abort on failed transferRyan Kemperoperations/cookbooksmaster+66 -1
Transferer: log the reason a copy failedRyan Kemperoperations/software/transferpymaster+17 -0
sre.wdqs: stop transferpy burying cookbook outputRyan Kemperoperations/cookbooksmaster+21 -2
wdqs: bound missing updater metricsRyan Kemperoperations/cookbooksmaster+102 -23
wdqs: restore Blazegraph exportersRyan Kemperoperations/cookbooksmaster+59 -7
prometheus: start exporter with BlazegraphRyan Kemperoperations/puppetproduction+31 -1
sre.wdqs: reject NaN lag in wait_for_updaterRyan Kemperoperations/cookbooksmaster+73 -0
installserver: Restore WDQS preseedsRyan Kemperoperations/puppetproduction+1 -1
wdqs: make wdqs-all flag for wdqs-main hostsRyan Kemperoperations/cookbooksmaster+15 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Sat, Jul 25, 7:57 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-25T19:57:32Z] <ryankemper@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host

Stashbot added a comment.Sat, Jul 25, 7:57 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-25T19:57:38Z] <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host (duration: 00m 05s)

gerritbot added a comment.Sat, Jul 25, 8:09 PM

Change #1317127 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] sre.wdqs.data-transfer: abort on failed transfer

https://gerrit.wikimedia.org/r/1317127

gerritbot added a comment.Sat, Jul 25, 8:09 PM

Change #1317128 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] sre.wdqs: stop transferpy burying cookbook output

https://gerrit.wikimedia.org/r/1317128

Stashbot added a comment.Sat, Jul 25, 8:18 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-25T20:18:21Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs1024.eqiad.wmnet -> wdqs2024.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Sat, Jul 25, 8:18 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-25T20:18:32Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs1017.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Sat, Jul 25, 9:06 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-25T21:06:50Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs1024.eqiad.wmnet -> wdqs2024.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Sat, Jul 25, 9:26 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-25T21:26:45Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs1017.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Sat, Jul 25, 10:06 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-25T22:06:17Z] <ryankemper> T430880 [WDQS] Repooled wdqs1017 and wdqs2024 after reimaging to bookworm, scap deploying, and data xfering

ops-monitoring-bot added a comment.Sun, Jul 26, 12:08 AM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2003 for host wdqs1018.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sun, Jul 26, 12:08 AM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2003 for host wdqs1019.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sun, Jul 26, 1:05 AM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2003 for host wdqs1018.eqiad.wmnet with OS bookworm completed:

  • wdqs1018 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"wdqs1018.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=wdqs-main,service=wdqs-main"}

  • Disabled Puppet
  • Host successfully migrated to the new VLAN
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607260038_ryankemper_430255_wdqs1018.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'name=wdqs1018\.eqiad\.wmnet,dc=eqiad,cluster=wdqs\-main,service=wdqs\-main' set/pooled=yes

  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sun, Jul 26, 1:09 AM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2003 for host wdqs1019.eqiad.wmnet with OS bookworm completed:

  • wdqs1019 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Set pooled=inactive for the following services on confctl:

{"wdqs1019.eqiad.wmnet": {"weight": 10, "pooled": "yes"}, "tags": "dc=eqiad,cluster=wdqs-main,service=wdqs-main"}

  • Disabled Puppet
  • Host successfully migrated to the new VLAN
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607260043_ryankemper_430341_wdqs1019.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'name=wdqs1019\.eqiad\.wmnet,dc=eqiad,cluster=wdqs\-main,service=wdqs\-main' set/pooled=yes

  • Updated Netbox data from PuppetDB
Stashbot added a comment.Mon, Jul 27, 4:48 AM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T04:48:11Z] <ryankemper@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host

Stashbot added a comment.Mon, Jul 27, 4:48 AM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T04:48:17Z] <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host (duration: 00m 05s)

Stashbot added a comment.Mon, Jul 27, 4:48 AM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T04:48:21Z] <ryankemper@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host

Stashbot added a comment.Mon, Jul 27, 4:48 AM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T04:48:26Z] <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host (duration: 00m 05s)

Stashbot added a comment.Mon, Jul 27, 4:51 AM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T04:51:42Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs1018.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 4:51 AM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T04:51:52Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs1019.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 6:00 AM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T06:00:30Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs1018.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 6:00 AM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T06:00:34Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs1019.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 7:16 AM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T07:16:47Z] <ryankemper> T430880 [WDQS] Reimaged wdqs1018 and wdqs1019 to Bookworm, restored data using test-cookbook change 1317128, and repooled both; 25/36 hosts complete

RKemper updated the task description. (Show Details)Mon, Jul 27, 7:21 AM
gerritbot added a comment.Mon, Jul 27, 1:39 PM

Change #1315664 merged by Bking:

[operations/puppet@production] prometheus: start exporter with Blazegraph

https://gerrit.wikimedia.org/r/1315664

bking updated the task description. (Show Details)Mon, Jul 27, 1:52 PM
gerritbot added a comment.Mon, Jul 27, 2:21 PM

Change #1315665 merged by Bking:

[operations/cookbooks@master] wdqs: restore Blazegraph exporters

https://gerrit.wikimedia.org/r/1315665

gerritbot added a comment.Mon, Jul 27, 2:25 PM

Change #1315666 merged by jenkins-bot:

[operations/cookbooks@master] wdqs: bound missing updater metrics

https://gerrit.wikimedia.org/r/1315666

Stashbot added a comment.Mon, Jul 27, 3:15 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:15:24Z] <bking@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880

Stashbot added a comment.Mon, Jul 27, 3:15 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:15:29Z] <bking@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 06s)

Stashbot added a comment.Mon, Jul 27, 3:15 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:15:43Z] <bking@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880

Stashbot added a comment.Mon, Jul 27, 3:15 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:15:51Z] <bking@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 12s)

Stashbot added a comment.Mon, Jul 27, 3:21 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:21:16Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards

gerritbot added a comment.Mon, Jul 27, 3:22 PM

Change #1317127 merged by jenkins-bot:

[operations/cookbooks@master] sre.wdqs.data-transfer: abort on failed transfer

https://gerrit.wikimedia.org/r/1317127

Stashbot added a comment.Mon, Jul 27, 3:26 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:26:06Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards

gerritbot added a comment.Mon, Jul 27, 3:26 PM

Change #1317128 merged by jenkins-bot:

[operations/cookbooks@master] sre.wdqs: stop transferpy burying cookbook output

https://gerrit.wikimedia.org/r/1317128

Stashbot added a comment.Mon, Jul 27, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:29:04Z] <bking@cumin2003> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 3:31 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:31:37Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 3:32 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:32:05Z] <bking@cumin2003> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 3:32 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:32:35Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 3:34 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:34:30Z] <bking@cumin2003> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 3:38 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:38:35Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards

bking updated the task description. (Show Details)Mon, Jul 27, 3:39 PM
Stashbot added a comment.Mon, Jul 27, 3:41 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:41:12Z] <bking@cumin2003> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 3:43 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T15:43:19Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 4:52 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T16:52:13Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.Mon, Jul 27, 5:13 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T17:13:19Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards

bking updated the task description. (Show Details)Mon, Jul 27, 5:42 PM
Stashbot added a comment.Mon, Jul 27, 7:47 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T19:47:41Z] <bking@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880

Stashbot added a comment.Mon, Jul 27, 7:47 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T19:47:50Z] <bking@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 46s)

bking updated the task description. (Show Details)Mon, Jul 27, 7:53 PM
Stashbot added a comment.Mon, Jul 27, 8:53 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T20:53:52Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1021.eqiad.wmnet, repooling source-only afterwards

bking updated the task description. (Show Details)Mon, Jul 27, 8:57 PM
Stashbot added a comment.Mon, Jul 27, 9:10 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-27T21:10:34Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs2011.codfw.wmnet, repooling source-only afterwards

Maintenance_bot removed a project: Patch-For-Review.Mon, Jul 27, 9:31 PM
bking updated the task description. (Show Details)Mon, Jul 27, 9:57 PM
Stashbot added a comment.Tue, Jul 28, 1:11 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T13:11:36Z] <bking@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880

Stashbot added a comment.Tue, Jul 28, 1:11 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T13:11:42Z] <bking@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 20s)

Stashbot added a comment.Tue, Jul 28, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T13:27:53Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1022.eqiad.wmnet, repooling source-only afterwards

bking updated the task description. (Show Details)Tue, Jul 28, 1:28 PM
Stashbot added a comment.Tue, Jul 28, 2:23 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T14:23:36Z] <bking@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880

Stashbot added a comment.Tue, Jul 28, 2:23 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T14:23:45Z] <bking@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 19s)

Stashbot added a comment.Tue, Jul 28, 2:36 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T14:36:47Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1022.eqiad.wmnet, repooling source-only afterwards

bking updated the task description. (Show Details)Tue, Jul 28, 3:06 PM
Stashbot added a comment.Tue, Jul 28, 3:07 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T15:07:53Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling source-only afterwards

bking updated the task description. (Show Details)Tue, Jul 28, 3:08 PM
bking updated the task description. (Show Details)Tue, Jul 28, 3:21 PM
Stashbot added a comment.Tue, Jul 28, 4:28 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T16:28:29Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling source-only afterwards

bking updated the task description. (Show Details)Tue, Jul 28, 7:29 PM
Stashbot added a comment.Tue, Jul 28, 8:52 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T20:52:58Z] <bking@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880

Stashbot added a comment.Tue, Jul 28, 8:53 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T20:53:06Z] <bking@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 30s)

Stashbot added a comment.Tue, Jul 28, 8:54 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T20:54:03Z] <bking@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880

Stashbot added a comment.Tue, Jul 28, 8:54 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T20:54:11Z] <bking@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 14s)

bking updated the task description. (Show Details)Tue, Jul 28, 8:54 PM
Stashbot added a comment.Tue, Jul 28, 8:55 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T20:55:18Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Tue, Jul 28, 8:55 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T20:55:47Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Tue, Jul 28, 10:08 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T22:08:06Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Tue, Jul 28, 10:11 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T22:11:19Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards

gerritbot added a comment.Wed, Jul 29, 5:25 AM

Change #1318972 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/software/transferpy@master] Transferer: log the reason a copy failed

https://gerrit.wikimedia.org/r/1318972

gerritbot added a project: Patch-For-Review.Wed, Jul 29, 5:25 AM
bking updated the task description. (Show Details)Wed, Jul 29, 2:48 PM
Stashbot added a comment.Wed, Jul 29, 3:41 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-29T15:41:24Z] <bking@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880

Stashbot added a comment.Wed, Jul 29, 3:41 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-29T15:41:32Z] <bking@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 18s)

Stashbot added a comment.Wed, Jul 29, 3:42 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-29T15:42:23Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards

bking updated the task description. (Show Details)Wed, Jul 29, 3:42 PM
Stashbot added a comment.Wed, Jul 29, 4:54 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-29T16:54:32Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Wed, Jul 29, 5:47 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-29T17:47:25Z] <bking@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880

Stashbot added a comment.Wed, Jul 29, 5:47 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-29T17:47:36Z] <bking@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 14s)

Stashbot added a comment.Wed, Jul 29, 5:47 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-29T17:47:50Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2021.codfw.wmnet, repooling source-only afterwards

bking updated the task description. (Show Details)Wed, Jul 29, 5:50 PM
Stashbot added a comment.Wed, Jul 29, 7:01 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-29T19:01:04Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2021.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Wed, Jul 29, 9:12 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-29T21:12:39Z] <bking@cumin2003> START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2022.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Wed, Jul 29, 10:25 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-29T22:25:39Z] <bking@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2022.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.Thu, Jul 30, 3:43 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-30T15:43:49Z] <bking@deploy1003> Started deploy [wdqs/wdqs@e8fb00c]: T430880

Stashbot added a comment.Thu, Jul 30, 3:44 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-30T15:43:59Z] <bking@deploy1003> Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 24s)

bking closed this task as Resolved.Thu, Jul 30, 3:44 PM
bking updated the task description. (Show Details)
bking updated the task description. (Show Details)
bking moved this task from In Progress to Done on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.

All hosts have migrated off Debian Bullseye:

[bking@cumin2003] ~$ sudo cumin A:wdqs-all 'cat /etc/debian_version'
36 hosts will be targeted:
wdqs[2007-2008,2010-2027].codfw.wmnet,wdqs[1011-1014,1016-1027].eqiad.wmnet
OK to proceed on 36 hosts? Enter the number of affected hosts to confirm or "q" to quit: 36
===== NODE GROUP =====
(36) wdqs[2007-2008,2010-2027].codfw.wmnet,wdqs[1011-1014,1016-1027].eqiad.wmnet
----- OUTPUT for command #1: 'cat /etc/debian_version' -----
12.15
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (36/36) [00:01<00:00, 26.37hosts/s]
FAIL |                                                                                                                                                                           |   0% (0/36) [00:01<?, ?hosts/s]
100.0% (36/36) success ratio (>= 100.0% threshold) for command #1: 'cat /etc/debian_version'.
100.0% (36/36) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

As such, I'm closing out this ticket.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Fri, Aug 7, 7:53 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits