Page MenuHomePhabricator
Paste P96546

(An Untitled Masterwork)
ActivePublic
Actions

Authored by ProdPasteBot on Thu, Oct 1, 8:55 PM.
Project Tags
None
Referenced Files
F105737011: raw-paste-data.txt
Thu, Oct 1, 8:55 PM
Subscribers
None
[bking@cumin2003] ~$ 2026-10-01T20:19:14^C
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:21:13.162041+00:00', 'owner': 'bking
@cumin2003 [2381014]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
Created silence ID e984ee7a-f19a-496d-baa6-183dbae6117c for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:05:15 UTC
===== NODE GROUP =====
(2) an-test-worker[1001,1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:05:14 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.70hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
===== NO OUTPUT =====
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:42<00:00, 14.07s/hosts]
FAIL | | 0% (0/3) [00:42<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...yarn-nodemanager'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:06:00 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:08:15 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.66hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [02:30<00:00, 54.13s/hosts]
^[[B^[[B^[[B^[[B^[[B^[[B^[[B | 0% (0/3) [00:00<?, ?hosts/s]
===== NO OUTPUT =====
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [04:30<00:00, 90.28s/hosts]
FAIL | | 0% (0/3) [04:30<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...op-hdfs-datanode'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:12:48 UTC
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:10:31 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:15:03 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.79hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
PASS | | 0% (0/3) [00:00<?, ?hosts/s^
0.0% (0/3) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
PASS | | 0% (0/3) [00:14<?, ?hosts/s]
Ctrl+c pressed███████████████████████████████▎ | 33% (1/3) [00:14<00:28, 14.18s/hosts]
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:21:13.162041+00:00', 'owner': 'bking
@cumin2003 [2381014]', 'ttl': 1800}
END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10^C
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:27:09.751313+00:00', 'owner': 'bking
@cumin2003 [2384250]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
Created silence ID 0a816126-d5b8-4300-90b9-f1ed18c175e3 for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(3) an-test-worker[1001-1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:21:34 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.78hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-yarn-nodemanager on an-test-worker[1001-1003].eqiad.wmnet: started at Thu 2026-10-01 20:21:34 UTC
No hadoop-yarn-nodemanager daemons need restarting.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:24:35 UTC
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:22:20 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.34hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-datanode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:24:35 UTC
Skipping hadoop-hdfs-datanode on an-test-worker[1001-1002].eqiad.wmnet: started at Thu 2026-10-01 20:22:20 UTC
No hadoop-hdfs-datanode daemons need restarting.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:12:48 UTC
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:26:52 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:15:03 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.40hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-journalnode on an-test-worker1001.eqiad.wmnet: started at Thu 2026-10-01 20:26:52 UTC
===== NO OUTPUT =====
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (2/2) [04:30<00:00, 135.39s/hosts]
FAIL | | 0% (0/2) [04:30<?, ?hosts/s]
100.0% (2/2) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...hdfs-journalnode'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Deleted silence ID 0a816126-d5b8-4300-90b9-f1ed18c175e3
All jvm restarts completed!
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:27:09.751313+00:00', 'owner': 'bking
@cumin2003 [2384250]', 'ttl': 1800}
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:34:41.199114+00:00', 'owner': 'bking
@cumin2003 [2385082]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['an-test-worker1002', 'an-test-worker1001', 'an-test-worker1
003']
Created silence ID 8c08552d-04be-4fd8-baf3-aa3679f469a4 for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(3) an-test-worker[1001-1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:21:34 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.72hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-yarn-nodemanager on an-test-worker[1001-1003].eqiad.wmnet: started at Thu 2026-10-01 20:21:34 UTC
No hadoop-yarn-nodemanager daemons need restarting.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:22:20 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:24:35 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.34hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-datanode on an-test-worker[1001-1002].eqiad.wmnet: started at Thu 2026-10-01 20:22:20 UTC
Skipping hadoop-hdfs-datanode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:24:35 UTC
No hadoop-hdfs-datanode daemons need restarting.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:26:52 UTC
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:27:35 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:29:50 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.39hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-journalnode on an-test-worker1001.eqiad.wmnet: started at Thu 2026-10-01 20:26:52 UTC
Skipping hadoop-hdfs-journalnode on an-test-worker1002.eqiad.wmnet: started at Thu 2026-10-01 20:27:35 UTC
Skipping hadoop-hdfs-journalnode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:29:50 UTC
No hadoop-hdfs-journalnode daemons need restarting.
Deleted silence ID 8c08552d-04be-4fd8-baf3-aa3679f469a4
All jvm restarts completed!
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:34:41.199114+00:00', 'owner': 'bking
@cumin2003 [2385082]', 'ttl': 1800}
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$ less ~/.tmux.conf
[bking@cumin2003] ~$ ls -lhtra
total 600K
-rw-r--r-- 1 bking wikidev 575 Feb 17 2026 .tmux.conf
-rw-r--r-- 1 bking wikidev 311 Apr 8 15:22 .bash_profile
-rw-r--r-- 1 bking wikidev 181K Jun 30 18:24 cluster_manager.json
-rw-r--r-- 1 bking wikidev 9.3K Jun 30 21:24 codfw-nodes.json
-rw-r--r-- 1 bking wikidev 23K Jul 7 02:40 shards.json
-rw-r--r-- 1 bking wikidev 94 Jul 8 20:47 21.json
-rw------- 1 bking wikidev 7.6K Jul 28 20:38 .viminfo
-rw-r--r-- 1 bking wikidev 392 Jul 28 20:38 snaps.json
-rw-r--r-- 1 bking wikidev 38K Aug 4 22:10 codfw_omega_settings.json
-rw-r--r-- 1 bking wikidev 104 Aug 26 14:21 diff.txt
-rw-r--r-- 1 bking wikidev 2 Aug 26 14:22 diff2.txt
-rw-r--r-- 1 bking wikidev 228K Aug 26 14:22 changes.txt
drwx------ 2 bking wikidev 4.0K Sep 16 13:47 .config
drwx------ 3 bking wikidev 4.0K Sep 16 13:47 .cache
drwxr-xr-x 78 root root 4.0K Sep 28 18:52 ..
drwx------ 2 bking wikidev 4.0K Sep 29 21:16 .ssh
-rw------- 1 bking wikidev 26K Oct 1 19:51 .bash_history
drwxr-xr-x 8 bking wikidev 4.0K Oct 1 20:34 cookbooks_testing
-rw------- 1 bking wikidev 145 Oct 1 20:41 .lesshst
-rw-r--r-- 1 bking wikidev 23K Oct 1 20:41 roll-restart.txt
drwxr-xr-x 6 bking wikidev 4.0K Oct 1 20:41 .
[bking@cumin2003] ~$ cat roll-restart.txt
[bking@cumin2003] ~$ 2026-10-01T20:19:14^C
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:21:13.162041+00:00', 'owner': 'bking
@cumin2003 [2381014]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
Created silence ID e984ee7a-f19a-496d-baa6-183dbae6117c for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:05:15 UTC
===== NODE GROUP =====
(2) an-test-worker[1001,1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:05:14 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.70hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
===== NO OUTPUT =====
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:42<00:00, 14.07s/hosts]
FAIL | | 0% (0/3) [00:42<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...yarn-nodemanager'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:06:00 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:08:15 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.66hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [02:30<00:00, 54.13s/hosts]
^[[B^[[B^[[B^[[B^[[B^[[B^[[B | 0% (0/3) [00:00<?, ?hosts/s]
===== NO OUTPUT =====
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [04:30<00:00, 90.28s/hosts]
FAIL | | 0% (0/3) [04:30<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...op-hdfs-datanode'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:12:48 UTC
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:10:31 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:15:03 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.79hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
PASS | | 0% (0/3) [00:00<?, ?hosts/s^
0.0% (0/3) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
PASS | | 0% (0/3) [00:14<?, ?hosts/s]
Ctrl+c pressed███████████████████████████████▎ | 33% (1/3) [00:14<00:28, 14.18s/hosts]
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:21:13.162041+00:00', 'owner': 'bking
@cumin2003 [2381014]', 'ttl': 1800}
END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10^C
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:27:09.751313+00:00', 'owner': 'bking
@cumin2003 [2384250]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
Created silence ID 0a816126-d5b8-4300-90b9-f1ed18c175e3 for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(3) an-test-worker[1001-1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:21:34 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.78hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-yarn-nodemanager on an-test-worker[1001-1003].eqiad.wmnet: started at Thu 2026-10-01 20:21:34 UTC
No hadoop-yarn-nodemanager daemons need restarting.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:24:35 UTC
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:22:20 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.34hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-datanode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:24:35 UTC
Skipping hadoop-hdfs-datanode on an-test-worker[1001-1002].eqiad.wmnet: started at Thu 2026-10-01 20:22:20 UTC
No hadoop-hdfs-datanode daemons need restarting.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:12:48 UTC
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:26:52 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:15:03 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.40hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-journalnode on an-test-worker1001.eqiad.wmnet: started at Thu 2026-10-01 20:26:52 UTC
===== NO OUTPUT =====
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (2/2) [04:30<00:00, 135.39s/hosts]
FAIL | | 0% (0/2) [04:30<?, ?hosts/s]
100.0% (2/2) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...hdfs-journalnode'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Deleted silence ID 0a816126-d5b8-4300-90b9-f1ed18c175e3
All jvm restarts completed!
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:27:09.751313+00:00', 'owner': 'bking
@cumin2003 [2384250]', 'ttl': 1800}
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:34:41.199114+00:00', 'owner': 'bking
@cumin2003 [2385082]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['an-test-worker1002', 'an-test-worker1001', 'an-test-worker1
003']
Created silence ID 8c08552d-04be-4fd8-baf3-aa3679f469a4 for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(3) an-test-worker[1001-1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:21:34 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.72hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-yarn-nodemanager on an-test-worker[1001-1003].eqiad.wmnet: started at Thu 2026-10-01 20:21:34 UTC
No hadoop-yarn-nodemanager daemons need restarting.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:22:20 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:24:35 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.34hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-datanode on an-test-worker[1001-1002].eqiad.wmnet: started at Thu 2026-10-01 20:22:20 UTC
Skipping hadoop-hdfs-datanode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:24:35 UTC
No hadoop-hdfs-datanode daemons need restarting.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:26:52 UTC
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:27:35 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:29:50 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.39hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-journalnode on an-test-worker1001.eqiad.wmnet: started at Thu 2026-10-01 20:26:52 UTC
Skipping hadoop-hdfs-journalnode on an-test-worker1002.eqiad.wmnet: started at Thu 2026-10-01 20:27:35 UTC
Skipping hadoop-hdfs-journalnode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:29:50 UTC
No hadoop-hdfs-journalnode daemons need restarting.
Deleted silence ID 8c08552d-04be-4fd8-baf3-aa3679f469a4
All jvm restarts completed!
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:34:41.199114+00:00', 'owner': 'bking
@cumin2003 [2385082]', 'ttl': 1800}
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$ less ~/.tmux.conf
[bking@cumin2003] ~$
[bking@cumin2003] ~$ ls -lhtra
total 600K
-rw-r--r-- 1 bking wikidev 575 Feb 17 2026 .tmux.conf
-rw-r--r-- 1 bking wikidev 311 Apr 8 15:22 .bash_profile
-rw-r--r-- 1 bking wikidev 181K Jun 30 18:24 cluster_manager.json
-rw-r--r-- 1 bking wikidev 9.3K Jun 30 21:24 codfw-nodes.json
-rw-r--r-- 1 bking wikidev 23K Jul 7 02:40 shards.json
-rw-r--r-- 1 bking wikidev 94 Jul 8 20:47 21.json
-rw------- 1 bking wikidev 7.6K Jul 28 20:38 .viminfo
-rw-r--r-- 1 bking wikidev 392 Jul 28 20:38 snaps.json
-rw-r--r-- 1 bking wikidev 38K Aug 4 22:10 codfw_omega_settings.json
-rw-r--r-- 1 bking wikidev 104 Aug 26 14:21 diff.txt
-rw-r--r-- 1 bking wikidev 2 Aug 26 14:22 diff2.txt
-rw-r--r-- 1 bking wikidev 228K Aug 26 14:22 changes.txt
drwx------ 2 bking wikidev 4.0K Sep 16 13:47 .config
drwx------ 3 bking wikidev 4.0K Sep 16 13:47 .cache
drwxr-xr-x 78 root root 4.0K Sep 28 18:52 ..
drwx------ 2 bking wikidev 4.0K Sep 29 21:16 .ssh
-rw------- 1 bking wikidev 26K Oct 1 19:51 .bash_history
drwxr-xr-x 8 bking wikidev 4.0K Oct 1 20:34 cookbooks_testing
-rw------- 1 bking wikidev 145 Oct 1 20:41 .lesshst
-rw-r--r-- 1 bking wikidev 23K Oct 1 20:41 roll-restart.txt
drwxr-xr-x 6 bking wikidev 4.0K Oct 1 20:41 .
[bking@cumin2003] ~$ cat roll-restart.txt | phaste
https://phabricator.wikimedia.org/P96545
[bking@cumin2003] ~$ ^C
[bking@cumin2003] ~$ ls -lhtra^C
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:22
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:22
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:44:35.991339+00:00', 'owner': 'bking
@cumin2003 [2386251]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['an-test-worker1003']
Created silence ID eef58f96-ca70-435c-b075-a8ececf9a587 for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(3) an-test-worker[1001-1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:21:34 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.77hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
===== NO OUTPUT =====
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:42<00:00, 14.06s/hosts]
FAIL | | 0% (0/3) [00:42<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...yarn-nodemanager'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:22:20 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:24:35 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.68hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-datanode on an-test-worker[1001-1002].eqiad.wmnet: started at Thu 2026-10-01 20:22:20 UTC
Skipping hadoop-hdfs-datanode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:24:35 UTC
No hadoop-hdfs-datanode daemons need restarting.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:26:52 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:29:50 UTC
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:27:35 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.40hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-journalnode on an-test-worker1001.eqiad.wmnet: started at Thu 2026-10-01 20:26:52 UTC
Skipping hadoop-hdfs-journalnode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:29:50 UTC
Skipping hadoop-hdfs-journalnode on an-test-worker1002.eqiad.wmnet: started at Thu 2026-10-01 20:27:35 UTC
No hadoop-hdfs-journalnode daemons need restarting.
Deleted silence ID eef58f96-ca70-435c-b075-a8ececf9a587
All jvm restarts completed!
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:44:35.991339+00:00', 'owner': 'bking
@cumin2003 [2386251]', 'ttl': 1800}
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:27
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:27
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:46:30.096408+00:00', 'owner': 'bking
@cumin2003 [2386548]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['an-test-worker1002', 'an-test-worker1001', 'an-test-worker1
003']
Created silence ID f1b356c1-9156-49e2-8410-6525b8118d66 for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(3) an-test-worker[1001-1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:45:08 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.79hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-yarn-nodemanager on an-test-worker[1001-1003].eqiad.wmnet: started at Thu 2026-10-01 20:45:08 UTC
No hadoop-yarn-nodemanager daemons need restarting.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:22:20 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:24:35 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.42hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
===== NO OUTPUT =====
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [04:30<00:00, 90.26s/hosts]
FAIL | | 0% (0/3) [04:30<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...op-hdfs-datanode'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:26:52 UTC
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:27:35 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:29:50 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.76hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-journalnode on an-test-worker1002.eqiad.wmnet: started at Thu 2026-10-01 20:27:35 UTC
Skipping hadoop-hdfs-journalnode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:29:50 UTC
----- OUTPUT for command #1: 'systemctl restar...hdfs-journalnode' -----
================
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [02:15<00:00, 135.30s/hosts]
FAIL | | 0% (0/1) [02:15<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...hdfs-journalnode'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Deleted silence ID f1b356c1-9156-49e2-8410-6525b8118d66
All jvm restarts completed!
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:46:30.096408+00:00', 'owner': 'bking
@cumin2003 [2386548]', 'ttl': 1800}
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits