Page MenuHomePhabricator
Paste P96545

(An Untitled Masterwork)
ActivePublic
Actions

Authored by ProdPasteBot on Thu, Oct 1, 8:42 PM.
Project Tags
None
Referenced Files
F105735043: raw-paste-data.txt
Thu, Oct 1, 8:42 PM
Subscribers
None
[bking@cumin2003] ~$ 2026-10-01T20:19:14^C
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:21:13.162041+00:00', 'owner': 'bking
@cumin2003 [2381014]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
Created silence ID e984ee7a-f19a-496d-baa6-183dbae6117c for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:05:15 UTC
===== NODE GROUP =====
(2) an-test-worker[1001,1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:05:14 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.70hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
===== NO OUTPUT =====
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:42<00:00, 14.07s/hosts]
FAIL | | 0% (0/3) [00:42<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...yarn-nodemanager'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:06:00 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:08:15 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.66hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [02:30<00:00, 54.13s/hosts]
^[[B^[[B^[[B^[[B^[[B^[[B^[[B | 0% (0/3) [00:00<?, ?hosts/s]
===== NO OUTPUT =====
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [04:30<00:00, 90.28s/hosts]
FAIL | | 0% (0/3) [04:30<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...op-hdfs-datanode'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:12:48 UTC
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:10:31 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:15:03 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.79hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
PASS | | 0% (0/3) [00:00<?, ?hosts/s^
0.0% (0/3) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
PASS | | 0% (0/3) [00:14<?, ?hosts/s]
Ctrl+c pressed███████████████████████████████▎ | 33% (1/3) [00:14<00:28, 14.18s/hosts]
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:21:13.162041+00:00', 'owner': 'bking
@cumin2003 [2381014]', 'ttl': 1800}
END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10^C
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:27:09.751313+00:00', 'owner': 'bking
@cumin2003 [2384250]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
Created silence ID 0a816126-d5b8-4300-90b9-f1ed18c175e3 for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(3) an-test-worker[1001-1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:21:34 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.78hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-yarn-nodemanager on an-test-worker[1001-1003].eqiad.wmnet: started at Thu 2026-10-01 20:21:34 UTC
No hadoop-yarn-nodemanager daemons need restarting.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:24:35 UTC
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:22:20 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.34hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-datanode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:24:35 UTC
Skipping hadoop-hdfs-datanode on an-test-worker[1001-1002].eqiad.wmnet: started at Thu 2026-10-01 20:22:20 UTC
No hadoop-hdfs-datanode daemons need restarting.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:12:48 UTC
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:26:52 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-09-10 19:15:03 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.40hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-journalnode on an-test-worker1001.eqiad.wmnet: started at Thu 2026-10-01 20:26:52 UTC
===== NO OUTPUT =====
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (2/2) [04:30<00:00, 135.39s/hosts]
FAIL | | 0% (0/2) [04:30<?, ?hosts/s]
100.0% (2/2) success ratio (>= 100.0% threshold) for command #1: 'systemctl restar...hdfs-journalnode'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Deleted silence ID 0a816126-d5b8-4300-90b9-f1ed18c175e3
All jvm restarts completed!
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:27:09.751313+00:00', 'owner': 'bking
@cumin2003 [2384250]', 'ttl': 1800}
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$ test-cookbook -c 1347713 sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:Change exists in project operations/cookbooks with latest patch set being 1
INFO:__main__:Setting up Cookbooks change 1347713 patch set 1 for testing
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 status --porcelain
INFO:__main__:No local modification found, fetching change from Gerrit
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 fetch https://gerrit.wikimedia.org/r/operations/cookbooks refs/chang
es/13/1347713/1
From https://gerrit.wikimedia.org/r/operations/cookbooks
* branch refs/changes/13/1347713/1 -> FETCH_HEAD
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 rev-parse --verify change-1347713-1
1f48f79c00a597709182087b7c6fbf537b5c4106
INFO:__main__:Switching to already existing branch change-1347713-1
INFO:__main__:Executing command /usr/bin/git -C /home/bking/cookbooks_testing/cookbooks-1347713 checkout change-1347713-1
Already on 'change-1347713-1'
INFO:__main__:==================================================
INFO:__main__:Executing: sudo cookbook -c /home/bking/cookbooks_testing/config.yaml sre.hadoop.roll-restart-workers test --start-datetime 2026-10-01T20:19:14
INFO:__main__:==================================================
Acquired lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:34:41.199114+00:00', 'owner': 'bking
@cumin2003 [2385082]', 'ttl': 1800}
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
Scheduling downtime for 2:00:00 on Icinga server alert1002.wikimedia.org for hosts: an-test-worker[1001-1003]
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['an-test-worker1002', 'an-test-worker1001', 'an-test-worker1
003']
Created silence ID 8c08552d-04be-4fd8-baf3-aa3679f469a4 for 2:00:00
Restarting Yarn Nodemanagers with batch size 5 and sleep 30.0..
===== NODE GROUP =====
(3) an-test-worker[1001-1003].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:21:34 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00, 2.72hosts/s]
FAIL | | 0% (0/3) [00:01<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-yarn-nodemanager on an-test-worker[1001-1003].eqiad.wmnet: started at Thu 2026-10-01 20:21:34 UTC
No hadoop-yarn-nodemanager daemons need restarting.
Restarting HDFS Datanodes with batch size 2 and sleep 120.0..
===== NODE GROUP =====
(2) an-test-worker[1001-1002].eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:22:20 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:24:35 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.34hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-datanode on an-test-worker[1001-1002].eqiad.wmnet: started at Thu 2026-10-01 20:22:20 UTC
Skipping hadoop-hdfs-datanode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:24:35 UTC
No hadoop-hdfs-datanode daemons need restarting.
Restarting HDFS Journalnodes with batch size 1 and sleep 120.0..
===== NODE GROUP =====
(1) an-test-worker1001.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:26:52 UTC
===== NODE GROUP =====
(1) an-test-worker1002.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:27:35 UTC
===== NODE GROUP =====
(1) an-test-worker1003.eqiad.wmnet
----- OUTPUT for command #1: 'TZ=UTC systemctl...imestamp --value' -----
Thu 2026-10-01 20:29:50 UTC
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00, 3.39hosts/s]
FAIL | | 0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command #1: 'TZ=UTC systemctl...imestamp --value'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Skipping hadoop-hdfs-journalnode on an-test-worker1001.eqiad.wmnet: started at Thu 2026-10-01 20:26:52 UTC
Skipping hadoop-hdfs-journalnode on an-test-worker1002.eqiad.wmnet: started at Thu 2026-10-01 20:27:35 UTC
Skipping hadoop-hdfs-journalnode on an-test-worker1003.eqiad.wmnet: started at Thu 2026-10-01 20:29:50 UTC
No hadoop-hdfs-journalnode daemons need restarting.
Deleted silence ID 8c08552d-04be-4fd8-baf3-aa3679f469a4
All jvm restarts completed!
Released lock for key /spicerack/locks/cookbooks/sre.hadoop.roll-restart-workers: {'concurrency': 20, 'created': '2026-10-01 20:34:41.199114+00:00', 'owner': 'bking
@cumin2003 [2385082]', 'ttl': 1800}
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
[bking@cumin2003] ~$ less ~/.tmux.conf
[bking@cumin2003] ~$
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits