I can no longer use cumin from labpuppetmaster1001. It can't shell out to anywhere.
andrew@labpuppetmaster1001:~$ sudo cumin --force --timeout 500 -o json "project:testlabs" "hostname"
42 hosts will be targeted:
abogott-proxy-canary.testlabs.eqiad.wmflabs,abogott-puppetclient.testlabs.eqiad.wmflabs,abogott-puppetmaster.testlabs.eqiad.wmflabs,canary[1001-1009,1012-1013,1015-1016,1018-1020,1023,1025-1030]-01.testlabs.eqiad.wmflabs,canary1026-02.testlabs.eqiad.wmflabs,canary-1014-01.testlabs.eqiad.wmflabs,canary-an[1001-1005]-01.testlabs.eqiad.wmflabs,gtirloni-puppetmaster-01.testlabs.eqiad.wmflabs,krenair-t219424-b.testlabs.eqiad.wmflabs,krenair-t219424-c.testlabs.eqiad.wmflabs,krenair-t171188-test.testlabs.eqiad.wmflabs,ldap-stretch.testlabs.eqiad.wmflabs,puppet-compiler-other.testlabs.eqiad.wmflabs,puppet-compiler-tools.testlabs.eqiad.wmflabs,util-abogott-stretch.testlabs.eqiad.wmflabs,util-abogott.testlabs.eqiad.wmflabs
FORCE mode enabled, continuing without confirmation
PASS: | | 0% (0/42) [00:00<?, ?hosts/s]Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: [00:00<?, ?hosts/s]
FAIL: |██▊ | 2% (1/42) [00:10<06:51, 10.03s/hosts]
Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: 03:09, 7.02s/hosts]
FAIL: |██████████████████████████████████████████████████████▊ | 48% (20/42) [00:20<02:01, 5.51s/hosts]
Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password:
FAIL: |████████████████████████████████████████████████████████████████████████████████████▉ | 74% (31/42) [00:30<00:45, 4.13s/hosts]
FAIL: |█████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 95% (40/42) [00:30<00:05, 2.90s/hosts]
===== NODE GROUP =====
(42) abogott-proxy-canary.testlabs.eqiad.wmflabs,abogott-puppetclient.testlabs.eqiad.wmflabs,abogott-puppetmaster.testlabs.eqiad.wmflabs,canary[1001-1009,1012-1013,1015-1016,1018-1020,1023,1025-1030]-01.testlabs.eqiad.wmflabs,canary1026-02.testlabs.eqiad.wmflabs,canary-1014-01.testlabs.eqiad.wmflabs,canary-an[1001-1005]-01.testlabs.eqiad.wmflabs,gtirloni-puppetmaster-01.testlabs.eqiad.wmflabs,krenair-t219424-b.testlabs.eqiad.wmflabs,krenair-t219424-c.testlabs.eqiad.wmflabs,krenair-t171188-test.testlabs.eqiad.wmflabs,ldap-stretch.testlabs.eqiad.wmflabs,puppet-compiler-other.testlabs.eqiad.wmflabs,puppet-compiler-tools.testlabs.eqiad.wmflabs,util-abogott-stretch.testlabs.eqiad.wmflabs,util-abogott.testlabs.eqiad.wmflabs
----- OUTPUT of 'hostname' -----
Connection timed out during banner exchange
================
PASS: | | 0% (0/42) [00:30<?, ?hosts/s]
FAIL: |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (42/42) [00:30<00:00, 1.39hosts/s]
100.0% (42/42) of nodes failed to execute command 'hostname': abogott-proxy-canary.testlabs.eqiad.wmflabs,abogott-puppetclient.testlabs.eqiad.wmflabs,abogott-puppetmaster.testlabs.eqiad.wmflabs,canary[1001-1009,1012-1013,1015-1016,1018-1020,1023,1025-1030]-01.testlabs.eqiad.wmflabs,canary1026-02.testlabs.eqiad.wmflabs,canary-1014-01.testlabs.eqiad.wmflabs,canary-an[1001-1005]-01.testlabs.eqiad.wmflabs,gtirloni-puppetmaster-01.testlabs.eqiad.wmflabs,krenair-t219424-b.testlabs.eqiad.wmflabs,krenair-t219424-c.testlabs.eqiad.wmflabs,krenair-t171188-test.testlabs.eqiad.wmflabs,ldap-stretch.testlabs.eqiad.wmflabs,puppet-compiler-other.testlabs.eqiad.wmflabs,puppet-compiler-tools.testlabs.eqiad.wmflabs,util-abogott-stretch.testlabs.eqiad.wmflabs,util-abogott.testlabs.eqiad.wmflabs
0.0% (0/42) success ratio (< 100.0% threshold) for command: 'hostname'. Aborting.
0.0% (0/42) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
_____FORMATTED_OUTPUT_____
{
"abogott-proxy-canary.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"abogott-puppetclient.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"abogott-puppetmaster.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary-1014-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary-an1001-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary-an1002-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary-an1003-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary-an1004-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary-an1005-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1001-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1002-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1003-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1004-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1005-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1006-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1007-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1008-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1009-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1012-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1013-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1015-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1016-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1018-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1019-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1020-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1023-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1025-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1026-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1026-02.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1027-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1028-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1029-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"canary1030-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"gtirloni-puppetmaster-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"krenair-t171188-test.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"krenair-t219424-b.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"krenair-t219424-c.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"ldap-stretch.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"puppet-compiler-other.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"puppet-compiler-tools.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"util-abogott-stretch.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
"util-abogott.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange"