Page MenuHomePhabricator

wmcs cumin broken
Closed, ResolvedPublic

Description

I can no longer use cumin from labpuppetmaster1001. It can't shell out to anywhere.

andrew@labpuppetmaster1001:~$ sudo cumin --force --timeout 500 -o json  "project:testlabs" "hostname"
42 hosts will be targeted:
abogott-proxy-canary.testlabs.eqiad.wmflabs,abogott-puppetclient.testlabs.eqiad.wmflabs,abogott-puppetmaster.testlabs.eqiad.wmflabs,canary[1001-1009,1012-1013,1015-1016,1018-1020,1023,1025-1030]-01.testlabs.eqiad.wmflabs,canary1026-02.testlabs.eqiad.wmflabs,canary-1014-01.testlabs.eqiad.wmflabs,canary-an[1001-1005]-01.testlabs.eqiad.wmflabs,gtirloni-puppetmaster-01.testlabs.eqiad.wmflabs,krenair-t219424-b.testlabs.eqiad.wmflabs,krenair-t219424-c.testlabs.eqiad.wmflabs,krenair-t171188-test.testlabs.eqiad.wmflabs,ldap-stretch.testlabs.eqiad.wmflabs,puppet-compiler-other.testlabs.eqiad.wmflabs,puppet-compiler-tools.testlabs.eqiad.wmflabs,util-abogott-stretch.testlabs.eqiad.wmflabs,util-abogott.testlabs.eqiad.wmflabs
FORCE mode enabled, continuing without confirmation
PASS:  |                                  |   0% (0/42) [00:00<?, ?hosts/s]Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password:  [00:00<?, ?hosts/s]     

FAIL:  |██▊                                                                                                                 |   2% (1/42) [00:10<06:51, 10.03s/hosts]












                                                                                                                                                                     Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: 03:09,  7.02s/hosts]





FAIL:  |██████████████████████████████████████████████████████▊                                                            |  48% (20/42) [00:20<02:01,  5.51s/hosts]








Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: Password: 

FAIL:  |████████████████████████████████████████████████████████████████████████████████████▉                              |  74% (31/42) [00:30<00:45,  4.13s/hosts]








FAIL:  |█████████████████████████████████████████████████████████████████████████████████████████████████████████████▌     |  95% (40/42) [00:30<00:05,  2.90s/hosts]
===== NODE GROUP =====                                                          
(42) abogott-proxy-canary.testlabs.eqiad.wmflabs,abogott-puppetclient.testlabs.eqiad.wmflabs,abogott-puppetmaster.testlabs.eqiad.wmflabs,canary[1001-1009,1012-1013,1015-1016,1018-1020,1023,1025-1030]-01.testlabs.eqiad.wmflabs,canary1026-02.testlabs.eqiad.wmflabs,canary-1014-01.testlabs.eqiad.wmflabs,canary-an[1001-1005]-01.testlabs.eqiad.wmflabs,gtirloni-puppetmaster-01.testlabs.eqiad.wmflabs,krenair-t219424-b.testlabs.eqiad.wmflabs,krenair-t219424-c.testlabs.eqiad.wmflabs,krenair-t171188-test.testlabs.eqiad.wmflabs,ldap-stretch.testlabs.eqiad.wmflabs,puppet-compiler-other.testlabs.eqiad.wmflabs,puppet-compiler-tools.testlabs.eqiad.wmflabs,util-abogott-stretch.testlabs.eqiad.wmflabs,util-abogott.testlabs.eqiad.wmflabs
----- OUTPUT of 'hostname' -----                                                
Connection timed out during banner exchange                                     
================                                                                
PASS:  |                                  |   0% (0/42) [00:30<?, ?hosts/s]     
FAIL:  |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (42/42) [00:30<00:00,  1.39hosts/s]
100.0% (42/42) of nodes failed to execute command 'hostname': abogott-proxy-canary.testlabs.eqiad.wmflabs,abogott-puppetclient.testlabs.eqiad.wmflabs,abogott-puppetmaster.testlabs.eqiad.wmflabs,canary[1001-1009,1012-1013,1015-1016,1018-1020,1023,1025-1030]-01.testlabs.eqiad.wmflabs,canary1026-02.testlabs.eqiad.wmflabs,canary-1014-01.testlabs.eqiad.wmflabs,canary-an[1001-1005]-01.testlabs.eqiad.wmflabs,gtirloni-puppetmaster-01.testlabs.eqiad.wmflabs,krenair-t219424-b.testlabs.eqiad.wmflabs,krenair-t219424-c.testlabs.eqiad.wmflabs,krenair-t171188-test.testlabs.eqiad.wmflabs,ldap-stretch.testlabs.eqiad.wmflabs,puppet-compiler-other.testlabs.eqiad.wmflabs,puppet-compiler-tools.testlabs.eqiad.wmflabs,util-abogott-stretch.testlabs.eqiad.wmflabs,util-abogott.testlabs.eqiad.wmflabs
0.0% (0/42) success ratio (< 100.0% threshold) for command: 'hostname'. Aborting.
0.0% (0/42) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
_____FORMATTED_OUTPUT_____
{
    "abogott-proxy-canary.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "abogott-puppetclient.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "abogott-puppetmaster.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary-1014-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary-an1001-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary-an1002-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary-an1003-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary-an1004-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary-an1005-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1001-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1002-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1003-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1004-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1005-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1006-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1007-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1008-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1009-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1012-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1013-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1015-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1016-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1018-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1019-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1020-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1023-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1025-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1026-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1026-02.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1027-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1028-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1029-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "canary1030-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "gtirloni-puppetmaster-01.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "krenair-t171188-test.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "krenair-t219424-b.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "krenair-t219424-c.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "ldap-stretch.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "puppet-compiler-other.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "puppet-compiler-tools.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "util-abogott-stretch.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange",
    "util-abogott.testlabs.eqiad.wmflabs": "Connection timed out during banner exchange"

Event Timeline

Andrew created this task.Apr 15 2019, 2:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 15 2019, 2:43 PM

Mentioned in SAL (#wikimedia-cloud) [2019-04-15T16:10:14Z] <Krenair> Changed bastion-restricted* hieradata to account for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/500640/ T220990

Krenair closed this task as Resolved.Apr 15 2019, 4:13 PM
Krenair added subscribers: aborrero, Volans, Krenair.

Fixed with a lot of help from @Volans and @aborrero