Page MenuHomePhabricator
Paste P12529

(An Untitled Masterwork)
ActivePublic

Authored by Marostegui on Sep 9 2020, 7:07 AM.
packet_write_wait: Connection to 10.192.0.141 port 22: Broken pipe
About to transfer /srv/sqldata from es2014.codfw.wmnet to ['es2026.codfw.wmnet']:['/srv/'] (9366738014294 bytes)
----- OUTPUT of 'netstat -altn | ...[0-9]$ || echo 0' -----
0
================
PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 2.48hosts/s]
FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'netstat -altn | ...[0-9]$ || echo 0'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of '/sbin/iptables -...t 4400 -j ACCEPT' -----
================
PASS |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 2.56hosts/s]
FAIL | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: '/sbin/iptables -...t 4400 -j ACCEPT'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of '/bin/bash -c "cd...odfw.wmnet 4400"' -----
================
PASS | | 0% (0/1) [25:01<?, ?hosts/s]
FAIL |█████████████████████████████████████████████████████████████████████████████| 100% (1/1) [25:01<00:00, 1501.78s/hosts]
100.0% (1/1) of nodes failed to execute command '/bin/bash -c "cd...odfw.wmnet 4400"': es2014.codfw.wmnet
100.0% (1/1) of nodes failed to execute command '/bin/bash -c "cd...odfw.wmnet 4400"': es2014.codfw.wmnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: '/bin/bash -c "cd...odfw.wmnet 4400"'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) for command: '/bin/bash -c "cd...odfw.wmnet 4400"'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
----- OUTPUT of '/sbin/iptables -...t 4400 -j ACCEPT' -----
iptables: Bad rule (does a matching rule exist in that chain?).
================
PASS | | 0% (0/1) [00:00<?, ?hosts/s]
FAIL |███████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00, 2.09hosts/s]
100.0% (1/1) of nodes failed to execute command '/sbin/iptables -...t 4400 -j ACCEPT': es2026.codfw.wmnet
100.0% (1/1) of nodes failed to execute command '/sbin/iptables -...t 4400 -j ACCEPT': es2026.codfw.wmnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: '/sbin/iptables -...t 4400 -j ACCEPT'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) for command: '/sbin/iptables -...t 4400 -j ACCEPT'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
WARNING: Firewall's temporary rule could not be deleted
ERROR: Copy from es2014.codfw.wmnet:/srv/sqldata to es2026.codfw.wmnet:/srv/ failed

Event Timeline

----- OUTPUT of '/bin/rmdir /tmp/....wmnet_4400.lock' -----
/bin/rmdir: failed to remove '/tmp/trnsfr_es2026.codfw.wmnet_4400.lock': Directory not empty
================
PASS |                                                                                                                                                                                                                                |   0% (0/1) [00:00<?, ?hosts/s]
FAIL |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00,  2.56hosts/s]
100.0% (1/1) of nodes failed to execute command '/bin/rmdir /tmp/....wmnet_4400.lock': es2026.codfw.wmnet
100.0% (1/1) of nodes failed to execute command '/bin/rmdir /tmp/....wmnet_4400.lock': es2026.codfw.wmnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: '/bin/rmdir /tmp/....wmnet_4400.lock'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) for command: '/bin/rmdir /tmp/....wmnet_4400.lock'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
WARNING: /tmp/trnsfr_es2026.codfw.wmnet_4400.lock temporary directory could not be deleted
2020-09-09 07:59:49  WARNING: Firewall's temporary rule could not be deleted
2020-09-09 07:59:49  INFO: Cleaning up....