spicerack elasticsearch fail
ActivePublic
Actions

Authored by Gehel on Feb 19 2019, 5:46 PM.

Tags

None

Referenced Files

	F28247043: spicerack elasticsearch fail
	Feb 19 2019, 5:46 PM

Subscribers

None

	gehel@cumin1001:/srv/deployment/spicerack$ sudo -i cookbook sre.elasticsearch.rolling-upgrade relforge "upgrade to elasticsearch 5.6.14" --task-id=T215931 --without-lvs
	START - Cookbook sre.elasticsearch.rolling-upgrade
	waiting for clusters to be green
	GET http://relforge1002.eqiad.wmnet:9200/_cluster/health?wait_for_status=green&timeout=1s [status:200 request:0.006s]
	GET http://relforge1002.eqiad.wmnet:9400/_cluster/health?wait_for_status=green&timeout=1s [status:200 request:0.010s]
	Fetch 3 node(s) from relforge to perform rolling restart on
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:200 request:0.027s]
	GET http://relforge1002.eqiad.wmnet:9400/_nodes [status:200 request:0.033s]
	Scheduling downtime on Icinga server icinga2001.wikimedia.org for hosts: relforge1002.eqiad.wmnet
	----- OUTPUT of 'icinga-downtime ...n1001 - T215931"' -----
	================
	PASS: \|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████\| 100% (1/1) [00:00<00:00, 1.51hosts/s]
	FAIL: \| \| 0% (0/1) [00:00<?, ?hosts/s]
	100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'icinga-downtime ...n1001 - T215931"'.
	100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
	Disabling Puppet with reason "upgrade to elasticsearch 5.6.14 - gehel@cumin1001 - T215931" on 1 hosts: relforge1002.eqiad.wmnet
	----- OUTPUT of 'disable-puppet "...n1001 - T215931"' -----
	================
	PASS: \|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████\| 100% (1/1) [00:03<00:00, 3.19s/hosts]
	FAIL: \| \| 0% (0/1) [00:03<?, ?hosts/s]
	100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'disable-puppet "...n1001 - T215931"'.
	100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
	Freezing writes on [<spicerack.elasticsearch_cluster.ElasticsearchCluster object at 0x7f5870e99a58>, <spicerack.elasticsearch_cluster.ElasticsearchCluster object at 0x7f5870e999b0>]
	Freezing all indices in <Elasticsearch([{'host': 'relforge1002.eqiad.wmnet', 'port': 9200}])>
	PUT http://relforge1002.eqiad.wmnet:9200/mw_cirrus_metastore/mw_cirrus_metastore/freeze-everything [status:201 request:0.460s]
	Freezing all indices in <Elasticsearch([{'host': 'relforge1002.eqiad.wmnet', 'port': 9400}])>
	PUT http://relforge1002.eqiad.wmnet:9400/mw_cirrus_metastore/mw_cirrus_metastore/freeze-everything [status:201 request:1.648s]
	Wait for a minimum time of 60sec to make sure all CirrusSearch writes are terminated
	Stopping elasticsearch replication in a safe way on relforge
	stopping replication on [<spicerack.elasticsearch_cluster.ElasticsearchCluster object at 0x7f5870e99a58>, <spicerack.elasticsearch_cluster.ElasticsearchCluster object at 0x7f5870e999b0>]
	stop replication - <Elasticsearch([{'host': 'relforge1002.eqiad.wmnet', 'port': 9200}])>
	Updating cluster settings: {'transient': {'cluster.routing.allocation.enable': 'primaries'}}
	PUT http://relforge1002.eqiad.wmnet:9200/_cluster/settings [status:200 request:0.058s]
	stop replication - <Elasticsearch([{'host': 'relforge1002.eqiad.wmnet', 'port': 9400}])>
	Updating cluster settings: {'transient': {'cluster.routing.allocation.enable': 'primaries'}}
	PUT http://relforge1002.eqiad.wmnet:9400/_cluster/settings [status:200 request:0.059s]
	flush markers on <Elasticsearch([{'host': 'relforge1002.eqiad.wmnet', 'port': 9200}])>
	POST http://relforge1002.eqiad.wmnet:9200/_flush?force=true [status:200 request:6.614s]
	POST http://relforge1002.eqiad.wmnet:9200/_flush/synced [status:200 request:2.966s]
	flush markers on <Elasticsearch([{'host': 'relforge1002.eqiad.wmnet', 'port': 9400}])>
	POST http://relforge1002.eqiad.wmnet:9400/_flush?force=true [status:200 request:0.829s]
	POST http://relforge1002.eqiad.wmnet:9400/_flush/synced [status:200 request:0.159s]
	Stopping all elasticsearch instances on relforge1002.eqiad.wmnet
	----- OUTPUT of 'systemctl stop "...earch_@" --all' -----
	================
	PASS: \|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████\| 100% (1/1) [00:03<00:00, 3.47s/hosts]
	FAIL: \| \| 0% (0/1) [00:03<?, ?hosts/s]
	100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'systemctl stop "...earch_@" --all'.
	100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
	----- OUTPUT of 'apt-get -o Dpkg:...h-search-plugins' -----
	Reading package lists...
	Building dependency tree...
	Reading state information...
	The following package was automatically installed and is no longer required:
	python-configobj
	Use 'apt autoremove' to remove it.
	The following packages will be upgraded:
	elasticsearch wmf-elasticsearch-search-plugins
	2 upgraded, 0 newly installed, 0 to remove and 35 not upgraded.
	Need to get 56.2 MB of archives.
	After this operation, 479 kB of additional disk space will be used.
	Get:1 http://apt.wikimedia.org/wikimedia stretch-wikimedia/component/elastic56 amd64 wmf-elasticsearch-search-plugins all 5.6.14-1~stretch [22.5 MB]
	Get:2 http://apt.wikimedia.org/wikimedia stretch-wikimedia/thirdparty/elastic56 amd64 elasticsearch all 5.6.14 [33.8 MB]
	INFO:debmonitor:Got 2 updates from dpkg hook version 3
	INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server
	debconf: unable to initialize frontend: Dialog
	debconf: (TERM is not set, so the dialog frontend is not usable.)
	debconf: falling back to frontend: Readline
	debconf: unable to initialize frontend: Readline
	debconf: (This frontend requires a controlling tty.)
	debconf: falling back to frontend: Teletype
	dpkg-preconfigure: unable to re-open stdin:
	Fetched 56.2 MB in 1s (48.5 MB/s)
	(Reading database ... 52509 files and directories currently installed.)
	Preparing to unpack .../wmf-elasticsearch-search-plugins_5.6.14-1~stretch_all.deb ...
	Unpacking wmf-elasticsearch-search-plugins (5.6.14-1~stretch) over (5.5.2-8~stretch) ...
	Preparing to unpack .../elasticsearch_5.6.14_all.deb ...
	Unpacking elasticsearch (5.6.14) over (5.5.2) ...
	Setting up elasticsearch (5.6.14) ...
	Installing new version of config file /usr/lib/systemd/system/elasticsearch.service ...
	Installing new version of config file /etc/init.d/elasticsearch ...

	Configuration file '/etc/default/elasticsearch'
	==> Deleted (by you or by a script) since installation.
	==> Package distributor has shipped an updated version.
	==> Keeping old config file as default.
	Processing triggers for systemd (232-25+deb9u9) ...
	Setting up wmf-elasticsearch-search-plugins (5.6.14-1~stretch) ...
	(Reading database ... 100%
	================
	PASS: \|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████\| 100% (1/1) [00:07<00:00, 7.35s/hosts]
	FAIL: \| \| 0% (0/1) [00:07<?, ?hosts/s]
	100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'apt-get -o Dpkg:...h-search-plugins'.
	100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
	Starting all elasticsearch instances on relforge1002.eqiad.wmnet
	----- OUTPUT of 'systemctl start ...earch_@" --all' -----
	================
	PASS: \|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████\| 100% (1/1) [00:00<00:00, 2.33hosts/s]
	FAIL: \| \| 0% (0/1) [00:00<?, ?hosts/s]
	100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'systemctl start ...earch_@" --all'.
	100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
	waiting for elasticsearch instances to come up on relforge1002.eqiad.wmnet
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.002s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870e99c18>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c566a0>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.003s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71b70>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71b38>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [1/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f58702044e0>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870204668>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870204748>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870204828>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [2/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f58702047f0>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71ba8>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71b70>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71780>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [3/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71668>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71160>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71080>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c46a58>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [4/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c46b70>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c46f28>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c564a8>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c56f28>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [5/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c569b0>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c56358>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c564e0>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f58702049b0>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [6/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870204780>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c56438>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c563c8>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c56208>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [7/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.002s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c564a8>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c46f28>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c46b70>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c46ef0>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [8/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c714a8>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c710f0>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c711d0>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.002s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71278>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [9/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71630>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71ba8>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870e99c18>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870204550>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [10/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870204940>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870204a90>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870204ba8>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.003s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870204cc0>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [11/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c46ef0>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c46a90>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c46eb8>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c717b8>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [12/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71828>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71208>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c71128>: Failed to establish a new connection: [Errno 111] Connection refused
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:N/A request:0.001s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
	(self.host, self.port), self.timeout, **extra_kw)
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
	raise err
	File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
	sock.connect(sa)
	ConnectionRefusedError: [Errno 111] Connection refused

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request
	conn.request(method, url, **httplib_request_kw)
	File "/usr/lib/python3.5/http/client.py", line 1107, in request
	self._send_request(method, url, body, headers)
	File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
	self.endheaders(body)
	File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
	self._send_output(message_body)
	File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
	self.send(msg)
	File "/usr/lib/python3.5/http/client.py", line 877, in send
	self.connect()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 163, in connect
	conn = self._new_conn()
	File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
	self, "Failed to establish a new connection: %s" % e)
	urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5870c710b8>: Failed to establish a new connection: [Errno 111] Connection refused
	Failed to call 'spicerack.elasticsearch_cluster.inner_wait' [13/120, retrying in 5.00s]: Could not connect to the cluster
	GET http://relforge1002.eqiad.wmnet:9200/_nodes [status:200 request:0.110s]
	start replication - <Elasticsearch([{'host': 'relforge1002.eqiad.wmnet', 'port': 9400}])>
	Updating cluster settings: {'transient': {'cluster.routing.allocation.enable': 'all'}}
	PUT http://relforge1002.eqiad.wmnet:9400/_cluster/settings [status:200 request:0.085s]
	start replication - <Elasticsearch([{'host': 'relforge1002.eqiad.wmnet', 'port': 9200}])>
	Updating cluster settings: {'transient': {'cluster.routing.allocation.enable': 'all'}}
	PUT http://relforge1002.eqiad.wmnet:9200/_cluster/settings [status:N/A request:10.011s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 391, in _make_request
	six.raise_from(e, None)
	File "<string>", line 2, in raise_from
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
	httplib_response = conn.getresponse()
	File "/usr/lib/python3.5/http/client.py", line 1198, in getresponse
	response.begin()
	File "/usr/lib/python3.5/http/client.py", line 297, in begin
	version, status, reason = self._read_status()
	File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
	line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
	File "/usr/lib/python3.5/socket.py", line 576, in readinto
	return self._sock.recv_into(b)
	socket.timeout: timed out

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 393, in _make_request
	self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 313, in _raise_timeout
	raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
	urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='relforge1002.eqiad.wmnet', port=9200): Read timed out. (read timeout=10)
	Unfreezing all indices in <Elasticsearch([{'host': 'relforge1002.eqiad.wmnet', 'port': 9400}])>
	DELETE http://relforge1002.eqiad.wmnet:9400/mw_cirrus_metastore/mw_cirrus_metastore/freeze-everything [status:200 request:0.325s]
	Unfreezing all indices in <Elasticsearch([{'host': 'relforge1002.eqiad.wmnet', 'port': 9200}])>
	DELETE http://relforge1002.eqiad.wmnet:9200/mw_cirrus_metastore/mw_cirrus_metastore/freeze-everything [status:N/A request:10.012s]
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 391, in _make_request
	six.raise_from(e, None)
	File "<string>", line 2, in raise_from
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
	httplib_response = conn.getresponse()
	File "/usr/lib/python3.5/http/client.py", line 1198, in getresponse
	response.begin()
	File "/usr/lib/python3.5/http/client.py", line 297, in begin
	version, status, reason = self._read_status()
	File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
	line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
	File "/usr/lib/python3.5/socket.py", line 576, in readinto
	return self._sock.recv_into(b)
	socket.timeout: timed out

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 393, in _make_request
	self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 313, in _raise_timeout
	raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
	urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='relforge1002.eqiad.wmnet', port=9200): Read timed out. (read timeout=10)
	Enabling Puppet with reason "upgrade to elasticsearch 5.6.14 - gehel@cumin1001 - T215931" on 1 hosts: relforge1002.eqiad.wmnet
	----- OUTPUT of 'enable-puppet "u...n1001 - T215931"' -----
	================
	PASS: \|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████\| 100% (1/1) [00:03<00:00, 3.23s/hosts]
	FAIL: \| \| 0% (0/1) [00:03<?, ?hosts/s]
	100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'enable-puppet "u...n1001 - T215931"'.
	100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
	----- OUTPUT of 'awk '/^\s*comman...cinga/icinga.cfg' -----
	/var/lib/icinga/rw/icinga.cmd
	================
	PASS: \|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████\| 100% (1/1) [00:00<00:00, 1.49hosts/s]
	FAIL: \| \| 0% (0/1) [00:00<?, ?hosts/s]
	100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'awk '/^\s*comman...cinga/icinga.cfg'.
	100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
	----- OUTPUT of 'echo -n "[155059...ga/rw/icinga.cmd' -----
	================
	PASS: \|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████\| 100% (1/1) [00:00<00:00, 1.44hosts/s]
	FAIL: \| \| 0% (0/1) [00:00<?, ?hosts/s]
	100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'echo -n "[155059...ga/rw/icinga.cmd'.
	100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
	Exception raised while executing cookbook sre.elasticsearch.rolling-upgrade:
	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 391, in _make_request
	six.raise_from(e, None)
	File "<string>", line 2, in raise_from
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
	httplib_response = conn.getresponse()
	File "/usr/lib/python3.5/http/client.py", line 1198, in getresponse
	response.begin()
	File "/usr/lib/python3.5/http/client.py", line 297, in begin
	version, status, reason = self._read_status()
	File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
	line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
	File "/usr/lib/python3.5/socket.py", line 576, in readinto
	return self._sock.recv_into(b)
	socket.timeout: timed out

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
	response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
	_stacktrace=sys.exc_info()[2])
	File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 311, in increment
	raise six.reraise(type(error), error, _stacktrace)
	File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
	raise value
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
	chunked=chunked)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 393, in _make_request
	self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
	File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 313, in _raise_timeout
	raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
	urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='relforge1002.eqiad.wmnet', port=9200): Read timed out. (read timeout=10)

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/spicerack/elasticsearch_cluster.py", line 487, in _unfreeze_writes
	id='freeze-everything')
	File "/usr/lib/python3/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped
	return func(args, params=params, *kwargs)
	File "/usr/lib/python3/dist-packages/elasticsearch/client/__init__.py", line 1078, in delete
	doc_type, id), params=params)
	File "/usr/lib/python3/dist-packages/elasticsearch/transport.py", line 312, in perform_request
	status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
	File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 122, in perform_request
	raise ConnectionTimeout('TIMEOUT', str(e), e)
	elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='relforge1002.eqiad.wmnet', port=9200): Read timed out. (read timeout=10))

	The above exception was the direct cause of the following exception:

	Traceback (most recent call last):
	File "/usr/lib/python3/dist-packages/spicerack/cookbook.py", line 394, in _run
	ret = self.module.run(args, self.spicerack)
	File "/srv/deployment/spicerack/cookbooks/sre/elasticsearch/rolling-upgrade.py", line 33, in run
	args.clustergroup, args.start_datetime, args.with_lvs, upgrade_elasticsearch
	File "/srv/deployment/spicerack/cookbooks/sre/elasticsearch/__init__.py", line 92, in execute_on_clusters
	elasticsearch_clusters.wait_for_green(timedelta(minutes=5))
	File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
	self.gen.throw(type, value, traceback)
	File "/usr/lib/python3/dist-packages/spicerack/elasticsearch_cluster.py", line 193, in frozen_writes
	yield [stack.enter_context(cluster.frozen_writes(reason)) for cluster in self._clusters]
	File "/usr/lib/python3.5/contextlib.py", line 360, in __exit__
	raise exc_details[1]
	File "/usr/lib/python3.5/contextlib.py", line 345, in __exit__
	if cb(*exc_details):
	File "/usr/lib/python3.5/contextlib.py", line 261, in _exit_wrapper
	return cm_exit(cm, *exc_details)
	File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
	self.gen.throw(type, value, traceback)
	File "/usr/lib/python3/dist-packages/spicerack/elasticsearch_cluster.py", line 460, in frozen_writes
	self._unfreeze_writes()
	File "/usr/lib/python3/dist-packages/spicerack/elasticsearch_cluster.py", line 491, in _unfreeze_writes
	) from e
	spicerack.elasticsearch_cluster.ElasticsearchClusterError: Encountered error while deleting document to unfreeze cluster writes
	END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)

Event Timeline

Gehel created this paste.Feb 19 2019, 5:46 PM

spicerack elasticsearch failActivePublicActions

Event Timeline

spicerack elasticsearch fail
ActivePublic
Actions