AS mentioned in T286473, en <> de pair having issue with timeout errors. Investigate and generate template parameter alignments for en > de wikis.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | KartikMistry | T286473 Generate template parameter alignments for additional wikis | |||
| Resolved | KartikMistry | T287655 Generate template parameter alignments for en > de wikis |
Error log:
en
reading word vectors from vectors/wiki.en.vec
reading word vectors from vectors/wiki.de.vec
== de
[Stage 3:===============
====> (70 + 35) / 200]ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1159, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 985, in send_command
response = connection.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1164, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39233)
Traceback (most recent call last):
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_2051/2886107906.py", line 83, in <module>
pairs = df2.toPandas()
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 2143, in toPandas
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
File "/usr/lib/spark2/python/pyspark/sql/dataframe.py", line 534, in collect
sock_info = self._jdf.collectToPython()
File "/usr/lib/spark2/python/pyspark/traceback_utils.py", line 78, in __exit__
self._context._jsc.setCallSite(None)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1255, in __call__
answer = self.gateway_client.send_command(command)
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 983, in send_command
connection = self._get_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 931, in _get_connection
connection = self._create_connection()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 937, in _create_connection
connection.start()
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1079, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/home/kartik/python3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2061, in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JNetworkError' object has no attribute '_render_traceback_'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
---------------------------------------------------------------------------
Py4JError Traceback (most recent call last)
/usr/lib/spark2/python/pyspark/sql/dataframe.py in collect(self)
533 with SCCallSiteSync(self._sc) as css:
--> 534 sock_info = self._jdf.collectToPython()
535 return list(_load_from_socket(sock_info, BatchedSerializer(PickleSerializer())))
/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
/usr/lib/spark2/python/pyspark/sql/utils.py in deco(*a, **kw)
62 try:
---> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
335 "An error occurred while calling {0}{1}{2}".
--> 336 format(target_id, ".", name))
337 else:
Py4JError: An error occurred while calling o72.collectToPython
During handling of the above exception, another exception occurred:
IndexError Traceback (most recent call last)
/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in _get_connection(self)
928 try:
--> 929 connection = self.deque.pop()
930 except IndexError:
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
ConnectionRefusedError Traceback (most recent call last)
/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in start(self)
1066 try:
-> 1067 self.socket.connect((self.address, self.port))
1068 self.stream = self.socket.makefile("rb")
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Py4JNetworkError Traceback (most recent call last)
/tmp/ipykernel_2051/2886107906.py in <module>
81 print('==',lang2_code)
82 df2 = df[df.wiki_db == '%swiki' % lang1_code].join(df[df.wiki_db == '%swiki' % lang2_code].withColumnRenamed("page", "page2").withColumnRenamed('wiki_db','wiki_db2'),on='item_id')
---> 83 pairs = df2.toPandas()
84 bilingual_dictionary = list(zip(pairs['page'],pairs['page2']))
85 ##common words
/usr/lib/spark2/python/pyspark/sql/dataframe.py in toPandas(self)
2141
2142 # Below is toPandas without Arrow optimization.
-> 2143 pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
2144
2145 dtype = {}
/usr/lib/spark2/python/pyspark/sql/dataframe.py in collect(self)
532 """
533 with SCCallSiteSync(self._sc) as css:
--> 534 sock_info = self._jdf.collectToPython()
535 return list(_load_from_socket(sock_info, BatchedSerializer(PickleSerializer())))
536
/usr/lib/spark2/python/pyspark/traceback_utils.py in __exit__(self, type, value, tb)
76 SCCallSiteSync._spark_stack_depth -= 1
77 if SCCallSiteSync._spark_stack_depth == 0:
---> 78 self._context._jsc.setCallSite(None)
/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
1253 proto.END_COMMAND_PART
1254
-> 1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
1257 answer, self.gateway_client, self.target_id, self.name)
/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in send_command(self, command, retry, binary)
981 if `binary` is `True`.
982 """
--> 983 connection = self._get_connection()
984 try:
985 response = connection.send_command(command)
/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in _get_connection(self)
929 connection = self.deque.pop()
930 except IndexError:
--> 931 connection = self._create_connection()
932 return connection
933
/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in _create_connection(self)
935 connection = GatewayConnection(
936 self.gateway_parameters, self.gateway_property)
--> 937 connection.start()
938 return connection
939
/usr/lib/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in start(self)
1077 "server ({0}:{1})".format(self.address, self.port)
1078 logger.exception(msg)
-> 1079 raise Py4JNetworkError(msg, e)
1080
1081 def _authenticate_connection(self):
Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:39233)@diego See logs above. It is latest errors I'm getting while generating en > de pair. Let me know if you need more information.
@KartikMistry this looks like a pyspark configuration issue, which kernel are you using?
Oh got it! This setup has been changed around one year ago. Now we all use the spark environments provided by the JupyterHub.
In short what this means is you should work through the web interface, doing:
ssh -N stat100X.eqiad.wmnet -L 8880:127.0.0.1:8880
go to your browser (localhost:8880) and then login with your LDAP user. Check more details about that setup on the link above.
Then you need to modify your current code to create the spark context, adding the following code on the top
import wmfdata # Get a predefined and preconfigured SparkSession type using get_session. spark = wmfdata.spark.get_session(type='yarn-large')
The rest of the code should work ok. Anyhow, please send me a pointer to the current code you are using, so I can have look to see if there is any other change needed to update to the current Spark setup.
Current code: https://github.com/kartikm/templatesAlignment
Also, it is possible to update code in main repository? (ie https://github.com/digitalTranshumant/templatesAlignment)
I've updated the code here https://github.com/digitalTranshumant/templatesAlignment/blob/master/02alignmentsSpark.ipynb
Pls let me know if you have further question.
@diego I'm getting timeout while running 02alignmentsSpark.ipynb.
There isn't any error but, kernel dies after,
en reading word vectors from vectors/wiki.en.vec
with,
The kernel appears to have died. It will restart automatically.
I tried adjusting memory from ~/.profile with,
pyspark2 --master yarn --deploy-mode client --executor-memory 8g --driver-memory 8g --conf spark.dynamicAllocation.maxExecutors=128
but result is same.
I took el <-> fa pairs (both have small-sized vectors model) to test further with memory issues, but the result seems empty files.
It seems the issue is running with JupyterHub. Redoing the process from scratch and will update it here.
pyspark2 --master yarn --deploy-mode client --executor-memory 8g --driver-memory 8g --conf spark.dynamicAllocation.maxExecutors=128but result is same.
In this case fasttext is running on the driver. You can increase the memory driver when calling the spark env, replacing this:
import wmfdata spark = wmfdata.spark.get_session(type='yarn-regular')
by this:
import wmfdata
spark = wmfdata.spark.get_session(
type='yarn-regular',
extra_settings={
'spark.driver.memory': '12G'
}
)Change 784625 had a related patch set uploaded (by KartikMistry; author: KartikMistry):
[mediawiki/services/cxserver@master] Added template parameter alignments for ckb, cs, eu and de
Change 784625 merged by jenkins-bot:
[mediawiki/services/cxserver@master] Added template parameter alignments for ckb, cs, eu and de
Change 785120 had a related patch set uploaded (by KartikMistry; author: KartikMistry):
[operations/deployment-charts@master] Update cxserver to 2022-04-21-081331-production
Change 785120 merged by jenkins-bot:
[operations/deployment-charts@master] Update cxserver to 2022-04-21-081331-production