Page MenuHomePhabricator
Paste P3366

cassandra bootstrap failures
ActivePublic
Actions

Authored by fgiunchedi on Jul 8 2016, 12:20 PM.
Tags
None
Referenced Files
F4252514: cassandra bootstrap failures
Jul 8 2016, 12:20 PM
Subscribers
None
ERROR [STREAM-IN-/10.64.48.135] 2016-07-08 01:42:01,480 StreamSession.java:640 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Remote peer 10.64.48.135 failed stream session.
WARN [CompactionExecutor:424] 2016-07-08 04:01:49,364 BigTableWriter.java:184 - Writing large partition local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data:en.wikipedia.org:Wikipedia\:VIAF/errors (158159893 bytes)
ERROR [STREAM-IN-/10.64.48.138] 2016-07-08 05:04:45,690 StreamSession.java:640 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Remote peer 10.64.48.138 failed stream session.
ERROR [STREAM-IN-/10.64.48.120] 2016-07-08 05:19:13,758 StreamSession.java:640 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Remote peer 10.64.48.120 failed stream session.
WARN [CompactionExecutor:424] 2016-07-08 05:43:22,045 BigTableWriter.java:184 - Writing large partition local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data:en.wikipedia.org:Wikipedia\:JSTOR/Errors (216237951 bytes)
ERROR [STREAM-IN-/10.64.48.137] 2016-07-08 07:15:54,599 StreamSession.java:640 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Remote peer 10.64.48.137 failed stream session.
ERROR [STREAM-IN-/10.64.48.136] 2016-07-08 08:19:54,151 StreamSession.java:640 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Remote peer 10.64.48.136 failed stream session.
ERROR [STREAM-IN-/10.64.48.130] 2016-07-08 08:41:04,719 StreamSession.java:640 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Remote peer 10.64.48.130 failed stream session.
ERROR [STREAM-IN-/10.64.48.139] 2016-07-08 09:02:16,004 StreamSession.java:640 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Remote peer 10.64.48.139 failed stream session.
ERROR [main] 2016-07-08 09:02:16,333 StorageService.java:1268 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted.
== 1009 errors
restbase1009:/var/log/cassandra$ grep -A10 'ERROR.*fe6ff550-4488-11e6-85be-e77a66119977' system-a.log
ERROR [STREAM-IN-/10.64.48.131] 2016-07-08 04:52:09,525 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
java.net.SocketTimeoutException: null
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) ~[na:1.8.0_91]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_91]
at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_91]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268) ~[apache-cassandra-2.2.6.jar:2.2.6]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
WARN [CompactionExecutor:3022] 2016-07-08 05:00:01,284 BigTableWriter.java:184 - Writing large partition system/hints:c42d329c-68d5-4d1b-b886-90aa752fb288 (522455462 bytes)
WARN [HintedHandoffManager:1] 2016-07-08 05:00:03,875 HintedHandoffMetrics.java:80 - /10.192.32.125 has 1626 dropped hints, because node is down past configured hint window.
INFO [Service Thread] 2016-07-08 05:08:53,571 GCInspector.java:284 - G1 Young Generation GC in 226ms. G1 Eden Space: 7281311744 -> 0; G1 Old Gen: 2868728320 -> 2878496264; G1 Survivor Space: 444596224 -> 629145600;
--
ERROR [STREAM-OUT-/10.64.48.131] 2016-07-08 05:19:12,232 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:125) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:565) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:370) ~[apache-cassandra-2.2.6.jar:2.2.6]
restbase1009:/var/log/cassandra$ grep -A10 'ERROR.*fe6ff550-4488-11e6-85be-e77a66119977' system-b.log
ERROR [STREAM-IN-/10.64.48.131] 2016-07-08 06:08:48,272 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
java.net.SocketTimeoutException: null
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) ~[na:1.8.0_91]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_91]
at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_91]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268) ~[apache-cassandra-2.2.6.jar:2.2.6]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
INFO [ScheduledTasks:1] 2016-07-08 06:11:08,316 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 18 for internal timeout and 0 for cross node timeout
INFO [ScheduledTasks:1] 2016-07-08 06:11:08,316 StatusLogger.java:52 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2016-07-08 06:11:08,331 StatusLogger.java:56 - MutationStage 0 0 7103724 0 0
--
ERROR [STREAM-OUT-/10.64.48.131] 2016-07-08 08:41:03,235 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:125) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:565) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:370) ~[apache-cassandra-2.2.6.jar:2.2.6]
restbase1009:/var/log/cassandra$
== 1015 errors
restbase1015:/var/log/cassandra$ grep -A10 'ERROR.*fe6ff550-4488-11e6-85be-e77a66119977' system-*.log
system-a.log:ERROR [STREAM-IN-/10.64.48.131] 2016-07-08 04:52:21,348 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
system-a.log-java.net.SocketTimeoutException: null
system-a.log- at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) ~[na:1.8.0_91]
system-a.log- at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_91]
system-a.log- at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_91]
system-a.log- at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-a.log- at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-a.log- at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
system-a.log-WARN [CompactionExecutor:1014] 2016-07-08 04:57:00,626 BigTableWriter.java:184 - Writing large partition system/hints:c42d329c-68d5-4d1b-b886-90aa752fb288 (459981414 bytes)
system-a.log-WARN [HintedHandoffManager:1] 2016-07-08 04:57:01,777 HintedHandoffMetrics.java:80 - /10.192.32.125 has 1150 dropped hints, because node is down past configured hint window.
system-a.log-INFO [STREAM-IN-/10.64.48.131] 2016-07-08 05:04:45,446 StreamResultFuture.java:182 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Session with /10.64.48.131 is complete
system-a.log:ERROR [STREAM-OUT-/10.64.48.131] 2016-07-08 05:04:45,490 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
system-a.log-java.lang.AssertionError: Memory was freed
system-a.log- at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-a.log- at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-a.log- at org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-a.log- at org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:125) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-a.log- at org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:565) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-a.log- at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-a.log- at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-a.log- at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-a.log- at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:370) ~[apache-cassandra-2.2.6.jar:2.2.6]
--
system-b.log:ERROR [STREAM-IN-/10.64.48.131] 2016-07-08 08:36:04,965 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
system-b.log-java.net.SocketTimeoutException: null
system-b.log- at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) ~[na:1.8.0_91]
system-b.log- at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_91]
system-b.log- at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_91]
system-b.log- at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-b.log- at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-b.log- at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
system-b.log-INFO [Service Thread] 2016-07-08 08:43:53,716 GCInspector.java:284 - G1 Young Generation GC in 216ms. G1 Eden Space: 7306477568 -> 0; G1 Old Gen: 2637334528 -> 2649365512; G1 Survivor Space: 419430400 -> 427819008;
system-b.log-INFO [Service Thread] 2016-07-08 08:43:53,717 StatusLogger.java:52 - Pool Name Active Pending Completed Blocked All Time Blocked
system-b.log-INFO [Service Thread] 2016-07-08 08:43:53,733 StatusLogger.java:56 - MutationStage 0 0 3571809 0 0
--
system-b.log:ERROR [STREAM-OUT-/10.64.48.131] 2016-07-08 09:02:15,276 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
system-b.log-java.lang.AssertionError: Memory was freed
system-b.log- at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-b.log- at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-b.log- at org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-b.log- at org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:125) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-b.log- at org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:565) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-b.log- at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-b.log- at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-b.log- at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.6.jar:2.2.6]
system-b.log- at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:370) ~[apache-cassandra-2.2.6.jar:2.2.6]
== 1014 errors
restbase1014:~$ grep -A10 'ERROR.*fe6ff550-4488-11e6-85be-e77a66119977' /var/log/cassandra/system*.log
/var/log/cassandra/system-a.log:ERROR [STREAM-IN-/10.64.48.131] 2016-07-08 01:38:25,913 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
/var/log/cassandra/system-a.log-java.net.SocketTimeoutException: null
/var/log/cassandra/system-a.log- at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) ~[na:1.8.0_91]
/var/log/cassandra/system-a.log- at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_91]
/var/log/cassandra/system-a.log- at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_91]
/var/log/cassandra/system-a.log- at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-a.log- at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-a.log- at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
/var/log/cassandra/system-a.log-WARN [CompactionExecutor:754] 2016-07-08 01:38:40,238 BigTableWriter.java:184 - Writing large partition system/hints:c42d329c-68d5-4d1b-b886-90aa752fb288 (479449765 bytes)
/var/log/cassandra/system-a.log-WARN [HintedHandoffManager:1] 2016-07-08 01:38:41,811 HintedHandoffMetrics.java:80 - /10.192.32.125 has 1023 dropped hints, because node is down past configured hint window.
/var/log/cassandra/system-a.log-INFO [Service Thread] 2016-07-08 01:39:51,466 GCInspector.java:284 - G1 Young Generation GC in 273ms. G1 Eden Space: 234881024 -> 0; G1 Old Gen: 4449695232 -> 3577995792; G1 Survivor Space: 402653184 -> 83886080;
--
/var/log/cassandra/system-a.log:ERROR [STREAM-OUT-/10.64.48.131] 2016-07-08 01:41:58,727 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
/var/log/cassandra/system-a.log-java.lang.AssertionError: Memory was freed
/var/log/cassandra/system-a.log- at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-a.log- at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-a.log- at org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-a.log- at org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:125) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-a.log- at org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:565) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-a.log- at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-a.log- at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-a.log- at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-a.log- at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:370) ~[apache-cassandra-2.2.6.jar:2.2.6]
--
/var/log/cassandra/system-b.log:ERROR [STREAM-IN-/10.64.48.131] 2016-07-08 07:34:03,374 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
/var/log/cassandra/system-b.log-java.net.SocketTimeoutException: null
/var/log/cassandra/system-b.log- at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) ~[na:1.8.0_91]
/var/log/cassandra/system-b.log- at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_91]
/var/log/cassandra/system-b.log- at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_91]
/var/log/cassandra/system-b.log- at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-b.log- at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-b.log- at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
/var/log/cassandra/system-b.log-WARN [CompactionExecutor:1268] 2016-07-08 07:35:24,585 BigTableWriter.java:184 - Writing large partition system/hints:c42d329c-68d5-4d1b-b886-90aa752fb288 (635270451 bytes)
/var/log/cassandra/system-b.log-WARN [HintedHandoffManager:1] 2016-07-08 07:35:28,560 HintedHandoffMetrics.java:80 - /10.192.32.125 has 2511 dropped hints, because node is down past configured hint window.
/var/log/cassandra/system-b.log-WARN [CompactionExecutor:1277] 2016-07-08 07:45:42,224 BigTableWriter.java:184 - Writing large partition system/hints:c42d329c-68d5-4d1b-b886-90aa752fb288 (635270451 bytes)
--
/var/log/cassandra/system-b.log:ERROR [STREAM-OUT-/10.64.48.131] 2016-07-08 08:19:52,364 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
/var/log/cassandra/system-b.log-java.lang.AssertionError: Memory was freed
/var/log/cassandra/system-b.log- at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-b.log- at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-b.log- at org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-b.log- at org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:125) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-b.log- at org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:565) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-b.log- at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-b.log- at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-b.log- at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-b.log- at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:370) ~[apache-cassandra-2.2.6.jar:2.2.6]
--
/var/log/cassandra/system-c.log:ERROR [STREAM-IN-/10.64.48.131] 2016-07-08 05:00:55,586 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
/var/log/cassandra/system-c.log-java.net.SocketTimeoutException: null
/var/log/cassandra/system-c.log- at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) ~[na:1.8.0_91]
/var/log/cassandra/system-c.log- at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_91]
/var/log/cassandra/system-c.log- at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_91]
/var/log/cassandra/system-c.log- at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-c.log- at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-c.log- at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
/var/log/cassandra/system-c.log-WARN [CompactionExecutor:1070] 2016-07-08 05:06:02,662 BigTableWriter.java:184 - Writing large partition system/hints:c42d329c-68d5-4d1b-b886-90aa752fb288 (495145104 bytes)
/var/log/cassandra/system-c.log-WARN [HintedHandoffManager:1] 2016-07-08 05:06:04,403 HintedHandoffMetrics.java:80 - /10.192.32.125 has 1346 dropped hints, because node is down past configured hint window.
/var/log/cassandra/system-c.log-INFO [Service Thread] 2016-07-08 05:15:30,610 GCInspector.java:284 - G1 Young Generation GC in 260ms. G1 Eden Space: 7365197824 -> 0; G1 Old Gen: 3683555312 -> 3690173440; G1 Survivor Space: 268435456 -> 260046848;
--
/var/log/cassandra/system-c.log:ERROR [STREAM-OUT-/10.64.48.131] 2016-07-08 07:15:52,729 StreamSession.java:524 - [Stream #fe6ff550-4488-11e6-85be-e77a66119977] Streaming error occurred
/var/log/cassandra/system-c.log-java.lang.AssertionError: Memory was freed
/var/log/cassandra/system-c.log- at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:338) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-c.log- at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-c.log- at org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-c.log- at org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:125) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-c.log- at org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:565) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-c.log- at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-c.log- at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-c.log- at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.6.jar:2.2.6]
/var/log/cassandra/system-c.log- at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:370) ~[apache-cassandra-2.2.6.jar:2.2.6]
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits