Page MenuHomePhabricator
Create Task
Maniphest T164773

Error replicating wikidata blazegraph setup
Closed, InvalidPublic
Actions

Description

Importing full Wikidata dataset after running munger.sh using

./loadRestAPI.sh -n wdq -d `pwd`/data/split

57 gz files (out of ~270) have been marked as "good". The rest were not renamed. The server has 128GB RAM, 12 cores, and a 2TB+ SSD drive

ERROR: Haltable.java:469: com.bigdata.bop.join.PipelineJoin$JoinTask{ joinOp=com.bigdata.bop.join.PipelineJoin[2]()[ BOp.bopId=2, JoinAnnotations.constraints=null, AST2BOpBase.simp$
eJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[1](x=null, a=null, b=null)[ IPredicate.relationName=[wdq.spo], IPredicat$
.timestamp=1494227742441, BOp.bopId=1, AST2BOpBase.estimatedCardinality=351079004, AST2BOpBase.originalIndex=SPO, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL]]]} : isFirstCause=t$
ue : com.bigdata.rwstore.sector.MemoryManagerClosedException
com.bigdata.rwstore.sector.MemoryManagerClosedException
        at com.bigdata.rwstore.sector.MemoryManager.assertOpen(MemoryManager.java:110)
        at com.bigdata.rwstore.sector.MemoryManager.allocate(MemoryManager.java:671)
        at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:195)
        at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:169)
        at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:159)
        at com.bigdata.rwstore.sector.AllocationContext.alloc(AllocationContext.java:359)
        at com.bigdata.rwstore.PSOutputStream.save(PSOutputStream.java:335)
        at com.bigdata.rwstore.PSOutputStream.getAddr(PSOutputStream.java:416)
        at com.bigdata.bop.solutions.SolutionSetStream.put(SolutionSetStream.java:297)
        at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:213)
        at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:147)
        at com.bigdata.bop.engine.StandaloneChunkHandler.handleChunk(StandaloneChunkHandler.java:90)
        at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1699)
        at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.addReorderAllowed(ChunkedRunningQuery.java:1628)
        at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1569)
        at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1453)
        at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:59)
        at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:14)
        at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.overflow(AbstractUnsynchronizedArrayBuffer.java:287)
        at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add2(AbstractUnsynchronizedArrayBuffer.java:215)
        at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add(AbstractUnsynchronizedArrayBuffer.java:173)
        at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.handleJoin2(PipelineJoin.java:1868)
        at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.call(PipelineJoin.java:1684)
        at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.runOneTask(PipelineJoin.java:1086)
        at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.call(PipelineJoin.java:995)
        at com.bigdata.bop.join.PipelineJoin$JoinTask.consumeSource(PipelineJoin.java:728)
        at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:623)
        at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:382)
        at java.util.concurrent.FutureTask.run(java.base@9-internal/FutureTask.java:266)
        at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
        at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1346)
        at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:926)
        at java.util.concurrent.Executors$RunnableAdapter.call(java.base@9-internal/Executors.java:514)
        at java.util.concurrent.FutureTask.run(java.base@9-internal/FutureTask.java:266)
        at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
        at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:821)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-internal/ThreadPoolExecutor.java:1158)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-internal/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-internal/Thread.java:804)

WARN : Haltable.java:474: com.bigdata.bop.join.PipelineJoin$JoinTask{ joinOp=com.bigdata.bop.join.PipelineJoin[2]()[ BOp.bopId=2, JoinAnnotations.constraints=null, AST2BOpBa[2/1153$
eJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[1](x=null, a=null, b=null)[ IPredicate.relationName=[wdq.spo], IPredicate
.timestamp=1494227742441, BOp.bopId=1, AST2BOpBase.estimatedCardinality=351079004, AST2BOpBase.originalIndex=SPO, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL]]]} : isFirstCause=fa
lse : java.lang.RuntimeException: java.lang.RuntimeException: com.bigdata.rwstore.sector.MemoryManagerClosedException
java.lang.RuntimeException: java.lang.RuntimeException: com.bigdata.rwstore.sector.MemoryManagerClosedException
        at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.call(PipelineJoin.java:1027)
        at com.bigdata.bop.join.PipelineJoin$JoinTask.consumeSource(PipelineJoin.java:728)
        at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:623)
        at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:382)
        at java.util.concurrent.FutureTask.run(java.base@9-internal/FutureTask.java:266)
        at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
        at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1346)
        at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:926)
        at java.util.concurrent.Executors$RunnableAdapter.call(java.base@9-internal/Executors.java:514)
        at java.util.concurrent.FutureTask.run(java.base@9-internal/FutureTask.java:266)
        at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
        at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:821)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-internal/ThreadPoolExecutor.java:1158)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-internal/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-internal/Thread.java:804)
Caused by: java.lang.RuntimeException: com.bigdata.rwstore.sector.MemoryManagerClosedException
        at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.handleJoin2(PipelineJoin.java:1961)
        at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.call(PipelineJoin.java:1684)
        at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.runOneTask(PipelineJoin.java:1086)
        at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.call(PipelineJoin.java:995)
        ... 14 more
Caused by: com.bigdata.rwstore.sector.MemoryManagerClosedException
        at com.bigdata.rwstore.sector.MemoryManager.assertOpen(MemoryManager.java:110)
        at com.bigdata.rwstore.sector.MemoryManager.allocate(MemoryManager.java:671)
        at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:195)
        at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:169)
        at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:159)
        at com.bigdata.rwstore.sector.AllocationContext.alloc(AllocationContext.java:359)
        at com.bigdata.rwstore.PSOutputStream.save(PSOutputStream.java:335)
        at com.bigdata.rwstore.PSOutputStream.getAddr(PSOutputStream.java:416)
        at com.bigdata.bop.solutions.SolutionSetStream.put(SolutionSetStream.java:297)
        at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:213)
        at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:147)
        at com.bigdata.bop.engine.StandaloneChunkHandler.handleChunk(StandaloneChunkHandler.java:90)
        at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1699)
        at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.addReorderAllowed(ChunkedRunningQuery.java:1628)
        at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1569)
        at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1453)
        at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:59)
        at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:14)
        at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.overflow(AbstractUnsynchronizedArrayBuffer.java:287)
        at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add2(AbstractUnsynchronizedArrayBuffer.java:215)
        at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add(AbstractUnsynchronizedArrayBuffer.java:173)
        at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.handleJoin2(PipelineJoin.java:1868)
        ... 17 more

Related Objects

Event Timeline

Yurik created this task.May 8 2017, 6:02 PM
Restricted Application added projects: Wikidata, Discovery-ARCHIVED. · View Herald TranscriptMay 8 2017, 6:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.May 9 2017, 10:07 AM
• Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
• Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
• Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
• Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
• Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
• Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
• Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
• Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
• Qse24h closed this task as a duplicate of T164723: New git repository: <repo name>.
Aklapper reopened this task as Open.May 9 2017, 10:19 AM
Sjoerddebruin moved this task from Incoming to Blazegraph on the Wikidata-Query-Service board.May 23 2017, 11:33 AM
Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.Jun 11 2017, 6:20 PM
Smalyshev added a comment.Jun 23 2017, 7:34 PM

Didn't see it happen on my side, is it reproducible?

Smalyshev closed this task as Invalid.Jul 7 2017, 11:39 PM

Since I don't have any way to reproduce it and proceed with this, I am closing this for now. If any new info regarding when/how it happens appears, please reopen.

Gehel reopened this task as Open.Aug 31 2017, 6:45 PM
Gehel subscribed.

Now that we have WDQS logs sent to logstash, I see a similar error. It sounds like the memory manager is closed while some code is still using it. Looking around in the code, I find AbstractRunningQuery.releaseNativeMemoryForQuery() where the doc is slightly scary. Beyond that, I have no idea what is going on, and if this is really an issue, or just a case of over verbose logging.

Smalyshev added a comment.Aug 31 2017, 6:51 PM

MemoryManagerClosedException by itself is not a big issue as far as I can see, it may be query timeout or something like that. That shouldn't happen on data loading though (that one should not time out).

Yurik added a comment.Aug 31 2017, 8:14 PM

I'm re-running the import with the default settings (mmx=16gb), almost 700 gz files (127GB). This time, all of the OSM data uses integer storage instead of the original strings ("osmnode:123" treats 123 as a prefix + an int). Will know the results in 2 days.

Yurik added a comment.EditedSep 2 2017, 5:26 PM

I just finished full re-import with -Xmx=16GB , and it worked fine. I am not sure what has caused the original issue. For future reference, here are the import stats (importing both Wikidata and OSM data from the same dir - 33GB total, 683 files of about the same size. The final wikidata.jnl size was 536GB. Total time was about 36 hours. First 329 files were OSM, second 354 - wikidata (note how the graph starts jumping, possibly due to per-file statement distribution in WD. OSM data is much more uniform in terms of statement per osm object)

image.png (660×1 px, 174 KB)

raw data
statements	time	rate	sets
107,179,984	640	167,220	6
189,557,036	1,260	150,378	11
255,343,483	1,884	135,523	15
323,230,658	2,533	127,587	19
389,362,380	3,231	120,504	23
454,181,776	3,971	114,351	27
519,770,290	4,654	111,667	31
585,537,711	5,358	109,275	35
669,713,329	6,025	111,138	40
753,727,112	6,704	112,415	45
835,803,507	7,485	111,656	50
901,307,143	8,218	109,668	54
966,655,528	8,963	107,845	58
1,015,648,545	9,570	106,123	61
1,080,889,536	10,310	104,829	65
1,147,921,605	11,098	103,427	69
1,218,376,248	11,800	103,245	73
1,288,894,078	12,464	103,401	77
1,340,627,982	13,252	101,158	80
1,374,208,625	13,948	98,521	82
1,408,174,205	14,568	96,659	84
1,459,277,475	15,389	94,824	87
1,510,100,090	16,207	93,174	90
1,560,517,757	16,986	91,866	93
1,627,435,035	17,713	91,877	97
1,693,942,260	18,401	92,053	101
1,760,534,757	19,151	91,927	105
1,827,353,450	19,934	91,669	109
1,877,349,669	20,571	91,260	112
1,927,029,732	21,190	90,937	115
1,976,839,801	21,803	90,665	118
2,043,621,669	22,602	90,417	122
2,093,614,307	23,222	90,153	125
2,143,530,955	23,884	89,744	128
2,193,300,587	24,504	89,506	131
2,259,768,850	25,281	89,382	135
2,309,402,067	25,896	89,178	138
2,359,045,834	26,524	88,937	141
2,408,813,042	27,193	88,582	144
2,458,534,137	27,866	88,224	147
2,508,173,248	28,587	87,736	150
2,557,695,348	29,236	87,481	153
2,607,093,365	29,879	87,254	156
2,672,980,080	30,659	87,181	160
2,722,486,059	31,293	86,997	163
2,771,962,432	31,913	86,859	166
2,837,660,998	32,616	87,001	170
2,887,208,486	33,270	86,779	173
2,936,799,169	33,925	86,567	176
2,986,273,795	34,593	86,325	179
3,035,728,568	35,243	86,136	182
3,085,184,315	35,859	86,035	185
3,151,468,039	36,608	86,086	189
3,201,011,063	37,213	86,017	192
3,267,535,396	37,935	86,134	196
3,334,686,516	38,576	86,443	200
3,401,886,904	39,240	86,692	204
3,468,645,120	39,954	86,814	208
3,534,648,676	40,730	86,781	212
3,600,523,802	41,492	86,775	216
3,666,138,573	42,282	86,706	220
3,731,829,153	43,077	86,629	224
3,797,535,576	43,829	86,643	228
3,863,523,948	44,582	86,659	232
3,928,985,444	45,258	86,812	236
3,994,209,028	45,921	86,979	240
4,059,507,043	46,604	87,104	244
4,125,187,426	47,359	87,104	248
4,190,652,309	48,108	87,108	252
4,256,123,933	48,782	87,246	256
4,321,507,795	49,466	87,362	260
4,386,987,648	50,120	87,528	264
4,452,442,948	50,739	87,750	268
4,517,769,637	51,363	87,956	272
4,583,567,763	52,006	88,133	276
4,649,357,008	52,705	88,213	280
4,715,013,078	53,387	88,316	284
4,780,538,748	54,036	88,468	288
4,845,958,188	54,701	88,588	292
4,912,126,601	55,324	88,787	296
4,977,535,397	55,951	88,961	300
5,042,533,501	56,599	89,091	304
5,107,487,478	57,217	89,264	308
5,172,445,286	57,824	89,450	312
5,253,524,269	58,545	89,734	317
5,305,784,367	59,194	89,633	320
5,343,676,632	60,030	89,016	322
5,381,274,598	60,966	88,265	324
5,418,420,058	61,807	87,665	326
5,468,584,289	62,692	87,228	329
5,518,098,321	63,442	86,977	333
5,564,009,469	64,131	86,759	339
5,619,110,296	64,763	86,763	345
5,673,559,079	65,380	86,777	354
5,715,132,031	65,986	86,610	360
5,756,082,773	66,616	86,406	368
5,812,130,630	67,248	86,428	376
5,858,211,979	67,888	86,291	382
5,917,602,276	68,563	86,308	390
5,972,627,425	69,255	86,241	396
6,033,689,471	70,387	85,720	400
6,061,126,737	71,089	85,260	402
6,090,631,401	71,841	84,778	405
6,120,734,012	72,539	84,377	409
6,153,355,223	73,182	84,082	413
6,207,067,286	73,873	84,023	420
6,253,755,885	74,611	83,817	427
6,284,713,321	75,280	83,484	432
6,319,282,922	76,000	83,147	439
6,355,867,878	76,609	82,964	444
6,399,714,387	77,288	82,803	450
6,436,640,839	78,036	82,482	455
6,481,234,379	78,650	82,405	461
6,526,759,664	79,391	82,209	466
6,579,995,114	80,501	81,737	470
6,599,378,396	81,211	81,261	471
6,625,131,210	82,089	80,705	473
6,645,098,333	82,772	80,281	475
6,669,624,733	83,502	79,873	478
6,699,812,271	84,314	79,462	482
6,738,122,743	84,990	79,281	486
6,774,046,102	85,662	79,078	492
6,808,866,336	86,267	78,927	497
6,833,736,316	86,915	78,625	501
6,860,334,945	87,592	78,320	506
6,887,971,112	88,315	77,992	511
6,919,244,750	88,952	77,786	515
6,953,331,447	89,571	77,628	520
6,978,517,812	90,206	77,361	523
7,007,112,877	90,843	77,134	527
7,044,596,712	91,468	77,016	532
7,080,015,049	92,212	76,779	536
7,124,088,113	93,041	76,569	540
7,145,384,434	94,007	76,008	541
7,163,582,483	94,892	75,491	542
7,177,490,061	95,537	75,127	543
7,198,819,202	96,483	74,611	545
7,217,160,866	97,276	74,191	547
7,239,559,213	98,115	73,786	550
7,263,224,958	98,910	73,432	553
7,291,006,386	99,618	73,189	556
7,327,155,852	100,322	73,035	561
7,348,363,609	100,952	72,790	565
7,379,345,399	101,720	72,545	569
7,396,569,140	102,359	72,260	572
7,417,391,783	103,081	71,956	576
7,438,740,940	103,845	71,632	580
7,467,690,245	104,563	71,417	584
7,488,657,687	105,164	71,208	587
7,518,220,972	105,879	71,007	591
7,540,436,150	106,623	70,719	594
7,562,375,588	107,282	70,490	597
7,598,002,915	108,023	70,336	602
7,625,559,187	108,697	70,154	605
7,653,098,007	109,323	70,004	608
7,687,155,971	110,042	69,856	611
7,710,473,517	111,457	69,178	612
7,727,861,201	112,465	68,713	613
7,740,780,981	113,152	68,410	614
7,761,818,411	114,232	67,947	616
7,779,652,176	115,128	67,573	618
7,794,807,466	115,763	67,333	620
7,808,565,243	116,378	67,096	622
7,826,326,493	117,025	66,876	624
7,842,746,787	117,642	66,666	626
7,877,394,670	118,251	66,615	630
7,894,298,591	118,872	66,409	633
7,918,025,733	119,505	66,256	637
7,941,220,171	120,317	66,002	640
7,958,978,075	121,044	65,752	643
7,974,745,220	121,677	65,539	646
7,994,758,935	122,487	65,270	650
8,013,719,634	123,209	65,041	653
8,037,431,063	123,928	64,855	656
8,063,684,080	124,612	64,709	660
8,088,059,972	125,510	64,441	663
8,110,100,469	126,283	64,221	666
8,130,658,259	126,996	64,022	669
8,160,908,753	127,709	63,901	673
8,188,651,027	128,478	63,735	676
8,215,993,683	129,194	63,594	679
8,250,187,727	129,996	63,464	682
Smalyshev closed this task as Resolved.Sep 5 2017, 7:53 PM
Smalyshev claimed this task.

Looks like we can close this one, the original error seems to not reproduce after all.

Yurik added a comment.Sep 17 2017, 8:04 AM

@Smalyshev & @Gehel I'm not sure if this is the same error or different. Today, the service froze in a peculiar way: all queries would time out (both from clients and update ones), and Blazegraph wouldn't quit with Ctrl+C. HTOP shows a single blazegraph process using 100% of a single CPU, but about once in a few minutes, almost all CPUs would jump to a 100% for about 5-10 seconds, and then go back to 0 except for a single process. The last errors in the log, might be unrelated:

at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:159) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.alloc(AllocationContext.java:359) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.PSOutputStream.save(PSOutputStream.java:335) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.PSOutputStream.getAddr(PSOutputStream.java:416) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.solutions.SolutionSetStream.put(SolutionSetStream.java:297) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:213) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:147) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.StandaloneChunkHandler.handleChunk(StandaloneChunkHandler.java:90) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1699) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.addReorderAllowed(ChunkedRunningQuery.java:1628) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1569) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1453) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:59) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:14) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.overflow(AbstractUnsynchronizedArrayBuffer.java:287) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add2(AbstractUnsynchronizedArrayBuffer.java:215) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add(AbstractUnsynchronizedArrayBuffer.java:173) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.handleJoin2(PipelineJoin.java:1868) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
... 17 common frames omitted

07:53:53.567 [com.bigdata.journal.Journal.executorService23435] ERROR com.bigdata.util.concurrent.Haltable IP: UA: - com.bigdata.bop.join.PipelineJoin$JoinTask{ joinOp=com.bigdata.bop.join.PipelineJoin[2]()[ BOp.bopId=2, JoinAnnotations.constraints=null, AST2BOpBase.simpleJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[1](osmId=null, Vocab(56)[https://wiki.openstreetmap.org/wiki/Key:place], place=null)[ IPredicate.relationName=[wdq.spo], IPredicate.timestamp=1505627633562, BOp.bopId=1, AST2BOpBase.estimatedCardinality=4549666, AST2BOpBase.originalIndex=POS, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL]]]} : isFirstCause=true : com.bigdata.rwstore.sector.MemoryManagerClosedException
com.bigdata.rwstore.sector.MemoryManagerClosedException: null
at com.bigdata.rwstore.sector.MemoryManager.assertOpen(MemoryManager.java:110) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.MemoryManager.allocate(MemoryManager.java:671) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:195) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:169) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:159) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.alloc(AllocationContext.java:359) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.PSOutputStream.save(PSOutputStream.java:335) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.PSOutputStream.getAddr(PSOutputStream.java:416) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.solutions.SolutionSetStream.put(SolutionSetStream.java:297) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:213) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:147) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.StandaloneChunkHandler.handleChunk(StandaloneChunkHandler.java:90) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1699) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.addReorderAllowed(ChunkedRunningQuery.java:1628) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1569) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1453) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:59) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:14) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.overflow(AbstractUnsynchronizedArrayBuffer.java:287) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add2(AbstractUnsynchronizedArrayBuffer.java:215) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add(AbstractUnsynchronizedArrayBuffer.java:173) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.handleJoin2(PipelineJoin.java:1868) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.call(PipelineJoin.java:1684) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.runOneTask(PipelineJoin.java:1086) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.call(PipelineJoin.java:995) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask.consumeSource(PipelineJoin.java:728) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:623) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:382) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1346) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:926) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:821) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

07:53:53.568 [com.bigdata.journal.Journal.executorService23435] WARN  com.bigdata.util.concurrent.Haltable IP: UA: - com.bigdata.bop.join.PipelineJoin$JoinTask{ joinOp=com.bigdata.bop.join.PipelineJoin[2]()[ BOp.bopId=2, JoinAnnotations.constraints=null, AST2BOpBase.simpleJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[1](osmId=null, Vocab(56)[https://wiki.openstreetmap.org/wiki/Key:place], place=null)[ IPredicate.relationName=[wdq.spo], IPredicate.timestamp=1505627633562, BOp.bopId=1, AST2BOpBase.estimatedCardinality=4549666, AST2BOpBase.originalIndex=POS, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL]]]} : isFirstCause=false : java.lang.RuntimeException: com.bigdata.rwstore.sector.MemoryManagerClosedException
java.lang.RuntimeException: com.bigdata.rwstore.sector.MemoryManagerClosedException
at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.handleJoin2(PipelineJoin.java:1961) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.call(PipelineJoin.java:1684) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.runOneTask(PipelineJoin.java:1086) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.call(PipelineJoin.java:995) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask.consumeSource(PipelineJoin.java:728) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:623) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:382) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1346) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:926) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:821) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: com.bigdata.rwstore.sector.MemoryManagerClosedException: null
at com.bigdata.rwstore.sector.MemoryManager.assertOpen(MemoryManager.java:110) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.MemoryManager.allocate(MemoryManager.java:671) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:195) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:169) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:159) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.alloc(AllocationContext.java:359) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.PSOutputStream.save(PSOutputStream.java:335) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.PSOutputStream.getAddr(PSOutputStream.java:416) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.solutions.SolutionSetStream.put(SolutionSetStream.java:297) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:213) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:147) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.StandaloneChunkHandler.handleChunk(StandaloneChunkHandler.java:90) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1699) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.addReorderAllowed(ChunkedRunningQuery.java:1628) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1569) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1453) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:59) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:14) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.overflow(AbstractUnsynchronizedArrayBuffer.java:287) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add2(AbstractUnsynchronizedArrayBuffer.java:215) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add(AbstractUnsynchronizedArrayBuffer.java:173) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.handleJoin2(PipelineJoin.java:1868) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
... 17 common frames omitted

07:53:53.569 [com.bigdata.journal.Journal.executorService23435] WARN  com.bigdata.util.concurrent.Haltable IP: UA: - com.bigdata.bop.join.PipelineJoin$JoinTask{ joinOp=com.bigdata.bop.join.PipelineJoin[2]()[ BOp.bopId=2, JoinAnnotations.constraints=null, AST2BOpBase.simpleJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[1](osmId=null, Vocab(56)[https://wiki.openstreetmap.org/wiki/Key:place], place=null)[ IPredicate.relationName=[wdq.spo], IPredicate.timestamp=1505627633562, BOp.bopId=1, AST2BOpBase.estimatedCardinality=4549666, AST2BOpBase.originalIndex=POS, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL]]]} : isFirstCause=false : java.lang.RuntimeException: java.lang.RuntimeException: com.bigdata.rwstore.sector.MemoryManagerClosedException
java.lang.RuntimeException: java.lang.RuntimeException: com.bigdata.rwstore.sector.MemoryManagerClosedException
at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.call(PipelineJoin.java:1027) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask.consumeSource(PipelineJoin.java:728) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:623) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:382) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1346) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:926) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:821) [bigdata-core-2.1.5-SNAPSHOT.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: java.lang.RuntimeException: com.bigdata.rwstore.sector.MemoryManagerClosedException
at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.handleJoin2(PipelineJoin.java:1961) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.call(PipelineJoin.java:1684) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.runOneTask(PipelineJoin.java:1086) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.call(PipelineJoin.java:995) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
... 14 common frames omitted
Caused by: com.bigdata.rwstore.sector.MemoryManagerClosedException: null
at com.bigdata.rwstore.sector.MemoryManager.assertOpen(MemoryManager.java:110) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.MemoryManager.allocate(MemoryManager.java:671) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:195) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:169) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.allocate(AllocationContext.java:159) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.sector.AllocationContext.alloc(AllocationContext.java:359) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.PSOutputStream.save(PSOutputStream.java:335) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.rwstore.PSOutputStream.getAddr(PSOutputStream.java:416) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.solutions.SolutionSetStream.put(SolutionSetStream.java:297) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:213) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.LocalNativeChunkMessage.<init>(LocalNativeChunkMessage.java:147) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.StandaloneChunkHandler.handleChunk(StandaloneChunkHandler.java:90) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1699) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.addReorderAllowed(ChunkedRunningQuery.java:1628) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1569) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1453) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:59) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:14) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.overflow(AbstractUnsynchronizedArrayBuffer.java:287) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add2(AbstractUnsynchronizedArrayBuffer.java:215) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add(AbstractUnsynchronizedArrayBuffer.java:173) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.handleJoin2(PipelineJoin.java:1868) ~[bigdata-core-2.1.5-SNAPSHOT.jar:na]
... 17 common frames omitted
Smalyshev added a comment.Sep 17 2017, 8:08 AM

Not sure whether it has some relation, hard to diagnose from this. Did you see the GC logs? What the status page for Blazegraph showed?

Yurik added a comment.Sep 17 2017, 8:34 AM

@Smalyshev, I force-killed it, and it dumped this (I couldn't copy all of it - just some parts):

at com.bigdata.rdf.sail.webapp.RESTServlet.doGet(RESTServlet.java:240)
  at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doGet(MultiTenancyServlet.java:271)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
  at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:244)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
  at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:51)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
  at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:42)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
  at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
  at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
  at org.eclipse.jetty.server.Server.handle(Server.java:497)
  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
  at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
  at java.lang.Thread.run(Thread.java:748)

"qtp931919113-27686" #27686 prio=5 os_prio=0 tid=0x00007fbb5c00c000 nid=0x2c11 waiting on condition [0x00007fba671f3000]
java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <0x00000007febc0168> (a java.util.concurrent.FutureTask)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
  at java.util.concurrent.FutureTask.get(FutureTask.java:191)
  at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:286)
  at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:653)
  at com.bigdata.rdf.sail.webapp.QueryServlet.doGet(QueryServlet.java:288)
  at com.bigdata.rdf.sail.webapp.RESTServlet.doGet(RESTServlet.java:240)
  at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doGet(MultiTenancyServlet.java:271)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
  at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:244)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
  at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:51)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
  at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:42)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
  at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
  at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
  at org.eclipse.jetty.server.Server.handle(Server.java:497)
  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
  at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
  at java.lang.Thread.run(Thread.java:748)

"com.bigdata.journal.Journal.executorService24128" #27684 daemon prio=5 os_prio=0 tid=0x00007fbb5458a000 nid=0x2c0c waiting on condition [0x00007fbab43c9000]
java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <0x0000000081c449c0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
  at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
  at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
  at com.bigdata.concurrent.NamedLock.acquireLock(NamedLock.java:106)
  at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:327)
  at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2068)
  at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:5236)
  at com.bigdata.rdf.sail.BigdataSail._getReadOnlyConnection(BigdataSail.java:1540)
  at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1503)
  at com.bigdata.rdf.sail.BigdataSailRepository.getReadOnlyConnection(BigdataSailRepository.java:139)
  at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:247)
  at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:221)
  at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:721)
  at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:670)
  at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

"com.bigdata.journal.Journal.executorService24129" #27685 daemon prio=5 os_prio=0 tid=0x00007fba8800a000 nid=0x2c0b waiting on condition [0x00007fba57af9000]
java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <0x0000000081c449c0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
  at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
  at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
  at com.bigdata.concurrent.NamedLock.acquireLock(NamedLock.java:106)
  at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:327)
  at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2068)
  at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:5236)
  at com.bigdata.rdf.sail.BigdataSail._getReadOnlyConnection(BigdataSail.java:1540)
  at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1503)
  at com.bigdata.rdf.sail.BigdataSailRepository.getReadOnlyConnection(BigdataSailRepository.java:139)
  at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:247)
  at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:221)
  at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:721)
  at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:670)
  at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

"com.bigdata.journal.Journal.executorService24127" #27683 daemon prio=5 os_prio=0 tid=0x00007fb97404c000 nid=0x2c0a waiting on condition [0x00007fbab6c06000]
java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <0x0000000081c449c0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
  at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
  at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
  at com.bigdata.concurrent.NamedLock.acquireLock(NamedLock.java:106)
  at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:327)
  at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2068)
  at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:5236)
  at com.bigdata.rdf.sail.BigdataSail._getReadOnlyConnection(BigdataSail.java:1540)
  at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1503)
  at com.bigdata.rdf.sail.BigdataSailRepository.getReadOnlyConnection(BigdataSailRepository.java:139)
  at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:247)
  at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:221)
  at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:721)
  at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:670)
  at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

"qtp931919113-27682" #27682 prio=5 os_prio=0 tid=0x00007fb8f5708000 nid=0x2c06 waiting on condition [0x00007fba648ca000]
java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <0x00000007feb93268> (a java.util.concurrent.FutureTask)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
  at java.util.concurrent.FutureTask.get(FutureTask.java:191)


at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"Scheduler-1267032364" #36 prio=5 os_prio=0 tid=0x00007fba61cec800 nid=0x46d2 waiting for monitor entry [0x00007fbab4bcd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked <0x00000007ff185f88> (a java.util.concurrent.TimeoutException)
at java.lang.Throwable.<init>(Throwable.java:265)
at java.lang.Exception.<init>(Exception.java:66)
at java.util.concurrent.TimeoutException.<init>(TimeoutException.java:65)
at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:156)
at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"com.bigdata.journal.ConcurrencyManager.sampleService1" #33 daemon prio=5 os_prio=0 tid=0x00007fb9c6e30000 nid=0x46cb waiting for monitor entry [0x00007fbab52d0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.HashMap$KeySet.iterator(HashMap.java:916)
at java.util.HashSet.iterator(HashSet.java:172)
at java.util.concurrent.ThreadPoolExecutor.getActiveCount(ThreadPoolExecutor.java:1836)
at com.bigdata.util.concurrent.ThreadPoolExecutorStatisticsTask.run(ThreadPoolExecutorStatisticsTask.java:322)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"com.bigdata.journal.WriteExecutorService$MyLockManager1" #32 daemon prio=5 os_prio=0 tid=0x00007fb9c6e1f000 nid=0x46ca waiting on condition [0x00007fbab53d1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000816c7370> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign$AcceptTask.awaitStateChange(NonBlockingLockManagerWithNewDesign.java:2161)
at com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign$AcceptTask.run(NonBlockingLockManagerWithNewDesign.java:2055)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"com.bigdata.rwstore.RWStore$11" #31 daemon prio=5 os_prio=0 tid=0x00007fb9f7f00800 nid=0x46c9 waiting on condition [0x00007fbab56d2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000816c7608> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at com.bigdata.io.writecache.WriteCacheService$WriteTask.awaitDirtyBuffer(WriteCacheService.java:1317)
at com.bigdata.io.writecache.WriteCacheService$WriteTask.doRun(WriteCacheService.java:1004)
at com.bigdata.io.writecache.WriteCacheService$WriteTask.call(WriteCacheService.java:923)
at com.bigdata.io.writecache.WriteCacheService$WriteTask.call(WriteCacheService.java:895)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"StaleReferenceCleaner1" #30 daemon prio=5 os_prio=0 tid=0x00007fbb973fe000 nid=0x46c0 waiting for monitor entry [0x00007fbab5be6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.concurrent.ConcurrentLinkedQueue.iterator(ConcurrentLinkedQueue.java:669)
at com.bigdata.cache.SynchronizedHardReferenceQueueWithTimeout$Cleaner.run(SynchronizedHardReferenceQueueWithTimeout.java:492)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"org.eclipse.jetty.server.session.HashSessionManager@7748410aTimer" #29 daemon prio=5 os_prio=0 tid=0x00007fbb9739b000 nid=0x46bf waiting on condition [0x00007fbab5ce7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000816c7a90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"qtp931919113-27-selector-ServerConnectorManager@479b29e7/0" #27 prio=5 os_prio=0 tid=0x00007fba6c001000 nid=0x46bd runnable [0x00007fbab6701000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap$KeySet.iterator(HashMap.java:916)
at java.util.HashSet.iterator(HashSet.java:172)
at sun.nio.ch.Util$3.iterator(Util.java:324)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:603)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
at java.lang.Thread.run(Thread.java:748)

"qtp931919113-23" #23 prio=5 os_prio=0 tid=0x00007fba7c001000 nid=0x46b9 waiting for monitor entry [0x00007fbab6b05000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:481)
at java.lang.ThreadLocal$ThreadLocalMap.access$100(ThreadLocal.java:298)
at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:184)
at java.lang.ThreadLocal.get(ThreadLocal.java:170)
at org.eclipse.jetty.server.HttpConnection.setCurrentConnection(HttpConnection.java:82)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:200)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
at java.lang.Thread.run(Thread.java:748)

"qtp931919113-20-selector-ServerConnectorManager@479b29e7/3" #20 prio=5 os_prio=0 tid=0x00007fbb96a9a800 nid=0x46b7 runnable [0x00007fbab70e7000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000962d2dd8> (a sun.nio.ch.Util$3)
- locked <0x00000000962d2de8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000962d2d90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
at java.lang.Thread.run(Thread.java:748)

"qtp931919113-18-selector-ServerConnectorManager@479b29e7/2" #18 prio=5 os_prio=0 tid=0x00007fbb96a97000 nid=0x46b5 runnable [0x00007fbab72e9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000962d2ee8> (a sun.nio.ch.Util$3)
        - locked <0x00000000962d2ef8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000962d2ea0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
        at java.lang.Thread.run(Thread.java:748)

"qtp931919113-17-selector-ServerConnectorManager@479b29e7/1" #17 prio=5 os_prio=0 tid=0x00007fbb96a95800 nid=0x46b4 runnable [0x00007fbab73ea000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000962d8f38> (a sun.nio.ch.Util$3)
        - locked <0x00000000962d8f28> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000962d8d50> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
        at java.lang.Thread.run(Thread.java:748)

"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x00007fbb9692c000 nid=0x46ad runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007fbb96926800 nid=0x46ac waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fbb96922800 nid=0x46ab runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fbb96920800 nid=0x46aa waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fbb9691e000 nid=0x46a9 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fbb9691c000 nid=0x46a8 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007fbb9691a000 nid=0x46a7 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fbb968f2800 nid=0x46a6 in Object.wait() [0x00007fbacc8f6000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x0000000083b39fe0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fbb968ee000 nid=0x46a5 in Object.wait() [0x00007fbacc9f7000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x0000000083b51190> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007fbb9400c000 nid=0x468a in Object.wait() [0x00007fbb9d22f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000009c0b8b30> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:502)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:373)
        - locked <0x000000009c0b8b30> (a java.lang.Object)
        at org.eclipse.jetty.server.Server.join(Server.java:560)
        at org.eclipse.jetty.runner.Runner.run(Runner.java:510)
        at org.eclipse.jetty.runner.Runner.main(Runner.java:557)

"VM Thread" os_prio=0 tid=0x00007fbb968e6800 nid=0x46a4 runnable 

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fbb94021000 nid=0x468b runnable 

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fbb94023000 nid=0x468c runnable 

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fbb94024800 nid=0x468d runnable 

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fbb94026800 nid=0x468e runnable 

"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007fbb94028000 nid=0x468f runnable 

"Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007fbb9402a000 nid=0x4690 runnable 

"Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x00007fbb9402b800 nid=0x4691 runnable 

"Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x00007fbb9402d800 nid=0x4692 runnable 

"Gang worker#8 (Parallel GC Threads)" os_prio=0 tid=0x00007fbb9402f000 nid=0x4693 runnable 

"Gang worker#9 (Parallel GC Threads)" os_prio=0 tid=0x00007fbb94031000 nid=0x4694 runnable 

"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007fbb9405d000 nid=0x46a0 runnable 

"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fbb9405f000 nid=0x46a1 runnable 

"Gang worker#1 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fbb94060800 nid=0x46a2 runnable 

"Gang worker#2 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fbb94062800 nid=0x46a3 runnable 

"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007fbb94045800 nid=0x469f runnable 

"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007fbb94043800 nid=0x469e runnable 

"G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007fbb94041800 nid=0x469d runnable 

"G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007fbb94040000 nid=0x469c runnable 

"G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007fbb9403e000 nid=0x469b runnable 

"G1 Concurrent Refinement Thread#5" os_prio=0 tid=0x00007fbb9403c800 nid=0x469a runnable 

"G1 Concurrent Refinement Thread#6" os_prio=0 tid=0x00007fbb9403a800 nid=0x4699 runnable 

"G1 Concurrent Refinement Thread#7" os_prio=0 tid=0x00007fbb94039000 nid=0x4698 runnable 

"G1 Concurrent Refinement Thread#8" os_prio=0 tid=0x00007fbb94037000 nid=0x4697 runnable 

"G1 Concurrent Refinement Thread#9" os_prio=0 tid=0x00007fbb94035800 nid=0x4696 runnable 

"G1 Concurrent Refinement Thread#10" os_prio=0 tid=0x00007fbb94033800 nid=0x4695 runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007fbb9692f000 nid=0x46ae waiting on condition 

JNI global references: 330

Heap
 garbage-first heap   total 31457280K, used 31336542K [0x0000000080000000, 0x0000000080807800, 0x0000000800000000)
  region size 8192K, 1 young (8192K), 0 survivors (0K)
 Metaspace       used 38577K, capacity 38824K, committed 39724K, reserved 1085440K
  class space    used 3950K, capacity 4064K, committed 4172K, reserved 1048576K

Killed
Hjfocs mentioned this in T185576: Improve performance of the search service.Jan 23 2018, 2:52 PM
Hjfocs reopened this task as Open.Jan 23 2018, 3:19 PM
Hjfocs subscribed.

Hitting the same MemoryManagerClosedException with the Wikidata-primary-sources /search service, production deployment (mvn package).
Does not seem to affect the service response, which works fine.

The test deployment (mvn jetty:run) does not hit the exception.

In T164773#3571097, @Gehel wrote:

Now that we have WDQS logs sent to logstash, I see a similar error. It sounds like the memory manager is closed while some code is still using it. Looking around in the code, I find AbstractRunningQuery.releaseNativeMemoryForQuery() where the doc is slightly scary. Beyond that, I have no idea what is going on, and if this is really an issue, or just a case of over verbose logging.

Stumbled upon this thread, didn't help, though:
https://sourceforge.net/p/bigdata/mailman/message/35380438/
Maybe the Blazegraph developers have an answer: could you please loop them in?

Smalyshev triaged this task as Medium priority.Feb 12 2018, 8:02 AM
Smalyshev added a comment.Mar 22 2018, 8:58 PM

Is this still an issue?

Yurik closed this task as Invalid.Mar 22 2018, 9:06 PM

Seem like lowering the upload batch size fixed most of the memory issues. The system still crashes on occasion, but works okayish otherwise. Thx!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits