Page MenuHomePhabricator

Gerrit out of heap
Closed, ResolvedPublicSecurity

Description


Upstream issue: https://bugs.chromium.org/p/gerrit/issues/detail?id=13858


On September 16 2020, our Gerrit primary instance on gerrit1001 went unresponsive and had to be restarted:

 	[production]
09:22 	<jynus> 	restarting gerrit service on gerrit1001, unresponsive

A quick glance at logs show:

[2020-09-16 08:59:26,287] [HTTP-964160] INFO  org.eclipse.jetty.io.ManagedSelector : Caught select() failure, trying to recover: java.lang.OutOfMemoryError: Java heap space
[2020-09-16 08:59:26,288] [HTTP-964151] INFO  org.eclipse.jetty.io.ManagedSelector : Caught select() failure, trying to recover: java.lang.OutOfMemoryError: Java heap space

Heap objects can be seen via JavaMelody: https://gerrit.wikimedia.org/r/monitoring?part=heaphisto and click details in the lower corner. Or using jmap -histo.

Memory usage can be seen at https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=usedMemory

Java memory usage over one year:

Gerrit_3months_usedMemory.png (499×1 px, 21 KB)

1-- Logs begin at Sat 2020-09-05 19:08:09 UTC, end at Wed 2020-09-16 09:35:51 UTC. --
2Sep 16 08:59:29 gerrit1001 java[25679]: Sep 16, 2020 8:59:29 AM com.google.common.util.concurrent.AggregateFuture log
3Sep 16 08:59:29 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
4Sep 16 08:59:29 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
5Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
6Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
7Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
8Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
9Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
10Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
11Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
12Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
13Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
14Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
15Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
16Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
17Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
18Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
19Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
20Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
21Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
22Sep 16 08:59:29 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
23Sep 16 08:59:29 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
24Sep 16 08:59:29 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
25Sep 16 08:59:29 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
26Sep 16 09:00:09 gerrit1001 java[25679]: Sep 16, 2020 9:00:09 AM com.google.common.util.concurrent.AggregateFuture log
27Sep 16 09:00:09 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
28Sep 16 09:00:09 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
29Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
30Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
31Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
32Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
33Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
34Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
35Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
36Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
37Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
38Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
39Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
40Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
41Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
42Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
43Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
44Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
45Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
46Sep 16 09:00:09 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
47Sep 16 09:00:09 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
48Sep 16 09:00:09 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
49Sep 16 09:00:09 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
50Sep 16 09:03:23 gerrit1001 java[25679]: Sep 16, 2020 9:03:23 AM com.google.common.util.concurrent.AggregateFuture log
51Sep 16 09:03:23 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
52Sep 16 09:03:23 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
53Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
54Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
55Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
56Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
57Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
58Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
59Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
60Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
61Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
62Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
63Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
64Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
65Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
66Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
67Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
68Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
69Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
70Sep 16 09:03:23 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
71Sep 16 09:03:23 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
72Sep 16 09:03:23 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
73Sep 16 09:03:23 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
74Sep 16 09:03:26 gerrit1001 java[25679]: Sep 16, 2020 9:03:26 AM com.google.common.util.concurrent.AggregateFuture log
75Sep 16 09:03:26 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
76Sep 16 09:03:26 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
77Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
78Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
79Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
80Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
81Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
82Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
83Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
84Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
85Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
86Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
87Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
88Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
89Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
90Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
91Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
92Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
93Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
94Sep 16 09:03:26 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
95Sep 16 09:03:26 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
96Sep 16 09:03:26 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
97Sep 16 09:03:26 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
98Sep 16 09:03:33 gerrit1001 java[25679]: Sep 16, 2020 9:03:33 AM com.google.common.util.concurrent.AggregateFuture log
99Sep 16 09:03:33 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
100Sep 16 09:03:33 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
101Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
102Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
103Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
104Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
105Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
106Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
107Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
108Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
109Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
110Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
111Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
112Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
113Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
114Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
115Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
116Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
117Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
118Sep 16 09:03:33 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
119Sep 16 09:03:33 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
120Sep 16 09:03:33 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
121Sep 16 09:03:33 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
122Sep 16 09:03:58 gerrit1001 java[25679]: Sep 16, 2020 9:03:58 AM com.google.common.util.concurrent.AggregateFuture log
123Sep 16 09:03:58 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
124Sep 16 09:03:58 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
125Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
126Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
127Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
128Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
129Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
130Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
131Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
132Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
133Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
134Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
135Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
136Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
137Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
138Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
139Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
140Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
141Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
142Sep 16 09:03:58 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
143Sep 16 09:03:58 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
144Sep 16 09:03:58 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
145Sep 16 09:03:58 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
146Sep 16 09:05:29 gerrit1001 java[25679]: Sep 16, 2020 9:05:29 AM com.google.common.util.concurrent.AggregateFuture log
147Sep 16 09:05:29 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
148Sep 16 09:05:29 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
149Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
150Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
151Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
152Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
153Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
154Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
155Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
156Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
157Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
158Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
159Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
160Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
161Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
162Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
163Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
164Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
165Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
166Sep 16 09:05:29 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
167Sep 16 09:05:29 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
168Sep 16 09:05:29 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
169Sep 16 09:05:29 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
170Sep 16 09:06:15 gerrit1001 java[25679]: Sep 16, 2020 9:06:15 AM com.google.common.util.concurrent.AggregateFuture log
171Sep 16 09:06:15 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
172Sep 16 09:06:15 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
173Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
174Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
175Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
176Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
177Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
178Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
179Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
180Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
181Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
182Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
183Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
184Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
185Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
186Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
187Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
188Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
189Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
190Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
191Sep 16 09:06:15 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
192Sep 16 09:06:15 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
193Sep 16 09:06:15 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
194Sep 16 09:06:15 gerrit1001 java[25679]: Sep 16, 2020 9:06:15 AM com.google.common.util.concurrent.AggregateFuture log
195Sep 16 09:06:15 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
196Sep 16 09:06:15 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
197Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
198Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
199Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
200Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
201Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
202Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
203Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
204Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
205Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
206Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
207Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
208Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
209Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
210Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
211Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
212Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
213Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
214Sep 16 09:06:15 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
215Sep 16 09:06:15 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
216Sep 16 09:06:15 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
217Sep 16 09:06:15 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
218Sep 16 09:06:34 gerrit1001 java[25679]: Sep 16, 2020 9:06:34 AM com.google.common.util.concurrent.AggregateFuture log
219Sep 16 09:06:34 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
220Sep 16 09:06:34 gerrit1001 java[25679]: com.google.gerrit.exceptions.StorageException: Failed to replace change 627755 in index version 60 (current patch set = 1)
221Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:214)
222Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.server.index.change.ChangeIndexer.doIndex(ChangeIndexer.java:177)
223Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.server.index.change.ChangeIndexer.access$700(ChangeIndexer.java:62)
224Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:372)
225Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:362)
226Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.server.index.change.ChangeIndexer$AbstractIndexTask.call(ChangeIndexer.java:351)
227Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
228Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
229Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
230Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
231Sep 16 09:06:34 gerrit1001 java[25679]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
232Sep 16 09:06:34 gerrit1001 java[25679]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
233Sep 16 09:06:34 gerrit1001 java[25679]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
234Sep 16 09:06:34 gerrit1001 java[25679]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
235Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:610)
236Sep 16 09:06:34 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
237Sep 16 09:06:34 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
238Sep 16 09:06:34 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
239Sep 16 09:06:34 gerrit1001 java[25679]: Caused by: com.google.gerrit.exceptions.StorageException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: NRT thread is dead
240Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:279)
241Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:109)
242Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:211)
243Sep 16 09:06:34 gerrit1001 java[25679]: ... 17 more
244Sep 16 09:06:34 gerrit1001 java[25679]: Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: NRT thread is dead
245Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:564)
246Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:545)
247Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:102)
248Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:274)
249Sep 16 09:06:34 gerrit1001 java[25679]: ... 19 more
250Sep 16 09:06:34 gerrit1001 java[25679]: Caused by: java.lang.IllegalStateException: NRT thread is dead
251Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
252Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
253Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
254Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
255Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
256Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
257Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
258Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
259Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
260Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
261Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
262Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
263Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
264Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
265Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
266Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
267Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
268Sep 16 09:06:34 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
269Sep 16 09:06:34 gerrit1001 java[25679]: ... 3 more
270Sep 16 09:06:35 gerrit1001 java[25679]: Sep 16, 2020 9:06:35 AM com.google.common.util.concurrent.AggregateFuture log
271Sep 16 09:06:35 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
272Sep 16 09:06:35 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
273Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
274Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
275Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
276Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
277Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
278Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
279Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
280Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
281Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
282Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
283Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
284Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
285Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
286Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
287Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
288Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
289Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
290Sep 16 09:06:35 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
291Sep 16 09:06:35 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
292Sep 16 09:06:35 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
293Sep 16 09:06:35 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
294Sep 16 09:06:40 gerrit1001 java[25679]: Sep 16, 2020 9:06:40 AM com.google.common.util.concurrent.AggregateFuture log
295Sep 16 09:06:40 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
296Sep 16 09:06:40 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
297Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
298Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
299Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
300Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
301Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
302Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
303Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
304Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
305Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
306Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
307Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
308Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
309Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
310Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
311Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
312Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
313Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
314Sep 16 09:06:40 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
315Sep 16 09:06:40 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
316Sep 16 09:06:40 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
317Sep 16 09:06:40 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
318Sep 16 09:08:13 gerrit1001 java[25679]: Sep 16, 2020 9:08:13 AM com.google.common.util.concurrent.AggregateFuture log
319Sep 16 09:08:13 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
320Sep 16 09:08:13 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
321Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
322Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
323Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
324Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
325Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
326Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
327Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
328Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
329Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
330Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
331Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
332Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
333Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
334Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
335Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
336Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
337Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
338Sep 16 09:08:13 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
339Sep 16 09:08:13 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
340Sep 16 09:08:13 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
341Sep 16 09:08:13 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
342Sep 16 09:09:13 gerrit1001 java[25679]: Sep 16, 2020 9:09:13 AM com.google.common.util.concurrent.AggregateFuture log
343Sep 16 09:09:13 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
344Sep 16 09:09:13 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
345Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
346Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
347Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
348Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
349Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
350Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
351Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
352Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
353Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
354Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
355Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
356Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
357Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
358Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
359Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
360Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
361Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
362Sep 16 09:09:13 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
363Sep 16 09:09:13 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
364Sep 16 09:09:13 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
365Sep 16 09:09:13 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
366Sep 16 09:12:46 gerrit1001 java[25679]: Sep 16, 2020 9:12:46 AM com.google.common.util.concurrent.AggregateFuture log
367Sep 16 09:12:46 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
368Sep 16 09:12:46 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
369Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
370Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
371Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
372Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
373Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
374Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
375Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
376Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
377Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
378Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
379Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
380Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
381Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
382Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
383Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
384Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
385Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
386Sep 16 09:12:46 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
387Sep 16 09:12:46 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
388Sep 16 09:12:46 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
389Sep 16 09:12:46 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
390Sep 16 09:12:51 gerrit1001 java[25679]: Sep 16, 2020 9:12:51 AM com.google.common.util.concurrent.AggregateFuture log
391Sep 16 09:12:51 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
392Sep 16 09:12:51 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
393Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
394Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
395Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
396Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
397Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
398Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
399Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
400Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
401Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
402Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
403Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
404Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
405Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
406Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
407Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
408Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
409Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
410Sep 16 09:12:51 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
411Sep 16 09:12:51 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
412Sep 16 09:12:51 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
413Sep 16 09:12:51 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
414Sep 16 09:12:54 gerrit1001 java[25679]: Sep 16, 2020 9:12:54 AM com.google.common.util.concurrent.AggregateFuture log
415Sep 16 09:12:54 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
416Sep 16 09:12:54 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
417Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
418Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
419Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
420Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
421Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
422Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
423Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
424Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
425Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
426Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
427Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
428Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
429Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
430Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
431Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
432Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
433Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
434Sep 16 09:12:54 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
435Sep 16 09:12:54 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
436Sep 16 09:12:54 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
437Sep 16 09:12:54 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
438Sep 16 09:13:01 gerrit1001 java[25679]: Sep 16, 2020 9:13:01 AM com.google.common.util.concurrent.AggregateFuture log
439Sep 16 09:13:01 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
440Sep 16 09:13:01 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
441Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
442Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
443Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
444Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
445Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
446Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
447Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
448Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
449Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
450Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
451Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
452Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
453Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
454Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
455Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
456Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
457Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
458Sep 16 09:13:01 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
459Sep 16 09:13:01 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
460Sep 16 09:13:01 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
461Sep 16 09:13:01 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
462Sep 16 09:13:52 gerrit1001 java[25679]: Sep 16, 2020 9:13:52 AM com.google.common.util.concurrent.AggregateFuture log
463Sep 16 09:13:52 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
464Sep 16 09:13:52 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
465Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
466Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
467Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
468Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
469Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
470Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
471Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
472Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
473Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
474Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
475Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
476Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
477Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
478Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
479Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
480Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
481Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
482Sep 16 09:13:52 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
483Sep 16 09:13:52 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
484Sep 16 09:13:52 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
485Sep 16 09:13:52 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
486Sep 16 09:14:16 gerrit1001 java[25679]: Sep 16, 2020 9:14:16 AM com.google.common.util.concurrent.AggregateFuture log
487Sep 16 09:14:16 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
488Sep 16 09:14:16 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
489Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
490Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
491Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
492Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
493Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
494Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
495Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
496Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
497Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
498Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
499Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
500Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
501Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
502Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
503Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
504Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
505Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
506Sep 16 09:14:16 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
507Sep 16 09:14:16 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
508Sep 16 09:14:16 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
509Sep 16 09:14:16 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
510Sep 16 09:14:37 gerrit1001 java[25679]: Sep 16, 2020 9:14:37 AM com.google.common.util.concurrent.AggregateFuture log
511Sep 16 09:14:37 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
512Sep 16 09:14:37 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
513Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
514Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
515Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
516Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
517Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
518Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
519Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
520Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
521Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
522Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
523Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
524Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
525Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
526Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
527Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
528Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
529Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
530Sep 16 09:14:37 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
531Sep 16 09:14:37 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
532Sep 16 09:14:37 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
533Sep 16 09:14:37 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
534Sep 16 09:20:20 gerrit1001 java[25679]: Sep 16, 2020 9:20:20 AM com.google.common.util.concurrent.AggregateFuture log
535Sep 16 09:20:20 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
536Sep 16 09:20:20 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
537Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
538Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
539Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
540Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
541Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
542Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
543Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
544Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
545Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
546Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
547Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
548Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
549Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
550Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
551Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
552Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
553Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
554Sep 16 09:20:20 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
555Sep 16 09:20:20 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
556Sep 16 09:20:20 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
557Sep 16 09:20:20 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
558Sep 16 09:20:53 gerrit1001 java[25679]: Sep 16, 2020 9:20:53 AM com.google.common.util.concurrent.AggregateFuture log
559Sep 16 09:20:53 gerrit1001 java[25679]: SEVERE: Got more than one input Future failure. Logging failures after the first
560Sep 16 09:20:53 gerrit1001 java[25679]: java.lang.IllegalStateException: NRT thread is dead
561Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.gerrit.lucene.AbstractLuceneIndex$NrtFuture.isDone(AbstractLuceneIndex.java:452)
562Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:797)
563Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)
564Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)
565Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)
566Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
567Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
568Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
569Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)
570Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.Futures$NonCancellationPropagatingFuture.run(Futures.java:790)
571Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
572Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)
573Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)
574Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:738)
575Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:131)
576Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:133)
577Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
578Sep 16 09:20:53 gerrit1001 java[25679]: at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
579Sep 16 09:20:53 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
580Sep 16 09:20:53 gerrit1001 java[25679]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
581Sep 16 09:20:53 gerrit1001 java[25679]: at java.lang.Thread.run(Thread.java:748)
582Sep 16 09:23:04 gerrit1001 systemd[1]: Stopping Gerrit code review tool...
583Sep 16 09:23:21 gerrit1001 java[25679]: log4j:WARN No appenders could be found for logger (com.google.gerrit.server.config.ScheduleConfig).
584Sep 16 09:23:21 gerrit1001 java[25679]: log4j:WARN Please initialize the log4j system properly.
585Sep 16 09:23:21 gerrit1001 java[25679]: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
586Sep 16 09:23:33 gerrit1001 systemd[1]: gerrit.service: Main process exited, code=exited, status=130/n/a
587Sep 16 09:23:33 gerrit1001 systemd[1]: gerrit.service: Failed with result 'exit-code'.
588Sep 16 09:23:33 gerrit1001 systemd[1]: Stopped Gerrit code review tool.
589Sep 16 09:23:33 gerrit1001 systemd[1]: Started Gerrit code review tool.
590Sep 16 09:23:38 gerrit1001 java[20477]: Sep 16, 2020 9:23:38 AM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject
591Sep 16 09:23:38 gerrit1001 java[20477]: WARNING: AssistedInject factory com.google.gerrit.sshd.DispatchCommand$Factory will be slow because class com.google.gerrit.sshd.DispatchCommand has assisted Provider dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!) The exact offending deps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.google.gerrit.sshd.BaseCommand.injector]
592Sep 16 09:23:42 gerrit1001 java[20477]: Sep 16, 2020 9:23:42 AM com.google.inject.servlet.GuiceFilter setPipeline
593Sep 16 09:23:42 gerrit1001 java[20477]: WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.
594Sep 16 09:23:43 gerrit1001 java[20477]: Sep 16, 2020 9:23:43 AM com.google.inject.servlet.GuiceFilter setPipeline
595Sep 16 09:23:43 gerrit1001 java[20477]: WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.
596Sep 16 09:23:43 gerrit1001 java[20477]: Sep 16, 2020 9:23:43 AM com.google.inject.servlet.GuiceFilter setPipeline
597Sep 16 09:23:43 gerrit1001 java[20477]: WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.
598Sep 16 09:23:43 gerrit1001 java[20477]: Sep 16, 2020 9:23:43 AM com.google.inject.servlet.GuiceFilter setPipeline
599Sep 16 09:23:43 gerrit1001 java[20477]: WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.
600Sep 16 09:23:43 gerrit1001 java[20477]: Sep 16, 2020 9:23:43 AM com.google.inject.servlet.GuiceFilter setPipeline
601Sep 16 09:23:43 gerrit1001 java[20477]: WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.
602Sep 16 09:23:43 gerrit1001 java[20477]: Sep 16, 2020 9:23:43 AM com.google.inject.servlet.GuiceFilter setPipeline
603Sep 16 09:23:43 gerrit1001 java[20477]: WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.
604Sep 16 09:23:43 gerrit1001 java[20477]: Sep 16, 2020 9:23:43 AM com.google.inject.servlet.GuiceFilter setPipeline
605Sep 16 09:23:43 gerrit1001 java[20477]: WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.
606Sep 16 09:23:44 gerrit1001 java[20477]: Sep 16, 2020 9:23:44 AM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject
607Sep 16 09:23:44 gerrit1001 java[20477]: WARNING: AssistedInject factory com.googlesource.gerrit.plugins.replication.Destination$Factory will be slow because class com.googlesource.gerrit.plugins.replication.Destination has assisted Provider dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!) The exact offending deps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.googlesource.gerrit.plugins.replication.Destination.<init>()[0]]
608Sep 16 09:23:45 gerrit1001 java[20477]: Sep 16, 2020 9:23:45 AM com.google.inject.servlet.GuiceFilter setPipeline
609Sep 16 09:23:45 gerrit1001 java[20477]: WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

Event Timeline

hashar updated the task description. (Show Details)

From

Gerrit_3months_usedMemory.png (499×1 px, 21 KB)

It looks like memory usage is slowing growing since mid august

That might be related to the deployment of git-protocol-v2 on clients. For CI that was T256844 done on August 18th and 19th:

August 18th, most jobs, 7d25340b0024631c194a06afa6d89f89a209ec99
August 19th, Quibble jobs, a566f054effdd9f46b5b9a3fa4031844fe0efd46

Which might be triggering an issue of some sort in jgit and our Gerrit deployment comes with jgit v5.8.1.202007141445-r.

I went to take a heap dump of current memory usage without objects pending finalization/garbage collection (dump:live):

sudo -u gerrit2 jmap -dump:live,format=b,file=/var/lib/gerrit-202009170755.hprof <pid of java here>

Which I have moved in my home dir. Ran jhat -stack false -refs false gerrit-202009170755.hprof which takes a while and boot a web server on port 7000.

On my local machine: ssh -C -L 8080:ip6-localhost:7000 gerrit1001.wikimedia.org and then http://ip6-localhost:8080/

Some metrics:

Instance Counts for All Classes (excluding platform)

2237744 instances of class org.eclipse.jgit.lib.ObjectId
2128766 instances of class org.eclipse.jgit.lib.ObjectIdRef$PeeledNonTag
735294 instances of class org.eclipse.jetty.util.thread.Locker
735294 instances of class org.eclipse.jetty.util.thread.Locker$Lock
735283 instances of class org.eclipse.jetty.server.session.Session
735283 instances of class org.eclipse.jetty.server.session.Session$SessionInactivityTimer
735283 instances of class org.eclipse.jetty.server.session.Session$SessionInactivityTimer$1
735283 instances of class org.eclipse.jetty.server.session.SessionData
198756 instances of class org.eclipse.jgit.internal.storage.file.ByteArrayWindow
198756 instances of class org.eclipse.jgit.internal.storage.file.WindowCache$Entry
198756 instances of class org.eclipse.jgit.internal.storage.file.WindowCache$SoftRef
172260 instances of class org.h2.value.ValueJavaObject
163182 instances of class com.codahale.metrics.WeightedSnapshot$WeightedSample
146256 instances of class org.h2.value.ValueTimestamp
136314 instances of class [Lorg.h2.value.Value;
134132 instances of class com.googlecode.javaewah.EWAHCompressedBitmap
134132 instances of class com.googlecode.javaewah.LongArray
134132 instances of class com.googlecode.javaewah.RunningLengthWord
130280 instances of class org.eclipse.jgit.internal.storage.file.BasePackBitmapIndex$StoredBitmap
113191 instances of class org.h2.result.Row
82653 instances of class org.eclipse.jgit.diff.Edit
70534 instances of class org.eclipse.jgit.revwalk.RevCommit
66529 instances of class [Lorg.eclipse.jgit.revwalk.RevCommit;
64096 instances of class org.eclipse.jgit.revwalk.RevTree
57190 instances of class org.eclipse.jgit.internal.storage.file.BasePackBitmapIndex$XorCompressedBitmap
...

Instance Counts for All Classes (Including platform)

4859984 instances of class java.lang.String
3839858 instances of class [C
3830534 instances of class java.util.concurrent.ConcurrentHashMap$Node
2237744 instances of class org.eclipse.jgit.lib.ObjectId
2128766 instances of class org.eclipse.jgit.lib.ObjectIdRef$PeeledNonTag
1830871 instances of class [B
750976 instances of class java.util.concurrent.ConcurrentHashMap
750486 instances of class java.util.concurrent.atomic.AtomicReference
745988 instances of class [I
743997 instances of class java.util.concurrent.locks.ReentrantLock$NonfairSync
742434 instances of class [Ljava.util.concurrent.ConcurrentHashMap$Node;
742425 instances of class java.util.concurrent.locks.ReentrantLock
735428 instances of class java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
735294 instances of class org.eclipse.jetty.util.thread.Locker
735294 instances of class org.eclipse.jetty.util.thread.Locker$Lock
735283 instances of class org.eclipse.jetty.server.session.Session
735283 instances of class org.eclipse.jetty.server.session.Session$SessionInactivityTimer
735283 instances of class org.eclipse.jetty.server.session.Session$SessionInactivityTimer$1
735283 instances of class org.eclipse.jetty.server.session.SessionData
229023 instances of class java.util.LinkedHashMap$Entry
209390 instances of class java.util.HashMap$Node
198756 instances of class org.eclipse.jgit.internal.storage.file.ByteArrayWindow
198756 instances of class org.eclipse.jgit.internal.storage.file.WindowCache$Entry
198756 instances of class org.eclipse.jgit.internal.storage.file.WindowCache$SoftRef
172260 instances of class org.h2.value.ValueJavaObject
166482 instances of class [J
163533 instances of class java.util.concurrent.ConcurrentSkipListMap$Node
163230 instances of class java.lang.Double
163182 instances of class com.codahale.metrics.WeightedSnapshot$WeightedSample
146256 instances of class org.h2.value.ValueTimestamp
136314 instances of class [Lorg.h2.value.Value;
134132 instances of class com.googlecode.javaewah.EWAHCompressedBitmap
134132 instances of class com.googlecode.javaewah.LongArray
134132 instances of class com.googlecode.javaewah.RunningLengthWord
130280 instances of class org.eclipse.jgit.internal.storage.file.BasePackBitmapIndex$StoredBitmap
113191 instances of class org.h2.result.Row
99759 instances of class [Ljava.lang.Object;
82653 instances of class org.eclipse.jgit.diff.Edit
...

Heap Histogram All Classes (excluding platform)

ClassInstance CountTotal Size
class [B18308713577406468
class [I745988597106484
class [C3839858317539484
class [Ljava.util.concurrent.ConcurrentHashMap$Node;742434117898752
class java.util.concurrent.ConcurrentHashMap$Node3830534107254952
class [J166482100194184
class org.eclipse.jetty.server.session.SessionData73528372057734
class org.eclipse.jgit.lib.ObjectIdRef$PeeledNonTag212876668120512
class java.util.concurrent.ConcurrentHashMap75097663081984
class java.lang.String485998458319808
class org.eclipse.jetty.server.session.Session73528349263961
class org.eclipse.jgit.lib.ObjectId223774444754880
class org.eclipse.jetty.server.session.Session$SessionInactivityTimer$173528323529056
class java.util.concurrent.locks.ReentrantLock$NonfairSync74399720831916
class [Lorg.eclipse.jgit.lib.Ref;492420170888
class959319659136
class java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject73542817650272
class [Lorg.eclipse.jgit.lib.ObjectIdOwnerMap$Entry;98016072000
class [Ljava.lang.Object;9975913955856
class org.eclipse.jgit.internal.storage.file.WindowCache$SoftRef19875613515408
class org.eclipse.jetty.util.thread.Locker73529411764704
class org.eclipse.jetty.server.session.Session$SessionInactivityTimer73528311764528
class java.util.LinkedHashMap$Entry22902310077012
class48019834464

you might also try something like https://heaphero.io/ to see if there are any additional insights it can give you. I think I'm still paying for fastthread.io (which is the same company -- may be useful).

Change 628338 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] gerrit: dump heap on out of memory error

https://gerrit.wikimedia.org/r/628338

you might also try something like https://heaphero.io/ to see if there are any additional insights it can give you. I think I'm still paying for fastthread.io (which is the same company -- may be useful).

Nice they have bunch of nice utilities. There are a couple issues though: the heap dump might well contain a bunch of credentials and personal information AND it was 5GBytes when I took one without the objects that were ready for garbage collection :]

Change 628338 merged by Dzahn:
[operations/puppet@production] gerrit: dump heap on out of memory error

https://gerrit.wikimedia.org/r/628338

Mentioned in SAL (#wikimedia-operations) [2020-09-18T15:09:07Z] <mutante> restarting gerrit service to apply gerrit::628338 to make it dump heap if out of memory (T263008)

Here is the memory report for the last 15 days:

gerrit_mem_usage_15days.png (499×1 px, 23 KB)

We had roughly 4GBytes heap mean usage increase over the last few days. Does not seem too concerning so far.

Based on https://gerrit.wikimedia.org/r/monitoring?part=heaphisto out of 20GBytes used:

Heap Classes: 10,523, Instances: 225,343,723, Kilo-Bytes: 20,459,013

ClassSize (Kb)% sizeInstances% instances
byte[]6,223,648305,242,6482
char[]1,993,422919,636,8868
java.util.concurrent.ConcurrentHashMap$Node1,960,992941,834,51118
java.util.concurrent.ConcurrentHashMap$Node[]1,376,31268,365,0243
org.eclipse.jetty.server.session.SessionData974,22548,313,3883
java.lang.String903,286428,905,15812
java.util.concurrent.ConcurrentHashMap851,00148,379,0943
int[]823,9324901,2270
org.eclipse.jetty.server.session.Session714,43138,313,3883

And after executing the garbage collector:

Garbage collector executed (5,703,919 Kb freed, 16,478,791 Kb still used)

Heap Classes: 8,819, Instances: 207,771,485, Kilo-Bytes: 16,613,165

ClassSize (Kb)% sizeInstances% instances
byte[]4,000,197241,888,0270
java.util.concurrent.ConcurrentHashMap$Node1,961,1331141,837,52020
char[]1,913,9901118,800,2899
java.util.concurrent.ConcurrentHashMap$Node[]1,376,53788,368,6744
org.eclipse.jetty.server.session.SessionData974,68158,317,2794
java.lang.String886,104528,355,34013
java.util.concurrent.ConcurrentHashMap851,33858,382,4094
org.eclipse.jetty.server.session.Session714,76648,317,2794
int[]597,7513762,6380

So it reclaimed 2GBytes of byte[] and 3.7G of other things.

And eventually 15 days after I triggered a full gc, we had very hight memory usage and @thcipriani had Gerrit restarted due to gc trashing:

17:43 	<thcipriani> 	restarting gerrit due to gc thrashing

gerrit_used_memory.png (499×1 px, 24 KB)

Could it be that we had full gc disabled somehow?

Gerrit tuning infos: https://ctf-dev-environment-vagrant.s3.amazonaws.com/Gerrit-Performance-Tuning-Cheat-Sheet.pdf

We probably want to look at jgit config:

modules/gerrit/templates/gerrit.config.erb
[core]
    packedGitOpenFiles = <%= @git_open_files %>  # 20000
    packedGitLimit = 4g
    packedGitWindowSize = 16k
    streamFileThreshold = 100m
    useRecursiveMerge = true

The packedGitOpenFiles got raised from 6k to 20k three years ago due to T168360: Gerrit constantly throws HTTP 500 error when reviewing patches (due to "Too many open files").

Gerrit tuning infos: https://ctf-dev-environment-vagrant.s3.amazonaws.com/Gerrit-Performance-Tuning-Cheat-Sheet.pdf

We probably want to look at jgit config:

modules/gerrit/templates/gerrit.config.erb
[core]
    packedGitOpenFiles = <%= @git_open_files %>  # 20000
    packedGitLimit = 4g

^ packedGitLimit is, I would guess, where we're running into issues. I last looked at this about a year ago in T221026#5113741

I have seen recommendations from jGit developers running git to have this be half the size of the heap. My hunch is that since we started using protocol v2 we're seeing more pack files than we had previously. We've also added 555 projects between April 2019 (when I last touched this) and now. Doubling this seems like a good starting point. Possibly also increasing overall heap size to accommodate the extra space without sacrificing anything else using heap.

Mentioned in SAL (#wikimedia-operations) [2020-11-03T21:16:43Z] <hashar> Gerrit: triggering java garbage collection # T263008

Found some more settings mentioned at http://gitblit.com/setup_scaling.html , they come from a presentation by Shawn Pearce ( page 18 of https://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/Scaling%20Up%20JGit%20-%20EclipseCon%202013.pdf ).

# -Xmx8g # gobs of JVM heap
# for `jgit daemon` or gerrit.config
[core]
    packedGitLimit = 4g # around 50% of JVM heap
    packedGitOpenFiles = 8192 # don't forget to configure ulimit
    streamFileThreshold = 2047m # avoids pathological inflate
    deltaBaseCacheLimit = 50m

# applies to `jgit gc`
[pack]
    bigFileThreshold = 20m # don't delta compress big binaries
    indexVersion = 2 # use index v2

So essentially, yes we will have to finely tweak jgit via settings at https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#core Some hints:

core.streamFileThreshold

Largest object size, in bytes, that JGit will allocate as a contiguous byte array. Any file revision larger than this threshold will have to be streamed, typically requiring the use of temporary files under '$GIT_DIR/objects' to implement pseudo-random access during delta decompression.

Servers with very high traffic should set this to be larger than the size of their common big files. For example a server managing the Android platform typically has to deal with ~10-12 MiB XML files, so 15 m would be a reasonable setting in that environment. Setting this too high may cause the JVM to run out of heap space when handling very big binary files, such as device firmware or CD-ROM ISO images.

Defaults to 25% of the available JVM heap, limited to 2g.

Common unit suffixes of 'k', 'm', or 'g' are supported.

So if I read it properly, we could have pack files up to 2G. Looking at lsof there are a few large ones:

SizeFile
1319788544/var/lib/gerrit2/review_site/cache/diff_summary.h2.db
734429184/var/lib/gerrit2/review_site/db/account_patch_reviews.h2.db
624658313/srv/gerrit/git/mediawiki/core.git/objects/pack/pack-08906534fd015f62a7a7070cc7a1fe663582e6c6.pack
530757632/var/lib/gerrit2/review_site/cache/mergeability.h2.db
466316294/srv/gerrit/git/operations/debs/druid.git/objects/pack/pack-ba20d7aad9993dc0e7118fdeb490da3c3f9469de.pack
379402357/srv/gerrit/git/operations/puppet.git/objects/pack/pack-740faa9207ecd93d0352c64e8162626e8579c5ea.pack
377612288/var/lib/gerrit2/review_site/cache/diff_intraline.h2.db
297239011/srv/gerrit/git/maps/kartotherian/deploy.git/objects/pack/pack-de5cad0dc13ec50b72c7db9994693378763b9085.pack
283981759/srv/gerrit/git/mediawiki/core.git/objects/pack/pack-ecaf47ec8db5b144cc4d2e250770997f20bc88db.pack
273512448/var/lib/gerrit2/review_site/cache/change_kind.h2.db
253257728/var/lib/gerrit2/review_site/cache/diff.h2.db
206188544/var/lib/gerrit2/review_site/cache/conflicts.h2.db

But I guess we want to support mediawiki/core @ 624MBytes

core.repositoryCacheExpireAfter

Time an unused repository should expire and be evicted from the repository cache.

Values can be specified using standard time unit abbreviations (ms, sec, min, etc.).
Default is 1 hour.

Maybe we can lower that one to force eviction. But what would be really nice, is to have some metrics about jgit daemon :/

That happened on the Gerrit replica gerrit2001:

Nov 07 13:43:35 gerrit2001 java[30197]: java.lang.OutOfMemoryError: Java heap space
Nov 07 13:43:35 gerrit2001 java[30197]: Dumping heap to /srv/gerrit/java_pid30197.hprof ...
Nov 07 13:47:02 gerrit2001 java[30197]: Heap dump file created [35616147146 bytes in 206.962 secs]

Since we now run Gerrit with:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/srv/gerrit

We have a nice 34GB heap dump to inspect on gerrit2001 at /srv/gerrit/java_pid30197.hprof.

Looks like I forgot to flag monitoring as a requirement to solve this one: T184086 . Without metrics, we dont quite know what is going on in Gerrit internals.

Happened again 2020-12-18:

gerrit_boom.png (499×1 px, 21 KB)

The exact same slow pill up of heap usage :-\

One of the things I anticipated was that the jgit block cache would fill up, but it seems like that's not the case. That is, it seems like the closer we get to running out of memory the more the jvm ditches jgit block cache space. This likely means slower operations, a bigger stack up of threads, and more memory flailing.

Screenshot-2020-12-21-13:09:10.png (749×1 px, 152 KB)

Mentioned in SAL (#wikimedia-operations) [2020-12-22T08:52:02Z] <hashar> gerrit: running jhat heap analyzer on gerrit2001 # T263008

hashar changed the task status from Open to Stalled.Dec 22 2020, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2020-12-22T08:52:02Z] <hashar> gerrit: running jhat heap analyzer on gerrit2001 # T263008

And jhat eventually ran out of HEAP (default 16GBytes) :(

I am marking this stalled pending:

  • upgrade to Java 11 T268225 which come with an improved gc
  • Gerrit 3.3 T262241

There seems to be a memory leak with jetty sessions:

1$ sudo -u gerrit2 jcmd 2347 GC.class_histogram|grep -i jetty
2 num #instances #bytes class name
3----------------------------------------------
4 5: 10042773 1205132760 org.eclipse.jetty.server.session.SessionData
5 8: 10042773 883764024 org.eclipse.jetty.server.session.Session
6 11: 10042773 482053104 org.eclipse.jetty.server.session.Session$SessionInactivityTimer$1
7 13: 10042779 321368928 org.eclipse.jetty.util.thread.Locker
8 14: 10042773 321368736 org.eclipse.jetty.server.session.Session$SessionInactivityTimer
9 17: 10042779 241026696 org.eclipse.jetty.util.thread.Locker$Lock
10

There is really no reason for having a million of jetty session being held.

There are some informations at https://github.com/dropwizard/dropwizard/issues/2455

And 19 hours after P13625:

hashar@gerrit2001:~$ sudo -u gerrit2 jcmd 2347 GC.class_histogram|grep -i jetty
   4:      10512257     1261470840  org.eclipse.jetty.server.session.SessionData
   7:      10512257      925078616  org.eclipse.jetty.server.session.Session
  10:      10512257      504588336  org.eclipse.jetty.server.session.Session$SessionInactivityTimer$1
  13:      10512267      336392544  org.eclipse.jetty.util.thread.Locker
  14:      10512257      336392224  org.eclipse.jetty.server.session.Session$SessionInactivityTimer
  17:      10512267      252294408  org.eclipse.jetty.util.thread.Locker$Lock

500k more objects / 56 MB more just for org.eclipse.jetty.server.session.SessionData. So it is slowly leaking memory.

Filed upstream https://bugs.chromium.org/p/gerrit/issues/detail?id=13858

It is neither Java 8 or Gerrit 3.2 related as far as I can tell, will drop the sub tasks.

hashar set Security to Software security bug.Jan 12 2021, 10:58 PM
hashar added projects: Security, Security-Team.
hashar changed the visibility from "Public (No Login Required)" to "Custom Policy".
hashar changed the subtype of this task from "Task" to "Security Issue".
hashar moved this task from Backlog to Patch proposed upstream on the Upstream board.

Reported to Upstream and we have found the root cause. It is being worked on :]

For Security-Team that is a potential denial of service in Gerrit. After a few millions of git operations over https, the Java VM runs out of HEAP memory. Though surely millions of requests in a short time frame might be noticed.

Workaround: restart Gerrit instance to clear out the accumulated zombie objects.

I guess upstream will release a security release over the next few weeks.

I have restarted both of our Gerrit instances to clear out the memory leak.

hashar changed the task status from Stalled to Open.Feb 2 2021, 8:46 AM
hashar updated the task description. (Show Details)
hashar changed the visibility from "Custom Policy" to "Public (No Login Required)".

Back to public. The fix has been released: https://www.gerritcodereview.com/3.2.html#327

hashar closed subtask T273223: Upgrade Gerrit to 3.2.7 as Resolved.

That was definitely epic and has been kindly addressed by upstream with Gerrit 3.2.7 \o/