Page MenuHomePhabricator
Create Task
Maniphest T423027

2026-04-12 Gerrit Outage (was: DiskSpace)
Closed, ResolvedPublic
Actions

Description

Common information

  • alertname: DiskSpace
  • cluster: misc
  • device: /dev/mapper/vg0-root
  • fstype: ext4
  • instance: gerrit2003:9100
  • job: node
  • mountpoint: /
  • prometheus: ops
  • severity: critical
  • site: codfw
  • source: prometheus
  • team: collaboration-services

Firing alerts

Needed incidents followups

  • revisit gerrit paging policy T423035 and T365148
  • broken Gerrit CI is not triggering any alerts T423123
  • non-existing runbooks and documented gerrit ssh commands are not working when Gerrit is stuck/down T423601
    • how to restart CI, how to restart gerrit, disk issues on Gerrit T423601
  • have gerrit ssh commands to elevate privileges T423601
  • move data away from / T333143

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
gerrit: migrate gerrit_site away from root partitionArnaudboperations/puppetproduction+9 -45
Customize query in gerrit

Related Objects
Search...

Event Timeline

phaultfinder created this task.Apr 12 2026, 10:08 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 12 2026, 10:08 AM
A_smart_kitten subscribed.Apr 12 2026, 10:54 AM
phaultfinder updated the task description. (Show Details)Apr 12 2026, 2:08 PM
A_smart_kitten triaged this task as Unbreak Now! priority.Apr 12 2026, 2:31 PM
A_smart_kitten added a project: Gerrit.

Gerrit currently seems completely down & I would assume that this is why.

upstream connect error or disconnect/reset before headers. reset reason: connection timeout
RhinosF1 subscribed.Apr 12 2026, 2:37 PM
RhinosF1 added a project: Wikimedia-Incident.Apr 12 2026, 2:40 PM
IKhitron subscribed.Apr 12 2026, 2:44 PM
Clement_Goubert subscribed.Apr 12 2026, 2:44 PM
cgoubert@gerrit2003:/var/log/apache2$ sudo lvextend -L+20G -r /dev/vg0/root 
  Size of logical volume vg0/root changed from 74.50 GiB (19073 extents) to 94.50 GiB (24193 extents).
  Logical volume vg0/root successfully resized.
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/mapper/vg0-root is mounted on /; on-line resizing required
old_desc_blocks = 10, new_desc_blocks = 12
The filesystem on /dev/mapper/vg0-root is now 24773632 (4k) blocks long.
RhinosF1 renamed this task from DiskSpace to 2026-04-12 Gerrit Outage (was: DiskSpace).Apr 12 2026, 2:45 PM
RhinosF1 created subtask T423035: Enable paging for Gerrit (was: Gerrit outage didn't page until 4.5 hours after the first alert).
RhinosF1 mentioned this in T423035: Enable paging for Gerrit (was: Gerrit outage didn't page until 4.5 hours after the first alert).
Alien333 subscribed.Apr 12 2026, 2:52 PM
IKhitron added a comment.EditedApr 12 2026, 3:04 PM

Much better, thank you. It has logical problems now, but at least it works, partially. (For example, a click on "save" saves the new version, but shows an error message "was not saved, same as original".) If you need details, please ping me, for some reason Phabricator subscription mailing does not work today, maybe a ping will. Thanks.

UPD: Also, "publish edit" works, but shows a message, something like "can't publish, edit does not exist, the code the same as before".
UPD UPD: The Jenkins bot does not run for 20 minutes now, 10 of them after "recheck".

Clement_Goubert lowered the priority of this task from Unbreak Now! to High.Apr 12 2026, 3:07 PM

Temporary fix was to extend the root LV by 20GB. This should hold us over until Monday when Collaboration-Services and Release-Engineering-Team can take a deeper look.

Lowering priority to High, as it's unbroken.

Jdforrester-WMF subscribed.Apr 12 2026, 6:19 PM

Zuul doesn't seem to be picking up events from gerrit. I'll have a poke.

Stashbot added a comment.Apr 12 2026, 6:21 PM

Mentioned in SAL (#wikimedia-releng) [2026-04-12T18:21:21Z] <James_F> jforrester@contint1002:~$ sudo /usr/sbin/service zuul restart && tail -f -n100 /var/log/zuul/zuul.log # T423027

Jdforrester-WMF added a comment.Apr 12 2026, 6:30 PM

I think the issue is in the gerrit<->contint connection. I've gracefully-restarted zuul but it isn't picking up any events. A manual zuul enqueue works (post-merge on 1269498), including the reporting back, but logged as an error:

2026-04-12 18:28:34,840 INFO zuul.IndependentPipelineManager: Reporting item <QueueItem 0x7f5df8c37f90 for <Change 0x7f5df97f1c90 1269498,2> in postmerge>, actions: [<GerritReporter connection: gerrit://gerrit>]
2026-04-12 18:28:35,233 ERROR zuul.IndependentPipelineManager: Exception while reporting:
Traceback (most recent call last):
  File "/srv/deployment/zuul/venv/lib/python2.7/site-packages/zuul/scheduler.py", line 1768, in _reportItem
    ret = self.sendReport(actions, self.pipeline.source, item)
  File "/srv/deployment/zuul/venv/lib/python2.7/site-packages/zuul/scheduler.py", line 1293, in sendReport
    ret = reporter.report(source, self.pipeline, item)
  File "/srv/deployment/zuul/venv/lib/python2.7/site-packages/zuul/reporter/gerrit.py", line 39, in report
    message, self.reporter_config)
  File "/srv/deployment/zuul/venv/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 291, in review
    out, err = self._ssh(cmd)
  File "/srv/deployment/zuul/venv/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 406, in _ssh
    raise Exception("Gerrit error executing %s" % command, err)
Exception: (u'Gerrit error executing gerrit review --project mediawiki/extensions/ReadingLists --message "Post-merge build succeeded.\n\n- https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/54148/console : SUCCESS in 1m 23s\n- https://integration.wikimedia.org/ci/job/mwext-codehealth-master-non-voting/95020/console : SUCCESS in 4m 14s (non-voting)\n- https://integration.wikimedia.org/ci/job/generic-node24-browser-coverage-publish/2/console : SUCCESS in 58s\n" --tag autogenerated:ci-postmerge 1269498,2', 'error: fatal: Cannot post review\n\nfatal: one or more reviews failed; review output above\n')

Despite the logged error above, it did report back successfully: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ReadingLists/+/1269498/2#message-242f3f28db2b4b81a339393a81c0a25500734292

Jdforrester-WMF added a subscriber: hashar.Apr 12 2026, 7:09 PM
IKhitron added a comment.Apr 12 2026, 7:43 PM

Looks like it's more than that. I tried to send recheck many times, but there is no way to write a replay.

Jdforrester-WMF added a comment.Apr 12 2026, 7:52 PM
In T423027#11812236, @IKhitron wrote:

Looks like it's more than that. I tried to send recheck many times, but there is no way to write a replay.

Yes, that's what I said, the gerrit<->contint connection is broken, so zuul never gets the messages. Please don't hammer the server expecting different results.

IKhitron added a comment.EditedApr 12 2026, 7:55 PM

Thank you. Sure, I will stop, if you think it's better. But as it looks for me, it's not that the recheck reply does not invoke the Jenkins bot, it's the Gerrit UI does not allow to write it at the first place, even as a simple text that anybody can read. Of course, I do not know how it is build from inside.
UPD: Many comments appeared now, at least three times more than I sent. Part of them have the word "recheck", others are empty somehow.

tstarling subscribed.Apr 13 2026, 3:25 AM

Here's the Gerrit error log when I hit +2 on a core change. Lucene throws an exception with "No space left on device".

[2026-04-13T03:23:16.121Z] [HTTP POST /r/changes/mediawiki%2Fcore~1250615/revisions/46/review (tstarling from [2620:0:860:4:208:80:153:116])] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /r/changes/mediawiki%2Fcore~1250615/revisions/46/review (view: restapi.change.PostReview): AlreadyClosedException [CONTEXT project="mediawiki/core" request="REST /r/*/mediawiki%2Fcore~1250615/*/46/*" ]
com.google.gerrit.server.update.UpdateException: java.util.concurrent.ExecutionException: com.google.gerrit.exceptions.StorageException: Failed to reindex change 1250615 in index version 86 (current patch set = 46)
        at com.google.gerrit.server.update.BatchUpdates.wrapAndThrowException(BatchUpdates.java:197)
        at com.google.gerrit.server.update.BatchUpdates.execute(BatchUpdates.java:139)
        at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:373)
        at com.google.gerrit.server.restapi.change.PostReview.lambda$runBatchUpdate$2(PostReview.java:436)
        at com.google.gerrit.server.update.RetryableChangeAction.lambda$new$0(RetryableChangeAction.java:48)
        at com.google.gerrit.server.update.RetryableAction.lambda$new$0(RetryableAction.java:94)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.google.gerrit.server.update.RetryHelper.executeWithTimeoutCount(RetryHelper.java:598)
        at com.google.gerrit.server.update.RetryHelper.execute(RetryHelper.java:541)
        at com.google.gerrit.server.update.RetryableAction.call(RetryableAction.java:192)
        at com.google.gerrit.server.update.RetryableChangeAction.call(RetryableChangeAction.java:84)
        at com.google.gerrit.server.restapi.change.PostReview.runBatchUpdate(PostReview.java:440)
        at com.google.gerrit.server.restapi.change.PostReview.apply(PostReview.java:352)
        at com.google.gerrit.server.restapi.change.PostReview.apply(PostReview.java:220)
        at com.google.gerrit.server.restapi.change.PostReview.apply(PostReview.java:117)
        at com.google.gerrit.httpd.restapi.RestApiServlet.lambda$invokeRestModifyViewWithRetry$7(RestApiServlet.java:890)
        at com.google.gerrit.server.update.RetryableAction.lambda$new$0(RetryableAction.java:94)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.google.gerrit.server.update.RetryHelper.executeWithTimeoutCount(RetryHelper.java:598)
        at com.google.gerrit.server.update.RetryHelper.execute(RetryHelper.java:541)
        at com.google.gerrit.server.update.RetryableAction.call(RetryableAction.java:192)
        at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestEndpointWithRetry(RestApiServlet.java:965)
        at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestModifyViewWithRetry(RestApiServlet.java:885)
        at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:533)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:293)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:283)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:403)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139)
        at com.googlesource.gerrit.plugins.readonly.ReadOnly.doFilter(ReadOnly.java:79)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
        at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:66)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
        at com.ericsson.gerrit.plugins.goimport.GoImportFilter.doFilter(GoImportFilter.java:147)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
        at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:56)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:92)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.util.concurrent.ExecutionException: com.google.gerrit.exceptions.StorageException: Failed to reindex change 1250615 in index version 86 (current patch set = 46)
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:594)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:573)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:110)
        at com.google.gerrit.server.update.BatchUpdates.execute(BatchUpdates.java:121)
        ... 91 more
Caused by: com.google.gerrit.exceptions.StorageException: Failed to reindex change 1250615 in index version 86 (current patch set = 46)
        at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:343)
        at com.google.gerrit.server.index.change.ChangeIndexer.doIndex(ChangeIndexer.java:297)
        at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:602)
        at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:563)
        at com.google.gerrit.server.index.change.ChangeIndexer$AbstractIndexTask.call(ChangeIndexer.java:551)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:703)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        ... 1 more
Caused by: com.google.gerrit.exceptions.StorageException: java.util.concurrent.ExecutionException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:226)
        at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:104)
        at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:339)
        ... 15 more
Caused by: java.util.concurrent.ExecutionException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:594)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:573)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:110)
        at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:221)
        ... 17 more
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:908)
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:921)
        at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1542)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1830)
        at com.google.gerrit.lucene.AutoCommitWriter.updateDocument(AutoCommitWriter.java:98)
        at com.google.gerrit.lucene.AbstractLuceneIndex.lambda$replace$2(AbstractLuceneIndex.java:295)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
        ... 3 more
Caused by: java.io.IOException: No space left on device
        at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at java.base/sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62)
        at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:97)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:67)
        at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:288)
        at java.base/java.nio.channels.Channels.writeFullyImpl(Channels.java:74)
        at java.base/java.nio.channels.Channels.writeFully(Channels.java:96)
        at java.base/java.nio.channels.Channels$1.write(Channels.java:171)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:400)
        at java.base/java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
        at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
        at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
        at org.apache.lucene.store.OutputStreamIndexOutput.getChecksum(OutputStreamIndexOutput.java:104)
        at org.apache.lucene.codecs.CodecUtil.writeCRC(CodecUtil.java:643)
        at org.apache.lucene.codecs.CodecUtil.writeFooter(CodecUtil.java:411)
        at org.apache.lucene.codecs.lucene90.Lucene90NormsConsumer.close(Lucene90NormsConsumer.java:74)
        at org.apache.lucene.util.IOUtils.close(IOUtils.java:85)
        at org.apache.lucene.util.IOUtils.close(IOUtils.java:72)
        at org.apache.lucene.index.IndexingChain.writeNorms(IndexingChain.java:468)
        at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:245)
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:392)
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:492)
        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:671)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:570)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345)
        at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
        at com.google.gerrit.lucene.WrappableSearcherManager.refreshIfNeeded(WrappableSearcherManager.java:142)
        at com.google.gerrit.lucene.WrappableSearcherManager.refreshIfNeeded(WrappableSearcherManager.java:63)
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
        at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
        at org.apache.lucene.search.ControlledRealTimeReopenThread.run(ControlledRealTimeReopenThread.java:232)
Stashbot added a comment.Apr 13 2026, 3:34 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-13T03:34:41Z] <TimStarling> on gerrit2003 restarted gerrit T423027

tstarling added a comment.Apr 13 2026, 3:38 AM

Presumably Lucene got that "no space" exception and closed the writer, storing the exception. Then every time it tried to write to the index, an AlreadyClosedException was thrown, wrapping the stored "no space" exception. I restarted Gerrit, that seems to have fixed it. Gate jobs are running now on my core change.

IKhitron added a comment.Apr 13 2026, 6:15 AM

Does this mean I can try to recheck once again now?

tstarling added a comment.Apr 13 2026, 6:51 AM
In T423027#11812690, @IKhitron wrote:

Does this mean I can try to recheck once again now?

Yes.

Jelto mentioned this in T423034: ProbeDown (gerrit2003:443).Apr 13 2026, 7:13 AM
Jelto claimed this task.Apr 13 2026, 9:12 AM
Jelto subscribed.

Thank you to everyone who troubleshooted and mitigated this issue on a weekend.

<TimStarling> on gerrit2003 restarted gerrit T423027

I can confirm CI job metrics look better after the restart: https://grafana.wikimedia.org/d/d4bf3e68-9396-4173-a59c-59e398172ba0/zuul-prometheus?orgId=1&from=now-4d&to=now&timezone=utc&viewPanel=panel-8 for example.

I did a bit more digging and maybe found some related traffic which might correlate with the disk issues https://superset.wikimedia.org/superset/dashboard/p/odArMxApBXl/. I created a throttling action but the traffic has stopped at the moment.

I'll also compile a list of incident-followups in the task description (multiple people already reached out to me for follow ups) and I'll tag already existing tasks accordingly.

Jelto updated the task description. (Show Details)Apr 13 2026, 9:20 AM
Jelto mentioned this in T333143: Move Gerrit data out of root partition.Apr 13 2026, 9:27 AM
IKhitron added a comment.Apr 13 2026, 9:31 AM
In T423027#11812709, @tstarling wrote:
In T423027#11812690, @IKhitron wrote:

Does this mean I can try to recheck once again now?

Yes.

Thank you. Approving it works.

A_smart_kitten removed a subtask: T423035: Enable paging for Gerrit (was: Gerrit outage didn't page until 4.5 hours after the first alert).Apr 13 2026, 9:50 AM
Jelto mentioned this in T423123: Alert when Gerrit CI (Zuul, Jenkins, Gearman) is down/stuck.Apr 13 2026, 12:37 PM
Jelto updated the task description. (Show Details)
Jelto mentioned this in T422954: SystemdUnitFailed (prometheus-ethtool-exporter.service on gerrit2003:9100).Apr 13 2026, 1:26 PM
Jelto moved this task from Incoming to Work in Progress on the Collaboration-Services board.Apr 13 2026, 3:55 PM
Dzahn subscribed.EditedApr 13 2026, 9:39 PM

We had this existing old ticket to move all Gerrit data out of the root partition; where a huge chunk of LFS data had already been moved but not all parts of the Gerrit Site Path (/var/lib/gerrit); which included this cache data.

I updated this over at T333143#11816922 with the current servers and sizes as of April 2026.

Also left a rough suggestion for a run book how to move the remaining data into /srv as well (T333143#11817082).

gerritbot added a comment.Apr 14 2026, 7:13 AM

Change #1270774 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: migrate gerrit_site away from root partition

https://gerrit.wikimedia.org/r/1270774

gerritbot added a project: Patch-For-Review.Apr 14 2026, 7:13 AM
Jelto created subtask T423601: Update and improve operation runbooks and documentation for Gerrit.Apr 16 2026, 2:19 PM
Jelto mentioned this in T423601: Update and improve operation runbooks and documentation for Gerrit.
Jelto updated the task description. (Show Details)
A_smart_kitten removed a subtask: T423601: Update and improve operation runbooks and documentation for Gerrit.Apr 16 2026, 4:36 PM
ABran-WMF added subtasks: T423035: Enable paging for Gerrit (was: Gerrit outage didn't page until 4.5 hours after the first alert), T423123: Alert when Gerrit CI (Zuul, Jenkins, Gearman) is down/stuck, T423601: Update and improve operation runbooks and documentation for Gerrit.Apr 17 2026, 9:02 AM
ABran-WMF added a subtask: T333143: Move Gerrit data out of root partition.
gerritbot added a comment.Apr 23 2026, 8:10 AM

Change #1270774 merged by Jelto:

[operations/puppet@production] gerrit: migrate gerrit_site away from root partition

https://gerrit.wikimedia.org/r/1270774

Maintenance_bot removed a project: Patch-For-Review.Apr 23 2026, 8:31 AM
ABran-WMF mentioned this in T424196: Document how to restart CI when Gerrit had a production issue.Apr 23 2026, 9:30 AM
Jelto closed this task as Resolved.Apr 23 2026, 2:32 PM
Jelto added a subscriber: ABran-WMF.

I'll resolve this task. Sub-tasks have been open for the follow up action items. The successful migration in T333143: Move Gerrit data out of root partition will prevent the root disk from filling up again with Gerrit cache files. We are also looking into improving the existing documentation and alerting (thanks @ABran-WMF ).

Thanks again to the SREs on-call for mitigating this incident on a weekend!

ABran-WMF closed subtask T423601: Update and improve operation runbooks and documentation for Gerrit as Resolved.Apr 27 2026, 1:33 PM
ABran-WMF changed the status of subtask T423035: Enable paging for Gerrit (was: Gerrit outage didn't page until 4.5 hours after the first alert) from Open to In Progress.Apr 28 2026, 6:57 AM
ABran-WMF closed subtask T423035: Enable paging for Gerrit (was: Gerrit outage didn't page until 4.5 hours after the first alert) as Resolved.Apr 28 2026, 9:03 AM
Jelto closed subtask T333143: Move Gerrit data out of root partition as Resolved.Apr 30 2026, 8:05 AM
MLechvien-WMF moved this task from Active investigation to Complete on the Wikimedia-Incident board.Jun 8 2026, 12:51 PM
MLechvien-WMF added a project: Incident Severity 3.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits