Page MenuHomePhabricator

Gerrit errors on change #650396, missing blob
Closed, ResolvedPublic

Description

Loading https://gerrit.wikimedia.org/r/c/mediawiki/core/+/650396/1 pops up

An error occurred

Error 500 (Server Error): Internal server error

Endpoint: /changes/*~*/revisions/*/files

Looking in logstash it seems to be Error in GET /r/changes/mediawiki%2Fcore~650396/revisions/1/files: missing_object [CONTEXT project="mediawiki/core" ]. The exception message is java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing blob be668dfac285b80062cf4a2fb9ffd9e7489fe966

Event Timeline

Legoktm renamed this task from Gerrit errors on change #650396 "missing_object" to Gerrit errors on change #650396, missing blob.Dec 18 2020, 6:14 AM
Legoktm updated the task description. (Show Details)

Redoing the merge locally and then squashing in the vendor / composer.lock update seems to have made gerrit happy with the change. I don't see any difference between PS1 and PS2 besides the vendor pointer and composer.lock.

Oops, I spoke too soon. The first time I looked at PS2 in the gerrit UI it loaded correctly and showed the list of changed files, though zuul didn't seem to pick it up for testing. After giving the patch a C+2 (and commenting 'recheck'), the gerrit UI is back to showing

Error 500 (Server Error): Internal server error

Endpoint: /changes/*~*/revisions/*/files

Seems to be complaining about the same missing blob: https://logstash.wikimedia.org/app/kibana#/dashboard/AW1f-0k0ZKA7RpirlnKV?_g=()

I've tried searching for that blob locally in mediawiki fundraising/REL1_35 and all its submodules using this:

git log --pretty=format:'%T %h %s' | grep be668df

as suggested by https://stackoverflow.com/questions/223678/which-commit-has-this-blob

but it doesn't show up anywhere locally. Any idea how to find a pointer to a blob?

I've forgotten where this script comes from, but this is the one I use:

#!/bin/bash

searchfor=$1

startpoints="master"  # branch names or HEAD or whatever

git rev-list $startpoints |
    while read commithash; do
        if git ls-tree -d -r --full-tree $commithash | grep $searchfor; then
            echo " -- found at $commithash"
        fi
    done

This will find the blob if it's pointed to by a tree in your commit somewhere. I wasn't able to find the blob using this method locally.

So I found a *commit* matching that sha1. It's a submodule commit for CategoryTree: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/07df2a859666f4e4e429fd6b9bf5b5ae080d69e6/extensions/CategoryTree

so now the question is why does that revision think that commit is a *blob*?

Thanks for the sleuthing @thcipriani !

So CategoryTree is included in the submodules for base REL1_35, but we removed it after forking off fundraising/REL1_35. In the 1.35.1 merge from REL1_35 to fundraising/REL1_35, I had to use git mergetool to resolve some conflicts on submodule pointers where the fundraising branch (local) had the submodule removed. CategoryTree was one of those.

I do see it getting removed: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/619814

It seems like that should cause a merge conflict so having to resolve on merging in updates from REL1_35 is normal since that submodule was updated since you removed it. Not sure what could cause gerrit to get confused like this.

I tried to recreate the general case to see if this was a gerrit bug using these steps:

  1. Make branch "main" on a repo. In branch "main" have a submodule "sub" pointing to commit xxx
  2. Make new branch "foo" from branch "main"
  3. Delete submodule "sub" from branch "foo"
  4. Update submodule to yyy on branch "main"
  5. Update README on branch "main"
  6. Merge branch "main" into branch "foo"
  7. Resolve conflicts locally (git rm sub)
  8. push merge commit for review

that worked as normal. Something else is strange about this merge or its parents...

@thcipriani in your test case does the foo branch still have a 'sub' section in its .gitmodules?

In looking for a workaround, we found that accepting the 'remote' version of the conflicting submodules and also restoring the deleted sections from .gitmodules was enough to make gerrit load it happily: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/650524/3

We needed to make a follow-on patch to remove those submodules again, but we are able to end up in the desired state.

<Charlie Brown aaaaauuuuugh... > now reloading that workaround patch in gerrit is giving me the 500 error again. I swear it loaded the changed file list happily the first time I looked at it.

This is maybe a UI problem in gerrit...

I was able to recreate this using the example from

I tried to recreate the general case to see if this was a gerrit bug using these steps:

  1. Make branch "main" on a repo. In branch "main" have a submodule "sub" pointing to commit xxx
  2. Make new branch "foo" from branch "main"
  3. Delete submodule "sub" from branch "foo"
  4. Update submodule to yyy on branch "main"
  5. Update README on branch "main"
  6. Merge branch "main" into branch "foo"
  7. Resolve conflicts locally (git rm sub)
  8. push merge commit for review

Visiting an the change was fine. Visiting an individual patchset threw a 500 error:

2020-12-18T11 53 03-07 00.gif (661×1 px, 316 KB)

This might be an upstream bug.

Stacktrace

[2020-12-18T11:52:36.131-0700] [HTTP GET /changes/T270472~21/revisions/1/files (thcipriani from [0:0:0:0:0:0:0:1])] WARN  com.google.gerrit.server.patch.PatchListLoader : Error computing PatchListKey[BASE..9751bd6e1b2f720ab2575fc35fe1160cc652243f IGNORE_NONE] [CONTEXT project="T270472" ]
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 3bfb66cbb575d0f9ecbdc18aa7dc381ec0bc87c1
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:564)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:525)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:102)
        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:237)
        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2313)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2279)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
        at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:132)
        at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:105)
        at com.google.gerrit.server.change.FileInfoJson.toFileInfoMap(FileInfoJson.java:74)
        at com.google.gerrit.server.change.FileInfoJson.toFileInfoMap(FileInfoJson.java:67)
        at com.google.gerrit.server.change.FileInfoJson.toFileInfoMap(FileInfoJson.java:56)
        at com.google.gerrit.server.change.FileInfoJson.toFileInfoMap(FileInfoJson.java:49)
        at com.google.gerrit.server.restapi.change.Files$ListFiles.apply(Files.java:186)
        at com.google.gerrit.server.restapi.change.Files$ListFiles.apply(Files.java:100)
        at com.google.gerrit.httpd.restapi.RestApiServlet.lambda$invokeRestReadViewWithRetry$6(RestApiServlet.java:801)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.google.gerrit.server.update.RetryHelper.executeWithTimeoutCount(RetryHelper.java:561)
        at com.google.gerrit.server.update.RetryHelper.execute(RetryHelper.java:504)
        at com.google.gerrit.server.update.RetryableAction.call(RetryableAction.java:172)
        at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestEndpointWithRetry(RestApiServlet.java:892)
        at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestReadViewWithRetry(RestApiServlet.java:796)
        at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:507)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:386)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139)
        at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:56)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:572)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob 3bfb66cbb575d0f9ecbdc18aa7dc381ec0bc87c1
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
        at org.eclipse.jgit.lib.ObjectReader$Filter.open(ObjectReader.java:515)
        at org.eclipse.jgit.merge.ResolveMerger.getRawText(ResolveMerger.java:1063)
        at org.eclipse.jgit.merge.ResolveMerger.contentMerge(ResolveMerger.java:850)
        at org.eclipse.jgit.merge.ResolveMerger.processEntry(ResolveMerger.java:789)
        at org.eclipse.jgit.merge.ResolveMerger.mergeTreeWalk(ResolveMerger.java:1310)
        at org.eclipse.jgit.merge.ResolveMerger.mergeTrees(ResolveMerger.java:1259)
        at org.eclipse.jgit.merge.ResolveMerger.mergeImpl(ResolveMerger.java:357)
        at org.eclipse.jgit.merge.Merger.merge(Merger.java:233)
        at org.eclipse.jgit.merge.Merger.merge(Merger.java:186)
        at org.eclipse.jgit.merge.ThreeWayMerger.merge(ThreeWayMerger.java:96)
        at com.google.gerrit.server.patch.AutoMerger.createAutoMergeCommit(AutoMerger.java:161)
        at com.google.gerrit.server.patch.AutoMerger.lambda$merge$0(AutoMerger.java:109)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.google.gerrit.server.update.RetryHelper.executeWithTimeoutCount(RetryHelper.java:561)
        at com.google.gerrit.server.update.RetryHelper.execute(RetryHelper.java:504)
        at com.google.gerrit.server.update.RetryableAction.call(RetryableAction.java:172)
        at com.google.gerrit.server.patch.AutoMerger.merge(AutoMerger.java:110)
        at com.google.gerrit.server.patch.PatchListLoader.aFor(PatchListLoader.java:569)
        at com.google.gerrit.server.patch.PatchListLoader.readPatchList(PatchListLoader.java:161)
        at com.google.gerrit.server.patch.PatchListLoader.call(PatchListLoader.java:129)
        at com.google.gerrit.server.patch.PatchListLoader.call(PatchListLoader.java:79)
        at com.google.gerrit.server.cache.h2.H2CacheImpl.lambda$get$1(H2CacheImpl.java:142)
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
        ... 79 more
thcipriani triaged this task as Medium priority.Dec 18 2020, 8:48 PM

Filed upstream as: https://bugs.chromium.org/p/gerrit/issues/detail?id=13845

There is a (possibly related) issue of zuul refusing to test this merge commit: T270524: Zuul not running tests for gerrit change 650524

Probably also related, the wikibugs IRC bot didn't announce anything about this change in the channels that it usually announces changes for this branch.

Filed upstream as: https://bugs.chromium.org/p/gerrit/issues/detail?id=13845

There is a (possibly related) issue of zuul refusing to test this merge commit: T270524: Zuul not running tests for gerrit change 650524

Probably also related, the wikibugs IRC bot didn't announce anything about this change in the channels that it usually announces changes for this branch.

If it works of gerrit stream events, this seems likely. In my local testing this is fixed in Gerrit 3.3 which we're planning for early 2021.

hashar added a subscriber: hashar.

From T270524#6702561 the Gerrit event lacks the comment-added message because the reference can not be seen.

Seems this works now? At least I don't get the error.

According to T270472#6706141 fixed with the 3.3 update in T262241: Upgrade Gerrit to 3.3, plus cannot reproduce either with the given URLs.