Page MenuHomePhabricator
Create Task
Maniphest T270472

Gerrit errors on change #650396, missing blob
Closed, ResolvedPublic
Actions

Description

Loading https://gerrit.wikimedia.org/r/c/mediawiki/core/+/650396/1 pops up

An error occurred

Error 500 (Server Error): Internal server error

Endpoint: /changes/*~*/revisions/*/files

Looking in logstash it seems to be Error in GET /r/changes/mediawiki%2Fcore~650396/revisions/1/files: missing_object [CONTEXT project="mediawiki/core" ]. The exception message is java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing blob be668dfac285b80062cf4a2fb9ffd9e7489fe966

Details

ProjectBranchLines +/-Subject
mediawiki/corefundraising/REL1_35+1 K -786(DO NOT SUBMIT) Merge commit '16ec13f4f53' into fundraising/REL1_35
Customize query in gerrit

Related Objects
Search...

Event Timeline

Legoktm created this task.Dec 18 2020, 6:12 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 18 2020, 6:12 AM
Legoktm renamed this task from Gerrit errors on change #650396 "missing_object" to Gerrit errors on change #650396, missing blob.Dec 18 2020, 6:14 AM
Legoktm updated the task description. (Show Details)
Ejegg added a comment.Dec 18 2020, 2:53 PM

Redoing the merge locally and then squashing in the vendor / composer.lock update seems to have made gerrit happy with the change. I don't see any difference between PS1 and PS2 besides the vendor pointer and composer.lock.

Ejegg added a comment.Dec 18 2020, 2:59 PM

Oops, I spoke too soon. The first time I looked at PS2 in the gerrit UI it loaded correctly and showed the list of changed files, though zuul didn't seem to pick it up for testing. After giving the patch a C+2 (and commenting 'recheck'), the gerrit UI is back to showing

Error 500 (Server Error): Internal server error

Endpoint: /changes/*~*/revisions/*/files

Ejegg added a comment.Dec 18 2020, 3:14 PM

Seems to be complaining about the same missing blob: https://logstash.wikimedia.org/app/kibana#/dashboard/AW1f-0k0ZKA7RpirlnKV?_g=()

I've tried searching for that blob locally in mediawiki fundraising/REL1_35 and all its submodules using this:

git log --pretty=format:'%T %h %s' | grep be668df

as suggested by https://stackoverflow.com/questions/223678/which-commit-has-this-blob

but it doesn't show up anywhere locally. Any idea how to find a pointer to a blob?

thcipriani added a subscriber: thcipriani.Dec 18 2020, 3:17 PM

I've forgotten where this script comes from, but this is the one I use:

#!/bin/bash

searchfor=$1

startpoints="master"  # branch names or HEAD or whatever

git rev-list $startpoints |
    while read commithash; do
        if git ls-tree -d -r --full-tree $commithash | grep $searchfor; then
            echo " -- found at $commithash"
        fi
    done

This will find the blob if it's pointed to by a tree in your commit somewhere. I wasn't able to find the blob using this method locally.

thcipriani added a comment.Dec 18 2020, 3:26 PM

So I found a *commit* matching that sha1. It's a submodule commit for CategoryTree: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/07df2a859666f4e4e429fd6b9bf5b5ae080d69e6/extensions/CategoryTree

so now the question is why does that revision think that commit is a *blob*?

Ejegg added a comment.Dec 18 2020, 3:32 PM

Thanks for the sleuthing @thcipriani !

So CategoryTree is included in the submodules for base REL1_35, but we removed it after forking off fundraising/REL1_35. In the 1.35.1 merge from REL1_35 to fundraising/REL1_35, I had to use git mergetool to resolve some conflicts on submodule pointers where the fundraising branch (local) had the submodule removed. CategoryTree was one of those.

thcipriani added a comment.Dec 18 2020, 4:08 PM

I do see it getting removed: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/619814

It seems like that should cause a merge conflict so having to resolve on merging in updates from REL1_35 is normal since that submodule was updated since you removed it. Not sure what could cause gerrit to get confused like this.

I tried to recreate the general case to see if this was a gerrit bug using these steps:

  1. Make branch "main" on a repo. In branch "main" have a submodule "sub" pointing to commit xxx
  2. Make new branch "foo" from branch "main"
  3. Delete submodule "sub" from branch "foo"
  4. Update submodule to yyy on branch "main"
  5. Update README on branch "main"
  6. Merge branch "main" into branch "foo"
  7. Resolve conflicts locally (git rm sub)
  8. push merge commit for review

that worked as normal. Something else is strange about this merge or its parents...

thcipriani edited projects, added Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)); removed Release-Engineering-Team.Dec 18 2020, 4:09 PM
thcipriani moved this task from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)) board.
Ejegg mentioned this in T270507: Apply Mediawiki 1.35.1 update to payments-wiki.Dec 18 2020, 4:33 PM
Ejegg added a comment.Dec 18 2020, 4:54 PM

@thcipriani in your test case does the foo branch still have a 'sub' section in its .gitmodules?

In looking for a workaround, we found that accepting the 'remote' version of the conflicting submodules and also restoring the deleted sections from .gitmodules was enough to make gerrit load it happily: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/650524/3

We needed to make a follow-on patch to remove those submodules again, but we are able to end up in the desired state.

Ejegg added a comment.Dec 18 2020, 5:19 PM

<Charlie Brown aaaaauuuuugh... > now reloading that workaround patch in gerrit is giving me the 500 error again. I swear it loaded the changed file list happily the first time I looked at it.

thcipriani added a comment.Dec 18 2020, 6:49 PM

Interestingly if you load: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/650524 it's fine, but https://gerrit.wikimedia.org/r/c/mediawiki/core/+/650524/3 shows a 500 error.

The error is from an ajax request to: https://gerrit.wikimedia.org/r/changes/mediawiki%2Fcore~650524/revisions/3/files

The parent revision does this request instead: https://gerrit.wikimedia.org/r/changes/mediawiki%2Fcore~650524/revisions/3/files?parent=1

This is maybe a UI problem in gerrit...

thcipriani added a comment.EditedDec 18 2020, 6:57 PM
In T270472#6702234, @thcipriani wrote:

This is maybe a UI problem in gerrit...

I was able to recreate this using the example from

In T270472#6701803, @thcipriani wrote:

I tried to recreate the general case to see if this was a gerrit bug using these steps:

  1. Make branch "main" on a repo. In branch "main" have a submodule "sub" pointing to commit xxx
  2. Make new branch "foo" from branch "main"
  3. Delete submodule "sub" from branch "foo"
  4. Update submodule to yyy on branch "main"
  5. Update README on branch "main"
  6. Merge branch "main" into branch "foo"
  7. Resolve conflicts locally (git rm sub)
  8. push merge commit for review

Visiting an the change was fine. Visiting an individual patchset threw a 500 error:

2020-12-18T11 53 03-07 00.gif (661×1 px, 316 KB)

This might be an upstream bug.

Stacktrace

[2020-12-18T11:52:36.131-0700] [HTTP GET /changes/T270472~21/revisions/1/files (thcipriani from [0:0:0:0:0:0:0:1])] WARN  com.google.gerrit.server.patch.PatchListLoader : Error computing PatchListKey[BASE..9751bd6e1b2f720ab2575fc35fe1160cc652243f IGNORE_NONE] [CONTEXT project="T270472" ]
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 3bfb66cbb575d0f9ecbdc18aa7dc381ec0bc87c1
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:564)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:525)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:102)
        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:237)
        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2313)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2279)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
        at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:132)
        at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:105)
        at com.google.gerrit.server.change.FileInfoJson.toFileInfoMap(FileInfoJson.java:74)
        at com.google.gerrit.server.change.FileInfoJson.toFileInfoMap(FileInfoJson.java:67)
        at com.google.gerrit.server.change.FileInfoJson.toFileInfoMap(FileInfoJson.java:56)
        at com.google.gerrit.server.change.FileInfoJson.toFileInfoMap(FileInfoJson.java:49)
        at com.google.gerrit.server.restapi.change.Files$ListFiles.apply(Files.java:186)
        at com.google.gerrit.server.restapi.change.Files$ListFiles.apply(Files.java:100)
        at com.google.gerrit.httpd.restapi.RestApiServlet.lambda$invokeRestReadViewWithRetry$6(RestApiServlet.java:801)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.google.gerrit.server.update.RetryHelper.executeWithTimeoutCount(RetryHelper.java:561)
        at com.google.gerrit.server.update.RetryHelper.execute(RetryHelper.java:504)
        at com.google.gerrit.server.update.RetryableAction.call(RetryableAction.java:172)
        at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestEndpointWithRetry(RestApiServlet.java:892)
        at com.google.gerrit.httpd.restapi.RestApiServlet.invokeRestReadViewWithRetry(RestApiServlet.java:796)
        at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:507)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:386)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139)
        at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:56)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:572)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob 3bfb66cbb575d0f9ecbdc18aa7dc381ec0bc87c1
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
        at org.eclipse.jgit.lib.ObjectReader$Filter.open(ObjectReader.java:515)
        at org.eclipse.jgit.merge.ResolveMerger.getRawText(ResolveMerger.java:1063)
        at org.eclipse.jgit.merge.ResolveMerger.contentMerge(ResolveMerger.java:850)
        at org.eclipse.jgit.merge.ResolveMerger.processEntry(ResolveMerger.java:789)
        at org.eclipse.jgit.merge.ResolveMerger.mergeTreeWalk(ResolveMerger.java:1310)
        at org.eclipse.jgit.merge.ResolveMerger.mergeTrees(ResolveMerger.java:1259)
        at org.eclipse.jgit.merge.ResolveMerger.mergeImpl(ResolveMerger.java:357)
        at org.eclipse.jgit.merge.Merger.merge(Merger.java:233)
        at org.eclipse.jgit.merge.Merger.merge(Merger.java:186)
        at org.eclipse.jgit.merge.ThreeWayMerger.merge(ThreeWayMerger.java:96)
        at com.google.gerrit.server.patch.AutoMerger.createAutoMergeCommit(AutoMerger.java:161)
        at com.google.gerrit.server.patch.AutoMerger.lambda$merge$0(AutoMerger.java:109)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.google.gerrit.server.update.RetryHelper.executeWithTimeoutCount(RetryHelper.java:561)
        at com.google.gerrit.server.update.RetryHelper.execute(RetryHelper.java:504)
        at com.google.gerrit.server.update.RetryableAction.call(RetryableAction.java:172)
        at com.google.gerrit.server.patch.AutoMerger.merge(AutoMerger.java:110)
        at com.google.gerrit.server.patch.PatchListLoader.aFor(PatchListLoader.java:569)
        at com.google.gerrit.server.patch.PatchListLoader.readPatchList(PatchListLoader.java:161)
        at com.google.gerrit.server.patch.PatchListLoader.call(PatchListLoader.java:129)
        at com.google.gerrit.server.patch.PatchListLoader.call(PatchListLoader.java:79)
        at com.google.gerrit.server.cache.h2.H2CacheImpl.lambda$get$1(H2CacheImpl.java:142)
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
        ... 79 more
thcipriani edited projects, added Release-Engineering-Team-TODO; removed Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)).Dec 18 2020, 8:33 PM
thcipriani moved this task from Should be empty (use Release-Engineering-Team) to Watching/External on the Release-Engineering-Team-TODO board.Dec 18 2020, 8:48 PM

Filed upstream as: https://bugs.chromium.org/p/gerrit/issues/detail?id=13845

There is a (possibly related) issue of zuul refusing to test this merge commit: T270524: Zuul not running tests for gerrit change 650524

thcipriani triaged this task as Medium priority.Dec 18 2020, 8:48 PM
thcipriani mentioned this in P13606 (An Untitled Masterwork).Dec 18 2020, 9:07 PM
Ejegg added a comment.Dec 21 2020, 5:15 PM
In T270472#6702548, @thcipriani wrote:

Filed upstream as: https://bugs.chromium.org/p/gerrit/issues/detail?id=13845

There is a (possibly related) issue of zuul refusing to test this merge commit: T270524: Zuul not running tests for gerrit change 650524

Probably also related, the wikibugs IRC bot didn't announce anything about this change in the channels that it usually announces changes for this branch.

thcipriani added a comment.Dec 21 2020, 5:20 PM
In T270472#6706086, @Ejegg wrote:
In T270472#6702548, @thcipriani wrote:

Filed upstream as: https://bugs.chromium.org/p/gerrit/issues/detail?id=13845

There is a (possibly related) issue of zuul refusing to test this merge commit: T270524: Zuul not running tests for gerrit change 650524

Probably also related, the wikibugs IRC bot didn't announce anything about this change in the channels that it usually announces changes for this branch.

If it works of gerrit stream events, this seems likely. In my local testing this is fixed in Gerrit 3.3 which we're planning for early 2021.

hashar mentioned this in T270524: Zuul not running tests for gerrit change 650524.Dec 22 2020, 10:17 AM
hashar merged a task: T270524: Zuul not running tests for gerrit change 650524.
hashar added a subscriber: hashar.

From T270524#6702561 the Gerrit event lacks the comment-added message because the reference can not be seen.

hashar added a parent task: T262241: Upgrade Gerrit to 3.3.Dec 22 2020, 10:20 AM
thcipriani edited projects, added Release-Engineering-Team (Radar); removed Release-Engineering-Team-TODO.Apr 20 2021, 3:33 AM
thcipriani moved this task from Limbo to Watching/External on the Release-Engineering-Team (Radar) board.Apr 20 2021, 3:34 AM
Paladox added a subscriber: Paladox.Aug 4 2021, 1:12 AM

Seems this works now? At least I don't get the error.

Aklapper closed this task as Resolved.Sep 1 2021, 3:08 PM

According to T270472#6706141 fixed with the 3.3 update in T262241: Upgrade Gerrit to 3.3, plus cannot reproduce either with the given URLs.

hashar edited projects, added Gerrit (Gerrit 3.3); removed Gerrit.Jan 3 2022, 10:19 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL