Page MenuHomePhabricator

Gerrit: remote unpack failed: Missing tree / blob
Closed, ResolvedPublic

Description

I'm getting errors like

error: remote unpack failed: error Missing tree 303f2006e325c9665c23e722fccf5c83f1c97d3b

or

error: remote unpack failed: error Missing blob 4966483667bd342047378b6c9d90f860d8e1ae86

when running git review.

E.g.

$ git review
You are about to submit multiple commits. This is expected if you are
submitting a commit that is dependent on one or more in-review
commits, or if you are submitting multiple self-contained but
dependent changes. Otherwise you should consider squashing your
changes into one commit before submitting (for indivisible changes) or
submitting from separate branches (for independent changes).

The outstanding commits are:

4a2f27198e (HEAD -> review/gerg_tisza/641590) Structure test for Job constructors
4044e7f460 Document Job's required constructor signature

Do you really want to submit the above commits?
Type 'yes' to confirm, other to cancel: yes
error: remote unpack failed: error Missing tree 303f2006e325c9665c23e722fccf5c83f1c97d3b
fatal: Unpack error, check server log
To ssh://gerrit.wikimedia.org:29418/mediawiki/core.git
 ! [remote rejected]       HEAD -> refs/for/master%topic=641590 (n/a (unpacker error))
error: failed to push some refs to 'ssh://tgr@gerrit.wikimedia.org:29418/mediawiki/core.git'

This one was immediately after checking out the patch with git review -d and amending it, so the local and remote repo can't possibly disagree on parentage.
Rebasing helps, but it also makes for ugly diffs, so this is a bit of an annoyance.

(git 2.17.1, git-review 1.27.0)

Event Timeline

hashar subscribed.

I believe the server side stacktrace is:

[2020-11-18 09:23:31,379] [SSH git-receive-pack /mediawiki/core.git (tgr)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user YYY account XXX) during git-receive-pack '/mediawiki/core.git'
com.google.gerrit.sshd.BaseCommand$Failure: fatal: Unpack error, check server log
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:128)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:107)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:72)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:469)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:610)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unpack error on project "mediawiki/core":
  AdvertiseRefsHook: org.eclipse.jgit.transport.AdvertiseRefsHookChain@73471d1eclass org.eclipse.jgit.transport.AdvertiseRefsHookChain

        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:127)
        ... 13 more
Caused by: org.eclipse.jgit.errors.UnpackException: Exception while parsing pack stream
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:2194)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:2120)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:98)
        ... 13 more
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree 303f2006e325c9665c23e722fccf5c83f1c97d3b
        at org.eclipse.jgit.internal.transport.connectivity.FullConnectivityChecker.checkObjects(FullConnectivityChecker.java:196)
        at org.eclipse.jgit.internal.transport.connectivity.FullConnectivityChecker.checkConnectivity(FullConnectivityChecker.java:52)
        at org.eclipse.jgit.transport.ReceivePack.checkConnectivity(ReceivePack.java:1545)
        at org.eclipse.jgit.transport.ReceivePack.receivePackAndCheckConnectivity(ReceivePack.java:1215)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:2189)
        ... 15 more

The devil is finding out what Missing tree 303f2006e325c9665c23e722fccf5c83f1c97d3b is about.

The tree object is on the server:

$ git ls-tree 303f2006e325c9665c23e722fccf5c83f1c97d3b
100644 blob fe1094f512b1e32eb5a3b26e673c33376db9cfe6	HamcrestPHPUnitIntegration.php
100644 blob aeb035f984c2b903ab7d5d39a41b67d0174f016c	LessFileCompilationTest.php
100644 blob 26d52179ec82fec352e7396372677b6d221c30bd	Makefile
...

But I don't think it is attached to any commit, crawling the whole commit log listing the commit abbreviated sha1 (%h) and the attached tree object (%T) yields nothing:

git log --all --pretty=format:"%h %T %d"|grep 303f2006e325c9665c23e722fccf5c83f1c97d3b

The commit has disappeared?

The other object error: remote unpack failed: error Missing blob 4966483667bd342047378b6c9d90f860d8e1ae86 is a blob and it is on the server.

hashar changed the task status from Open to Stalled.Nov 18 2020, 10:19 AM
hashar claimed this task.

The first error occurred at 2020-11-17 15:02:35 UTC. And there is no other match when I crawl all the way up to October 17th.

I am pretty sure it is related to T268009 (private). A fix today should fix it.

Marking as stalled pending resolution of T268009.

FTR, I experienced this error when pushing to core as well, but it was fixed after running git fetch --all. HTH

Me too:

error: remote unpack failed: error Missing tree 42ec2840a99f9e73c57781e9ac77bf7ac153e104
fatal: Unpack error, check server log

On at least https://gerrit.wikimedia.org/r/c/mediawiki/core/+/641522

Re-cloning the repo did not help.

Edit:
git fetch --all does seem to help sometimes, so does rebasing. But the error keeps coming back, on different patches.

The devil is finding out what Missing tree 303f2006e325c9665c23e722fccf5c83f1c97d3b is about.

The tree object is on the server:

$ git ls-tree 303f2006e325c9665c23e722fccf5c83f1c97d3b
100644 blob fe1094f512b1e32eb5a3b26e673c33376db9cfe6	HamcrestPHPUnitIntegration.php
100644 blob aeb035f984c2b903ab7d5d39a41b67d0174f016c	LessFileCompilationTest.php
100644 blob 26d52179ec82fec352e7396372677b6d221c30bd	Makefile
...

But I don't think it is attached to any commit, crawling the whole commit log listing the commit abbreviated sha1 (%h) and the attached tree object (%T) yields nothing:

git log --all --pretty=format:"%h %T %d"|grep 303f2006e325c9665c23e722fccf5c83f1c97d3b

The commit has disappeared?

The other object error: remote unpack failed: error Missing blob 4966483667bd342047378b6c9d90f860d8e1ae86 is a blob and it is on the server.

The tree is still on the server and attached to a commit. You can see the tree in gitiles here: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/3c262bffdf062f6130b217b2ea29710fd317a58f/tests/phpunit/

The first error occurred at 2020-11-17 15:02:35 UTC. And there is no other match when I crawl all the way up to October 17th.

I am pretty sure it is related to T268009 (private). A fix today should fix it.

Marking as stalled pending resolution of T268009.

I agree with this.

I have no idea why git fetch --all would fix it though :-\

@Ottomata had a similar issue. Trying to cherry pick https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/639622 (master branch) to wmf/1.36.0-wmf.18 although a cherry pick to wmf/1.36.0-wmf.16 worked.

Eventually @Urbanecm managed to do the cherry pick ( https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/641768 ).

It might be a weird side effect with git protocol v2. If someone encounter the issue again, it would be nice to try again with the default protocol using:

git -c protocol.version=0 push origin HEAD:refs/for/XXXXX

(replace XXXXX with the target branch). And report back here whether that fixes it.

I did not enable the v2 protocol.

I did not enable the v2 protocol.

Great thank you for confirming. I guess that rules out a protocol issue.

still on hold due to private task T268009

hashar changed the task status from Stalled to Open.Nov 19 2020, 10:04 PM

T268009 is now public. That was a nasty security issue in Gerrit which is addressed by our upgrade to Gerrit version 3.2.5. There was still a couple hits after the upgrade, so the issue is not solved.

The count of all errors we had:

gerrit.log.2020-11-17.gz:23
gerrit.log.2020-11-18.gz:39
gerrit.log.2020-11-19.gz:7
gerrit.log.2020-11-20.gz:5

It disappeared as we have removed the workaround mitigations for T268009.