A manual rebase didn't work. Ideally what I want is to find where that missing object is (in someone's clone, on github, etc) then push it back into the repo where it belongs.

Okay, how can we find it? Should we just search our local repos logs for it? And if we find it, what can we do to fix it? How can we push it to the repo?

I've had another look at the changes that can't be merged. It looks like the issue occurs on BlueSpiceExtensions and BlueSpiceFoundation repo, only on REL1_27. So it's two object ids that are missing.

• BlueSpiceFoundation misses 9220f9faf8bb3edc9576752de78a2afdbcf9932e, which can be found here https://github.com/wikimedia/mediawiki-extensions-BlueSpiceFoundation/commit/9220f9faf8bb3edc9576752de78a2afdbcf9932e
• BlueSpiceExtensions misses a177cdabc7e83c30b71786f19e42055db873cfda which can be found here https://github.com/wikimedia/mediawiki-extensions-BlueSpiceExtensions/commit/a177cdabc7e83c30b71786f19e42055db873cfda

What do I need to do to fix the issue?

@Osnard Hi, I'm wondering in .git/object/ is there an object with rEBSF9220f9faf8bb or rEBSEa177cdabc7e8

should be like git/object/<num>/rEBSF9220f9faf8bb

should be like git/object/<num>/rEBSEa177cdabc7e8

@Paladox I did not find such files within my local clone

Ok, I wonder if we can create an object?

I've no idea 😄

The only thing I see we could try is to git mirror everything from GitHub to gerrit to see if that will remove the broken object.

Or do we have a backup of the repo's @demon ?

If it helps, we can backup the outstanding changes, remove REL1_27 and recreate it...

@Osnard that may help, could you do a backup please and wait for @demon to agree as it is a 50 50 chance it could make things worse or fix it please.

Okay, I've created a backup of the changes. Waiting for you or @demon

Thanks.

The git repositories on the Gerrit server do not have the commits:

$ git -C /srv/gerrit/git/mediawiki/extensions/BlueSpiceFoundation.git \
   show 9220f9faf8bb3edc9576752de78a2afdbcf9932e
fatal: bad object 9220f9faf8bb3edc9576752de78a2afdbcf9932e

$ git -C /srv/gerrit/git/mediawiki/extensions/BlueSpiceExtensions.git \
  show a177cdabc7e83c30b71786f19e42055db873cfda
fatal: bad object a177cdabc7e83c30b71786f19e42055db873cfda

REL1_27 heads for reference:

The BlueSpiceFoundation change that fails to merge https://gerrit.wikimedia.org/r/#/c/327214/ is a direct child of the tip of REL1_27 (0ba042d27).

Adding the repo from github and fetching it:

$ git remote add github https://github.com/wikimedia/mediawiki-extensions-BlueSpiceFoundation
$ git fetch github
From https://github.com/wikimedia/mediawiki-extensions-BlueSpiceFoundation
 * [new branch]      REL1_22    -> github/REL1_22
 * [new branch]      REL1_23    -> github/REL1_23
 * [new branch]      REL1_27    -> github/REL1_27
 * [new branch]      master     -> github/master

HEADS are the same:

$ git branch -vr --list '*/REL1_27'
  github/REL1_27 ba042d2 Ext.view.Table: buffered store issue fixed
  origin/REL1_27 ba042d2 Ext.view.Table: buffered store issue fixed

But there is no such commit of rEBSF9220f9faf8bb referenced. And even if I fetch all references from both repositories, the commit is still not found:

$ git fetch origin '+refs/*:refs/remotes/origin/*'
...
$ git fetch github '+refs/*:refs/remotes/github/*'
...
$ git show 9220f9faf8bb3edc9576752de78a2afdbcf9932e
fatal: bad object 9220f9faf8bb3edc9576752de78a2afdbcf9932e

All the commits from the branch point of master/REL1_27 (git merge-base):

$ git log --oneline $(git merge-base origin/REL1_27 master)..origin/REL1_27 
ba042d2 Ext.view.Table: buffered store issue fixed
06ca499 Installation: Updated links to installation manual
e9de5a2 StringHelper: Added return statement with boolean
cc9ab15 Set default review branch
63c15be BS.store:LocalNamespaces: Fixed 'all' and 'main'
3440075 Added dependency to module "bluespice.extjs"
f30e56a Bugfix for "grouping feature" in gridpanel
bfa4fc3 Non-ASCII characters. Replacing umlauts with transliterated values rather than removing them

The missing object 9220f9faf8bb is a commit by Chad that created the REL1_27 branch and you can see that is no more listed in the branch.

git merge-base origin/REL1_27 master gives me 05bf530871ee9635643e71acc2770b59217137c1. However Chad commit 9220f9faf8bb that created the branch has commit 0bf932b5ccfe95d71712ca5baf19576b941d0e8c

I am 100% sure that is because the REL1_27 branch has been force pushed at some point.

Yep, is there a way we can delete the bad object server side?

And the million dollars question is why Gerrit think that https://gerrit.wikimedia.org/r/#/c/327214/ depends on a no more existing commit. Maybe it still internally think that REL1_27 head is that missing commit, maybe because the internal database or some git part think it is when it got updated via a force push.

At least the Gerrit branch configuration at https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/BlueSpiceFoundation,branches is consistent with the on disk repository:

@Osnard hi did you delete the branch REL1_27 and recreated it from the master branch?

@hashar if we delete REL1_27 and recreated it, will that work?

I created https://bugs.chromium.org/p/gerrit/issues/detail?id=5194 

please feel free to add more info to there as I only gave them details that are described on this task (so not really sure)

@hashar you carn't force push in gerrit, unless the repo has fast forward which this repo doesn't.

tested on gerrit.git.wmflabs.org and I get this rejection error

$ git push origin HEAD:test --force
Total 0 (delta 0), reused 0 (delta 0)
remote: Processing changes: done
To https://gerrit.git.wmflabs.org/r/a/mediawiki/extensions/test
! [rejected] HEAD -> test (non-fast-forward)
error: failed to push some refs to 'https://paladox@gerrit.git.wmflabs.org/r/a/mediawiki/extensions/test'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. Check out this branch and integrate the remote changes
hint: (e.g. 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

In T153079#2895827, @Paladox wrote:

@Osnard hi did you delete the branch REL1_27 and recreated it from the master branch?

@Paladox Yes I did. When MW 1.27 got released I removed the REL1_27 branch because there was no stable release of BlueSpice for 1.27 yet (There should only be branches that we actually support). Later this year, when we finally released the new version of BlueSpice I recreated the REL1_27 branch from master using the gerrit web UI.

oh.

I did that on a test install and could not reproduce. but creating the branch, the create a commit on branch, then delete it and recreate the branch seemed to still work.

so hopefully that will fix it here.

I was not aware that this could cause problems. But why did the issues occure just now? We've been working with the new REL1_27 branch for quite a while now. And I've done this on a lot of repos, not just BlueSpiceFoundation and BlueSpiceExtensions. What do you suggest to solve the issue? Can I do something or is it a task for someone who has access to the git/gerrit infrastructure of WMF?

The problem with jgit should be fixed in gerrit 2.12 just now the reminents of the problem is still there.

In T153079#2898164, @Osnard wrote:

I was not aware that this could cause problems. But why did the issues occure just now? We've been working with the new REL1_27 branch for quite a while now. And I've done this on a lot of repos, not just BlueSpiceFoundation and BlueSpiceExtensions. What do you suggest to solve the issue? Can I do something or is it a task for someone who has access to the git/gerrit infrastructure of WMF?

Per @hashar jgit gc broke this, it's now disabled so it should not happen to any more branches / repos.

@hashar I'm wondering can we delete and recreate the branch? Well create a new branch of REL1_27 then delete that branch and recreate it?

@Osnard we have to identify the issue in Gerrit/Git which is impacting a few other repositories (see sub tasks of T151676). Unfortunately its xmas/new year break so unlikely any progress will be done until January :-/

I might try to investigate the BlueSpiceFoundation git repo on Gerrit. But no promise.

I'm wondering would checking out the commit you have open for review on branch REL1_.27 then set Submit Type: on the repo to fast forward then force merge that commit with --force?

Or what about setting submit type to cherry-pick if that dosent work?

then change the submit type back to what it was if that works?

@hashar thank you very much. My coworkers and me are on holidays too. So no need to hurry. I wish you and @Paladox a pleasent holiday an a happy new year. Thanks for all the help so far.

And you too @Osnard :)

Ok, this is what we can do, we should clone the whole of BlueSpiceFoundation and BlueSpiceExtensions repo's including all refs.

Then we should mirror it all back to gerrit. I tested this on a test repo and I did not get a bad object when I tried merging a change on branch REL1_27 BlueSpiceFoundation

@Osnard could you try pushing straight into REL1_27 branch by checking out the ref you want to merge and then do git push origin REL1_27 --force please?

I believe this problem happened because of a buggy reindex in gerrit 2.13. See T154205

i managed to reproduce data loss in gerrit after restarting it.

(these were merged changed too)

@Osnard I'm wondering did this problem start happening on wednesday the 7th? Since if so this may be a bug in gerrit that wiped data during the upgrade. Did merging work the day before?

@Paladox The last successful merge was on December 9th for BlueSpiceFoundation and on December 8th for BlueSpiceExtensions. This is roughly around 7th, but not exactly. Both commits were initiated on December 8th.

You can see all REL1_27 commits for BlueSpice herEe https://gerrit.wikimedia.org/r/#/q/project:%22%255E.*(BlueSpice%257CBS).*%22+branch:REL1_27

Thank you for the link and info @Mglaser looking at the last successful merge for the affected repo's, it was on the 9th of december 2016 at around 12:47pm utc + 0.

On the same day we had to do some emergency fixes for logins/accounts due to a bug. Which brings us to this https://gerrit.wikimedia.org/r/#/c/326154/ which was merged and later upgraded during the day of the 9th december after your patch was merged.

Ah ha, cherry pick seems to be using parent commit i.e. cherry pick master -> REL1_27 will use master parent. Looking at https://gerrit.wikimedia.org/r/#/c/325921/ that one uses REL1_27 parent commit.

Oh seems that it is the same commit in master + REL1_27, i think the difference is it did not have the cherry pick line in the commit msg.

@Mglaser or @Osnard could you try re c+2 for https://gerrit.wikimedia.org/r/#/c/327214/ please? I think it may work now but not sure.

Oh, i think i know where the problem came from, since when we first upgraded we used gerrit 2.12 to stop it which did not have the bug in the reindex where gerrit 2.12 will wait for the index to finish before stopping in gerrit 2.13 the index is broken where if you stop it, it will just hide all changes until you reindex manually. Luckly upstream have found the problem patch, but also then again i doint really know if it will fix it. Im trying to get it to reproduce on one of my test installs but i carn't. I think a delete of REL1_27 and recreate it may work but what would work is mirroring from GitHub to the repo on gerrit as that should just delete everything.

But also it could have been gc too when it broke things.

Yes this seems more like reindex as it seems that if you merge a change, restart gerrit then look at open changes, it is shown as open but it is actually merged.

@Paladox, you are probably already a step further, but just for your information: CR+2 on https://gerrit.wikimedia.org/r/#/c/327214 did not work. Same error as always. Neither did git push origin HEAD:refs/heads/REL1_27 --force. Got: [remote rejected] HEAD -> REL1_27 (prohibited by Gerrit)

@Osnard this https://gerrit.wikimedia.org/r/#/c/329694/ should fix the prohibited by gerrit error, i clicked the force push option for you. You should v+2 and c+2 and click submit as jenkins won't be able to merge that without failing i think, so if jenkins fails just remove it as a reviewer and you should be able to merge the commit and do the --force push again :)

Try this git push origin HEAD:refs/heads/REL1_27 --force again and see if gerrit will work, it should but not sure if that object error will happen.

It worked :)

Seems to work:

git push origin HEAD:refs/heads/REL1_27 --force 
Total 0 (delta 0), reused 0 (delta 0)
remote: Processing changes: refs: 1, done    
To ssh://rvogel@gerrit.wikimedia.org:29418/mediawiki/extensions/BlueSpiceFoundation
ba042d2..e7454ce  HEAD -> REL1_27

@Osnard https://phabricator.wikimedia.org/rEBSFe7454ce58bf88a27001baac73da94a6919a34d8e

should probably see if you can now c+2 from gerri's ui? Do a rebase on the patches to first :)

@Osnard could you also c+2 and v+2 https://gerrit.wikimedia.org/r/#/c/329695/1 please? Also press the submit button please :)

@demon as @Osnard was able to do a force push and not get the object error then i am thinking this is not gc, but gerri's bug in the reindexer.

I tried to rebase on REL1_27, commit --amend and run review again. But it's telling me that the review is aready closed (which is correct): [remote rejected] HEAD -> refs/publish/REL1_27/327205-REL1_27 (change https://gerrit.wikimedia.org/r/327214 closed)

So we've got a solution for the outstanding changes now. But what about future changes on the REL1_27 branch?

Yep that's correct.

@Osnard what your seeing is a bug in gerrit index see T154205

Changes for that repo will look open (only the changes i think that were done before we last restarted gerrit will be affected).

we will need to wait for a reindex to fix the changes.

anyways it shows as merged here https://gerrit.wikimedia.org/r/#/c/327214/

This part will require https://gerrit.wikimedia.org/r/#/c/329695/

@Osnard try force merging one of the REL1_27 change on BlueSpiceExtensions then try c+2 on some of the changes after first merging one of the changes.

Ie try force merging https://gerrit.wikimedia.org/r/#/c/327458/ then try doing a c+2 on https://gerrit.wikimedia.org/r/#/c/328906/ (no force merging this one) as we want to see weather force merging a change fixes it so we can c+2 changes again on this branch :)

please.

also im not sure what you mean by outstanding changes?

I force pushed 327458 (worked) and then did a c+2 on 328906. Unfortunately 328906 did not get merged automatically. Pressing 'submit' in gerrit UI results in the known error.

With "outstanding changes" I just meant the changes that are now on gerrit. I could force push them all, but obviously this is not the desired workflow.

@Osnard yep, im hopping a reindex will fix this. Since if it was really a missing object then force push should fail too.

@Osnard try c+2 https://gerrit.wikimedia.org/r/#/c/328906/ again as i have rebased it :)

thanks.

@Osnard could you try c+2 on https://gerrit.wikimedia.org/r/#/c/329700/ please? It is a test change i created.

That may also be a bug to, gerri's ui should support force merge too for changes that doint conflict.

I created this https://groups.google.com/forum/#!topic/repo-discuss/KQ6Bgswf1Ls

@Paladox, thanks for your comment on https://gerrit.wikimedia.org/r/#/c/329700/

If I can help you in any way, please let me know. Unfortunately it's already late here and therefore I won't be able to take any actions today anymore. But I'll be available tomorrow again.

ok, your welcome, and thanks for doing some c+2 and some force pushes :)

I believe it's this change https://gerrit.wikimedia.org/r/#/q/a177cdabc7e83c30b71786f19e42055db873cfda

since in mediawiki/extensions the submodule says this change rEBSEa177cdabc7e8 but it doesn't look merged on gerri's ui, but it should, looks like a reindex should fix that.

ah ha, a commit that is in that link above ^^ was the one @demon created which seemed to have broken the whole thing.

https://github.com/wikimedia/mediawiki-extensions-BlueSpiceExtensions/commit/a177cdabc7e83c30b71786f19e42055db873cfda

I wonder what would happen if we force merge https://gerrit.wikimedia.org/r/#/c/326899/ ? since that would re add the broken object.

Change 329708 had a related patch set (by Paladox) published:
Update BlueSpiceExtensions (REL1_27) commit

https://gerrit.wikimedia.org/r/329708

Yayayayay i reproduced it, it is submodules.

@hashar and @demon and @Osnard and @Mglaser it was the submodule.

Should be fixed in https://gerrit.wikimedia.org/r/329708

I was able to reproduce this on a test instance and inserted a false commit id, i.e. a commit that did not exist in the repo. I then got the error that is shown in the description on the patch when i pressed submit.

Change 329708 merged by Mglaser:
Update BlueSpiceExtensions (REL1_27) commit

https://gerrit.wikimedia.org/r/329708

In T153079#2907805, @Paladox wrote:

@hashar and @demon and @Osnard and @Mglaser it was the submodule.

Should be fixed in https://gerrit.wikimedia.org/r/329708

I was able to reproduce this on a test instance and inserted a false commit id, i.e. a commit that did not exist in the repo. I then got the error that is shown in the description on the patch when i pressed submit.

What a nice find, well done. Guess that might well fix other similar issues :}

@hashar could you c+2 https://gerrit.wikimedia.org/r/#/c/329700/ please, to confirm that it fixed it :)

Change 329713 had a related patch set (by Paladox) published:
Update BlueSpiceFoundations commit (REL1_27)

https://gerrit.wikimedia.org/r/329713

Debugging errors like this should be made easier in https://gerrit-review.googlesource.com/#/c/93466/ as it will now point to the submodule :)

Yay it worked https://gerrit.wikimedia.org/r/#/c/327744/3

@Paladox this looks really good. I will now try and merge the outstanding commits. The procedure:

• Rebase
• +2
• wait for merge

Change 329713 merged by Mglaser:
Update BlueSpiceFoundations commit (REL1_27)

https://gerrit.wikimedia.org/r/329713

In T153079#2907918, @Mglaser wrote:

@Paladox this looks really good. I will now try and merge the outstanding commits. The procedure:

• Rebase
• +2
• wait for merge

Thanks, looks like it is all working :)

https://phabricator.wikimedia.org/diffusion/MEXT/repository/REL1_27/

I just cherry-picked a change from master:

https://gerrit.wikimedia.org/r/#/c/329716/

And got it merged! Works like a charm :)

@Paladox: looks like we can mark this task as resolved

@Paladox Thanks for digging into this, and not givig up ;). It really helps us a lot, since REL1_27 is our current stable branch. Thanks to everyone on this thread, you all helped fixing the issue!

Your welcome :)