@Paladox pointed me to this ticket since I reopened T234533#5593695 earlier today.
Feel free to close that one and follow up on this task

Thanks!

More changes have been reported missing T234533#5593695

• https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/540006/

Has been merged, it is in cobalt and show has merged/reveted etc but on gerrit1001 the change is stall to some previous state.

• https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/534389/ reported by @Joe

I can not find any trace of it anywhere on cobalt, not even in refs/notes/review; That has previously been the subject of T234533

Here's another corrupted patch, merged but Gerrit displays as unmerged: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Popups/+/538641/

More reports at: T236135

So for https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/540006/ , I looked at the actual git repositories on the server.

The repository is in /srv/gerrit/git/operations/mediawiki-config.git

On gerrit1001 there is a meta file for this change dated Oct 1 04:55

$ stat refs/changes/06/540006/meta 
  File: refs/changes/06/540006/meta
  Size: 41        	Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 16385659    Links: 1
Access: (0644/-rw-r--r--)  Uid: (  114/ gerrit2)   Gid: (  119/ gerrit2)
Access: 2019-10-21 21:22:49.717939683 +0000
Modify: 2019-10-01 04:55:19.192599056 +0000
Change: 2019-10-02 21:23:59.594267158 +0000
 Birth: -

$ cat refs/changes/06/540006/meta 
bf7bb6baf43261fafe351ff9889232900d6b1616

And obviously that sha1 is the commit pointed to by that reference:

$ git rev-parse refs/changes/06/540006/meta
bf7bb6baf43261fafe351ff9889232900d6b1616

Which is the state of that change at that date and shows up as:

commit bf7bb6baf43261fafe351ff9889232900d6b1616 (refs/changes/06/540006/meta-back)
Author: Marostegui <3731@e9e9afe9-4712-486d-8885-f54b72dd1951>
Date:   Tue Oct 1 04:55:19 2019 +0000

    Update patch set 1
    
    Patch Set 1: Code-Review-2
    
    Do not depool before 10th October, as es1014 is in B1 and that host needs to be depooled the 10th of October for B1 PDU maintenance
    
    Patch-set: 1
    Reviewer: Marostegui <3731@e9e9afe9-4712-486d-8885-f54b72dd1951>
    Label: Code-Review=-2

• where antoine shows magic***

Now if I copy that repo under /tmp and then move that meta file to meta-back.

$ git rev-parse refs/changes/06/540006/meta
32ef7569b9bcb36a86125a942311548b016c239d

New commit! It is magic.

And that one reflects the reality, the change got merged and eventually got reverted which shows up as:

$ git show refs/changes/06/540006/meta
commit 32ef7569b9bcb36a86125a942311548b016c239d (refs/changes/06/540006/meta)
Author: Marostegui <3731@e9e9afe9-4712-486d-8885-f54b72dd1951>
Date:   Fri Oct 4 05:02:37 2019 +0000

    Update patch set 2
    
    Created a revert of this change as I6c5171fb6e470aa5d1f726fd048d9f99a7dc64fa
    
    Patch-set: 2
    Tag: autogenerated:gerrit:revert

And that is the same on cobalt.

But I have deleted the refs/changes/06/540006/meta file so how is that possible? Originally git tracked each references in individual files, that is why the changes references are namespaced using the two last digits of the change number. Eg: change 123 is under refs/changes/23/123 . That is to prevent reaching the limit of directory entries.

Nowadays git would happilly pack the references in a single file: .git/packed-refs. When looking for the sha1 of a given reference, git first look up for a file, else fallback to packed-refs.

My suspicion is that after Gerrit got stopped on cobalt, a rsync has been done that DID NOT DELETE FILES. Thus we carried over a lot of metadata file that got prepopulated as part of the migration preparation?

So to me it looks like we should verify all those meta files, find out whether that there is a more recent sha1 pointed at in packed-refs and if so backup the meta file. And that should fix it.

And obviously on cobalt the file refs/changes/06/540006/meta does not exist. It has been packed in .git/packed-refs.

The devil is that we might have a split brain if a change with a meta file on gerrit1001 has been updated when it also has been updated on cobalt after the inital sync :-\

But I guess we can just rename any old meta file to meta-back, then do a git rev-parse to have git retrieve the sha1 from the packed-refs and compare. If they are we are fine, else something is out of sync. At that point we would have to head to cobalt and verify what was the state there (in the meta file if it exists and packed-refs) then compare.

One can list all the meta files and their sha1 using:

(for x in $(find . -path '*/refs/changes/*/*/meta'); do grep -H . $x; done;)|sort

And diffing gives a bunch of various differences :-\

In T236114#5594704, @hashar wrote:

My suspicion is that after Gerrit got stopped on cobalt, a rsync has been done that DID NOT DELETE FILES. Thus we carried over a lot of metadata file that got prepopulated as part of the migration preparation?

We rsynced at some point in preparation for the move. We stopped gerrit on cobalt. We rsync'd again. We started gerrit on gerrit1001. I didn't do the actual rsyncing; however, I would guess you are right: we didn't use the --delete flag.

We run git gc on all repos every Friday, so any refs that got moved from loose ref files to the packed-refs file may still exist. I think this explanation is pretty plausible.

So to me it looks like we should verify all those meta files, find out whether that there is a more recent sha1 pointed at in packed-refs and if so backup the meta file. And that should fix it.

I went with a terrible script:

Output: