Page MenuHomePhabricator
Create Task
Maniphest T425722

Pushing to Gerrit returns an error, but succeeds anyway
Closed, DeclinedPublic
Actions

Description

For the past week and change, every time I push a patch to Gerrit, I've gotten the following error:

remote: error: internal error
To ssh://gerrit.wikimedia.org:29418/mediawiki/core
 ! [remote rejected]         HEAD -> refs/for/master%topic=login-with-passkey-reauth (internal error)
error: failed to push some refs to 'ssh://gerrit.wikimedia.org:29418/mediawiki/core'

But the patch then turns out to have been pushed successfully (for example, this was https://gerrit.wikimedia.org/r/c/mediawiki/core/+/472749/11 ).

The same thing happens when I make publish an edit to a patch in the web UI, or edit the commit message: I get an error, but when I refresh the page, there is a new patchset and my changes are in there.

(I have been doing a lot of work on old patches from 2018 lately, maybe that's part of why?)

Event Timeline

Catrope created this task.May 7 2026, 6:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2026, 6:13 PM
Catrope added a comment.May 7 2026, 9:48 PM

Pushing a new change did not produce this error (trace ID 778187236011-bbfaed23). But pushing https://gerrit.wikimedia.org/r/c/mediawiki/core/+/471664/29 and https://gerrit.wikimedia.org/r/c/mediawiki/core/+/471664/30 did (trace IDs 1778190246679-3985a3f5 and 1778190340140-35c006fa).

thcipriani subscribed.May 7 2026, 10:36 PM

Thanks @Catrope there is definitely info on these trace IDs:

thcipriani@gerrit2003:~$ grep 'TRACE_ID="1778190340140-35c006fa' /var/log/gerrit/error_log | wc -l
3274

Lots to sort through, nothing obvious at first glance. It must be something strange about this changeset, since you're the only one seeing it today:

thcipriani@gerrit2003:~$ grep 'error while processing push' /var/log/gerrit/error_log
[2026-05-07T17:13:22.406Z] [SSH git-receive-pack /mediawiki/core (catrope)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push
[2026-05-07T17:24:29.788Z] [SSH git-receive-pack /mediawiki/core (catrope)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push
[2026-05-07T21:44:10.595Z] [SSH git-receive-pack /mediawiki/core (catrope)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push
[2026-05-07T21:45:44.672Z] [SSH git-receive-pack /mediawiki/core (catrope)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push
thcipriani added a comment.May 7 2026, 11:23 PM

huh, so I think the thing for both of those changes is that patchsets are non-sequential. That is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/471664 jumps from 22 to 24. And https://gerrit.wikimedia.org/r/c/mediawiki/core/+/472749 jumps from 8 -> 10.

I actually think this is a bug in upstream:

Caused by: java.lang.NullPointerException: Cannot read field "value" because "anotherString" is null
        at java.base/java.lang.String.compareTo(String.java:2017)
        at org.eclipse.jgit.lib.RefComparator.compareTo(RefComparator.java:58)
        at org.eclipse.jgit.util.RefList.find(RefList.java:143)
        at org.eclipse.jgit.internal.storage.file.RefDirectory.readRef(RefDirectory.java:1112)
        at org.eclipse.jgit.internal.storage.file.RefDirectory.readAndResolve(RefDirectory.java:304)
        at org.eclipse.jgit.internal.storage.file.RefDirectory.exactRef(RefDirectory.java:325)
        at org.eclipse.jgit.lib.Repository.exactRef(Repository.java:1085)
        at com.google.gerrit.server.change.RevisionJson.toRevisionInfo(RevisionJson.java:387)
        at com.google.gerrit.server.change.RevisionJson.getRevisionInfo(RevisionJson.java:190)
        at com.google.gerrit.server.extensions.events.EventUtil.revisionInfo(EventUtil.java:99)
        at com.google.gerrit.server.extensions.events.RevisionCreated.fire(RevisionCreated.java:81)
        at com.google.gerrit.server.git.receive.ReplaceOp.postUpdate(ReplaceOp.java:537)
        at com.google.gerrit.server.update.BatchUpdate.executePostOps(BatchUpdate.java:806)
        at com.google.gerrit.server.update.BatchUpdates.execute(BatchUpdates.java:133)
        at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:373)
        at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:1170)
        at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:1070)
        at com.google.gerrit.server.git.receive.ReceiveCommits.processCommandsUnsafe(ReceiveCommits.java:825)
        at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:699)
        at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$preReceive$2(AsyncReceiveCommits.java:386)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:187)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:174)
        at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:47)
        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:85)
        at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:116)
        ... 8 more
A_smart_kitten subscribed.May 8 2026, 8:10 AM
thcipriani closed this task as Declined.May 8 2026, 10:54 PM

Turns out, not quite a bug upstream, we removed a few revisions from these changes at one point. As a result, valid upstream assumptions are broken. Since we did this via manual git surgery, it's not really an upstream issue.

The only work around is to create a new patchset with these change (or live with false errors).

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits