Page MenuHomePhabricator

Failure to push changes for review to Gerrit: "internal server error: Error inserting change/patchset" resulted in a buggy changeset
Closed, ResolvedPublic

Description

I have just received a very weird error message when pushing some changes for review.

F:\mediawiki\extensions\MobileFrontend>git st
On branch T210630
Your branch is ahead of 'gerrit/master' by 5 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

F:\mediawiki\extensions\MobileFrontend>git review -fy
remote:
remote: Processing changes: new: 1 (\)
remote: Processing changes: new: 1, refs: 1 (\)
remote: Processing changes: new: 1, refs: 1, done
remote:
remote: New Changes:
remote:   https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/486407 [WIP] VE: New loading screen design
remote:
To ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend.git
 ! [remote rejected]     HEAD -> refs/for/master%topic=T210630 (internal server error: Error inserting change/patchset)
error: failed to push some refs to 'ssh://matmarex@gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend.git'

The URL https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/486407 shows an "Internal server error". Searching for its Change-Id I0752904b6781091c15c1821c064e4b66ba27e73e does not find it (!), and I do not see the change on my dashboard.

F:\mediawiki\extensions\MobileFrontend>git review -fy
remote:
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done
To ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend.git
 ! [remote rejected]     HEAD -> refs/for/master%topic=T210630 (no new changes)
error: failed to push some refs to 'ssh://matmarex@gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend.git'

Clearly, the change with this Change-Id exists.

F:\mediawiki\extensions\MobileFrontend>git co --amend
[T210630 334db050e] [WIP] VE: New loading screen design
 Date: Thu Jan 24 02:13:59 2019 +0100
 6 files changed, 224 insertions(+), 4 deletions(-)
 create mode 100644 resources/mobile.editor/init.less

F:\mediawiki\extensions\MobileFrontend>git review -fy
remote:
remote: Processing changes: new: 1 (\)
remote: Processing changes: new: 1 (|)
remote: Processing changes: new: 1 (/)
remote: Processing changes: new: 1 (-)
remote: Processing changes: new: 1 (\)
remote: Processing changes: new: 1 (|)
remote: Processing changes: new: 1 (|)
remote: Processing changes: new: 1, done
remote:
remote: New Changes:
remote:   https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/486408 [WIP] VE: New loading screen design
remote:
To ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend.git
 * [new branch]          HEAD -> refs/for/master%topic=T210630
Switched to branch 'master'
Deleted branch 'T210630'

I edited the commit to remove the Change-Id and generate a new one. This time it went through successfully.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 25 2019, 12:39 AM

Requesting the revision I see: P8034

hashar added a subscriber: hashar.Jan 25 2019, 9:58 AM

The first error we have in the error_log for 486407 is the indexer failing to load the change:

[2019-01-25 00:29:32,156] [Index-Interactive-7] ERROR com.google.gerrit.index.Schema : error getting field added of ChangeData{486407}
com.google.gwtorm.server.OrmException: Unable to load change 486407
        at com.google.gerrit.server.query.change.ChangeData.reloadChange(ChangeData.java:570)
        at com.google.gerrit.server.query.change.ChangeData.change(ChangeData.java:557)
        at com.google.gerrit.server.query.change.ChangeData.getDiffSummary(ChangeData.java:491)
        at com.google.gerrit.server.query.change.ChangeData.computeChangedLines(ChangeData.java:514)
        at com.google.gerrit.server.query.change.ChangeData.changedLines(ChangeData.java:526)
        at com.google.gerrit.server.index.change.ChangeField.lambda$static$23(ChangeField.java:463)
        at com.google.gerrit.index.FieldDef.get(FieldDef.java:138)
        at com.google.gerrit.index.Schema$1.apply(Schema.java:185)
        at com.google.gerrit.index.Schema$1.apply(Schema.java:180)
        at com.google.common.collect.Iterators$6.transform(Iterators.java:788)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
        at com.google.common.collect.Iterators$5.computeNext(Iterators.java:638)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
        at com.google.gerrit.lucene.AbstractLuceneIndex.toDocument(AbstractLuceneIndex.java:289)
        at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:207)
        at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:102)
        at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:207)
        at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:409)
        at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:401)
        at com.google.gerrit.server.index.change.ChangeIndexer$AbstractIndexTask.call(ChangeIndexer.java:386)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:558)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.gerrit.server.project.NoSuchChangeException: 486407
        at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:703)
        at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:167)
        at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:141)
        at com.google.gerrit.server.query.change.ChangeData.reloadChange(ChangeData.java:568)
        ... 31 more

The change made it to the repository:

$ git ls-remote https://gerrit.wikimedia.org/r/p/mediawiki/extensions/MobileFrontend.git 'refs/changes/07/486407*'
1442a6900e63ce399b28e47fc38b9fecaa41ac16	refs/changes/07/486407/1
43346674fd8a8a5750831e3d393ebc610595209c	refs/changes/07/486407/meta

Which one can fetch and inspect locally:

$ git fetch origin +refs/changes/07/486407/*:refs/remotes/origin/changes/07/486407/*
remote: Counting objects: 3101, done
remote: Finding sources: 100% (3386/3386)
remote: Getting sizes: 100% (410/410)
remote: Compressing objects: 100% (1484/1484)
remote: Total 3386 (delta 2597), reused 3384 (delta 2596)
Receiving objects: 100% (3386/3386), 3.34 MiB | 934.00 KiB/s, done.
Resolving deltas: 100% (2597/2597), completed with 280 local objects.
From ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend
 * [new ref]             refs/changes/07/486407/1    -> origin/changes/07/486407/1
 * [new ref]             refs/changes/07/486407/meta -> origin/changes/07/486407/meta

$ git show --oneline --stat origin/changes/07/486407/1
1442a6900 [WIP] VE: New loading screen design
 extension.json                                                       |  3 +++
 resources/mobile.editor.overlay/VisualEditorOverlay.js               |  9 +++++++--
 resources/mobile.editor.overlay/editor.ve.less                       | 19 +++++++++++++++++++
 resources/mobile.editor.ve/ve.init.mw.MobileFrontendArticleTarget.js | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 resources/mobile.editor/init.js                                      | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 resources/mobile.editor/init.less                                    | 29 +++++++++++++++++++++++++++++
 6 files changed, 224 insertions(+), 4 deletions(-)

$ git show origin/changes/07/486407/meta
commit 43346674fd8a8a5750831e3d393ebc610595209c
Author: Bartosz Dziewoński <417@e9e9afe9-4712-486d-8885-f54b72dd1951>
Date:   Fri Jan 25 00:29:23 2019 +0000

    Create change
    
    Uploaded patch set 1.
    
    Patch-set: 1
    Change-id: I0752904b6781091c15c1821c064e4b66ba27e73e
    Subject: [WIP] VE: New loading screen design
    Branch: refs/heads/master
    Status: new
    Topic: T210630
    Commit: 1442a6900e63ce399b28e47fc38b9fecaa41ac16
    Tag: autogenerated:gerrit:newPatchSet
    Groups: 32088a966633517392abe70c3c491ef0bc1446cd
    Private: false
    Work-in-progress: false

I tried refreshing the index:

$ gerrit index changes 486407
warning: "486407" no such change
$
$ gerrit index project mediawiki/extensions/MobileFrontend
$

But it can't find the change either. So I have no idea what is happening there.

And eventually I flushed some cache:

gerrit flush-caches --cache changes
gerrit flush-caches --cache change_notes

And the change can now be seen at https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/486407/

I guess the cache got corrupted somehow or there is a race condition between upload/accessing the url/indexing/cache. Who knows really :(

hashar closed this task as Resolved.Feb 7 2019, 12:22 PM
hashar claimed this task.

I am blaming cosmic rays for that one. More seriously, there is no need to identify what is the root cause. I can't find a way to reproduce it.