Page MenuHomePhabricator

git review notes split brain
Closed, ResolvedPublic

Description

We have a few repos where the review notes diverged between cobalt and gerrit1001 during the server migration.

ERROR> human intervention due to split brain: /srv/gerrit/git/wikimedia/fundraising/crm.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/wikimedia/fundraising/tools.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/operations/deployment-charts.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/operations/cookbooks.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/operations/software/netbox-reports.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/wikibase/termbox.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/pywikibot/core.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/analytics/refinery.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/MultimediaViewer.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/AdminLinks.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/SiteMetrics.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/Loops.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/CodeMirror.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WikiLexicalData.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/CodeEditor.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/TemplateData.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WikimediaMaintenance.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/TwoColConflict.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/SecurePoll.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WebAuthn.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/FileExporter.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/GoogleLogin.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WikimediaEvents.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/tools/api-testing.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/skins/Vector.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/vagrant.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/services/kartotherian.git: "refs/notes/review"

These are the only remaining refs that haven't been vetted as "good" after the migration. They are a little trickier to resolve than standard refs because they're git-notes, rather than changes, tags, or branches.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 24 2019, 11:07 PM
thcipriani triaged this task as Medium priority.Oct 24 2019, 11:08 PM

Steps to fix

# Backup repo
$ cp -R /srv/gerrit/git/[repo].git /srv/gerrit/T236443/[repo].git
# Checkout repo
$ git clone /srv/gerrit/git/[repo].git /srv/gerrit/T236443/[repo]-$(date -I)-checkout
$ cd /srv/gerrit/T236443/[repo]-$(date -I)-checkout
# Fetch gerrit1001 notes
$ git fetch origin refs/notes/review:refs/notes/gerrit1001/review
# Fetch cobalt notes
$ git remote add cobalt /srv/gerrit/cobalt/git/[repo].git
$ git fetch cobalt refs/notes/review:refs/notes/cobalt/review
# Compare notes
$ git log --oneline --decorate --graph refs/notes/gerrit1001/review refs/notes/cobalt/review
# Update git username and email
$ git config user.email 'gerrit@wikimedia.org'
$ git config user.name 'Gerrit Code Review'
# checkout gerrit1001 notes
$ git checkout refs/notes/gerrit1001/review
# merge cobalt notes
$ git merge --log -m 'thcipriani: see T236443' refs/notes/cobalt/review
$ git update-ref refs/notes/review HEAD
# Check your work
$ git log --no-merges -1 --notes=review master
$ git log --no-merges -1 --notes=review [some commit from the merge]
# backup the gerrit1001 ref in the repo
$ git -C /srv/gerrit/git/[repo].git update-ref refs/notes/review-bak refs/notes/review
# Push merge to repo
$ git push origin HEAD:refs/notes/review
  • ERROR> human intervention due to split brain: /srv/gerrit/git/wikimedia/fundraising/crm.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/wikimedia/fundraising/tools.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/operations/deployment-charts.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/operations/cookbooks.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/operations/software/netbox-reports.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/wikibase/termbox.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/pywikibot/core.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/analytics/refinery.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/MultimediaViewer.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/AdminLinks.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/SiteMetrics.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/Loops.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/CodeMirror.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WikiLexicalData.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/CodeEditor.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/TemplateData.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WikimediaMaintenance.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/TwoColConflict.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/SecurePoll.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WebAuthn.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/FileExporter.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/GoogleLogin.git: "refs/notes/review"

Left to do

I've been doing some manual checking on these, so I've got a few left that I'll try to get done tomorrow.

ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WikimediaEvents.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/tools/api-testing.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/skins/Vector.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/vagrant.git: "refs/notes/review"
ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/services/kartotherian.git: "refs/notes/review"

thcipriani closed this task as Resolved.Nov 1 2019, 12:19 AM
thcipriani claimed this task.
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/extensions/WikimediaEvents.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/tools/api-testing.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/skins/Vector.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/vagrant.git: "refs/notes/review"
  • ERROR> human intervention due to split brain: /srv/gerrit/git/mediawiki/services/kartotherian.git: "refs/notes/review"

Tomorrow came early! All done!

Well done Tyler thank you :]