I have done:

1 Remove too broad permissions (recommended):

• Steps:
  • Replace the READ permission on refs/* in All-Projects (or any other project) with a READ permission on refs/heads/*
  • Remove the READ permission on refs/groups/* in All-Users

Which has:

• Side effects:
  • Group owners can no longer access the refs of owned groups
  • Refs of own change edits can no longer be accessed
  • Users that have the "Access Database" global capability assigned can no longer access arbitrary refs through Gitiles/GitWeb or the branch REST endpoints
  • Tags that point to commits that are not part of visible branches can no longer be accessed

Tags that point to commits that are not part of visible branches can no longer be accessed

Does this mean legacy git tags where we deleted the REL1_XX branch can no longer be fetched/checked out?

I guess T268053 is another side effect?

In T268009#6627009, @hashar wrote:

Upstream provides a .war file, unfortunately we still build our own Gerrit since we need at least one cherry pick on top of their stable-3.2 branch.

Not sure if we want to, but if we want to run their WAR file, we don't need to rebuild. We can simply add the jsonevent-layout.jar to upstream's WAR file (Similar to adding a new file to an existing .zip file).

In T268009#6628327, @Legoktm wrote:

Tags that point to commits that are not part of visible branches can no longer be accessed

Does this mean legacy git tags where we deleted the REL1_XX branch can no longer be fetched/checked out?

Indeed though in practice we keep the release branch as long as the release is supported. For mediawiki/core one can look at the tags still available using an anonymous request over https: git ls-remote https://gerrit.wikimedia.org/r/mediawiki/core.git refs/tags/* . The tags we are missing are 1.27.6 , 1.24.* and a bunch of older ones.

Status update

I thought the patch would be released today hence why I am still up around. Turns out it will be released tomorrow November 18th.

I have verified my local environment has all the build requirements and managed to build v3.2.3 from scratch and booted it successfully.

I have reviewed the upgraded documentation https://wikitech.wikimedia.org/wiki/Gerrit#Upgrade_procedures it is fairly complicated in the way plugins are managed, but for this issue we only need to rebuild Gerrit itself and that one is straightforward following the wiki doc.

In T268009#6628445, @QChris wrote:

In T268009#6627009, @hashar wrote:

Upstream provides a .war file, unfortunately we still build our own Gerrit since we need at least one cherry pick on top of their stable-3.2 branch.

Not sure if we want to, but if we want to run their WAR file, we don't need to rebuild. We can simply add the jsonevent-layout.jar to upstream's WAR file (Similar to adding a new file to an existing .zip file).

would be nice to run their .war file so we don't have to keep up with bazel/build changes.

I added WEB-INF/lib/jsonevent-layout-1.7.jar and WEB-INF/lib/json-smart-1.1.1.jar to upstream's .war; however, when trying to restart my testing gerrit I hit:

java.lang.NoSuchMethodError: java.nio.ByteBuffer.mark()Ljava/nio/ByteBuffer;
        at org.eclipse.jgit.util.RawParseUtils.decodeNoFallback(RawParseUtils.java:1140)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1106)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1065)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1044)
        at org.eclipse.jgit.storage.file.FileBasedConfig.load(FileBasedConfig.java:142)
        at com.google.gerrit.server.config.GerritServerConfigModule.getSecureStoreFromGerritConfig(GerritServerConfigModule.java:61)
        at com.google.gerrit.server.config.GerritServerConfigModule.getSecureStoreClassName(GerritServerConfigModule.java:37)
        at com.google.gerrit.pgm.util.SiteProgram.getConfiguredSecureStoreClass(SiteProgram.java:163)
        at com.google.gerrit.pgm.util.SiteProgram$1.configure(SiteProgram.java:94)
        at com.google.inject.AbstractModule.configure(AbstractModule.java:61)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:347)
        at com.google.inject.spi.Elements.getElements(Elements.java:104)
        at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:137)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:105)
        at com.google.inject.Guice.createInjector(Guice.java:87)
        at com.google.inject.Guice.createInjector(Guice.java:69)
        at com.google.inject.Guice.createInjector(Guice.java:59)
        at com.google.gerrit.pgm.util.SiteProgram.createDbInjector(SiteProgram.java:127)
        at com.google.gerrit.pgm.Daemon.start(Daemon.java:346)
        at com.google.gerrit.pgm.Daemon.run(Daemon.java:269)
        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:224)
        at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:120)
        at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:65)
        at Main.main(Main.java:28)

Which, I think, means that the upstream war is built with java11. Probably want to only update one thing at a time :\

A pre release of Gerrit has been deployed by some other users. At least one instance noticed issues which require some more fixes. Speaking with upstream, the security release is delayed. Once it goes out, that will be announced on their public mailing list: https://groups.google.com/g/repo-discuss/

The security fixes have been released in Gerrit 3.2.5, the release notes have the full details https://www.gerritcodereview.com/3.2.html#325

The two security bugs have been made public by upstream:

The two breaking changes (ElasticSearch EOL versions no more support and change to --console-log) do not affect us, we use neither.

I will update our wmf/stable-3.2 branch, rebuild Gerrit, publish the resulting new gerrit.war and scap deploy it on both our instances.

I am going to deploy the Gerrit release 3.2.5.

We have deployed Gerrit 3.2.5 and applied the All-Projects and All-Users workaround. This task can now be made public given upstream disclosed the security issues.

An aftermath reported by some persons on IRC is that an URL such as https://gerrit.wikimedia.org/r/123 ends up redirected to an url including :80 which breaks it. T268260 and https://bugs.chromium.org/p/gerrit/issues/detail?id=13705

Gerrit has been upgraded. Next thing is to find out whether we can drop the work around we have applied to All-Projects / All-Users. Notably the READ permission for refs/* was quite helpful.

We also need to ensure refs/meta/config and refs/config/external-ids are properly replicated to the Gerrit replica. The later should not be replicated to Github.

In T268009#6634679, @hashar wrote:

Gerrit has been upgraded. Next thing is to find out whether we can drop the work around we have applied to All-Projects / All-Users. Notably the READ permission for refs/* was quite helpful.

tl;dr: that's fine to drop. I haven't dropped it yet, but will in my AM if it isn't dropped already by then.

Too many details:

My plan of attack here was to determine whether this workaround was ok to drop was to

1. Recreate the issue using the old gerrit version locally
2. Swap out the gerrit war for a war with the patch applied
3. Verify I no longer have the issue

Setup

Setup of a local gerrit dev site is easy:

$ git clone https://gerrit.wikimedia.org/r/operations/software/gerrit
$ git reset --hard HEAD^
$ git fat init && git fat pull
$ java -jar gerrit.war init --batch --dev -d ~/gerrit_testsite
$ cp plugins/gitiles.jar ~/gerrit_testsite/plugins
$ ~/gerrit_testsite/bin/gerrit.sh restart

was enough to get version 3.2.3 running with gitiles locally.

Recreating the issue

On an unpatched gerrit version I was able to leak some content using gerrit's rest api: https://gerrit-review.googlesource.com/Documentation/rest-api-projects.html#get-content

On an unpatched gerrit version I could dump a file if I knew it to exist from refs/meta/external-ids:

$ curl -s localhost:8080/projects/All-Users/branches/refs%2Fmeta%2fexternal-ids/files/2a6f4e470a1b9ef493f4ac83aa9456102a14f5c4/content | base64 -d
[externalId "mailto:john.doe@example.com"]
        accountId = 1000000
        email = john.doe@example.com

Verifying the fix

Updated to the latest gerrit.war locally:

$ git reset --hard origin/deploy/wmf/stable-3.2
$ git fat pull
$ java -jar gerrit.war version
gerrit version 3.2.5-2-gade85f3c32
$ cp gerrit.war ~/gerrit_testsite/bin/gerrit.war
$ ~/gerrit_testsite/bin/gerrit.sh restart

The same rest command from above is now a 404:

$ curl -s localhost:8080/projects/All-Users/branches/refs%2Fmeta%2fexternal-ids/files/2a6f4e470a1b9ef493f4ac83aa9456102a14f5c4/content 
Not found: refs/meta/external-ids

Change 642464 had a related patch set uploaded (by Hashar; owner: Hashar):
[All-Projects@refs/meta/config] Revert mitigations

https://gerrit.wikimedia.org/r/642464

Change 642464 merged by Thcipriani:
[All-Projects@refs/meta/config] Revert mitigations

https://gerrit.wikimedia.org/r/642464