Page MenuHomePhabricator
Create Task
Maniphest T287122

resolve gerrit.config disprepancy between managed config and gerrit init
Closed, ResolvedPublic
Actions

Description

When doing Gerrit deployment we extract bundled plugins with: java -jar bin/gerrit.war init --batch --install-all-plugins which actually runs the whole initialization process. One side effect is it normalizes the gerrit.config which is later overwritten by the Puppet one.

A capture of the differences:

gerrit2@gerrit2001:~/review_site/etc$ diff gerrit.config.before-gerrit-init gerrit.config
67c67
<     link = "/q/$1"
---
>     link = /q/$1
76c76
<     html = "$1<a href=\"/q/$2\">$2</a>"
---
>     html = $1<a href=\"/q/$2\">$2</a>
110,111c110,111
<     javaOptions = "-XX:+UseG1GC"
<     javaOptions = "-Xmx32g -Xms32g"
---
>     javaOptions = -XX:+UseG1GC
>     javaOptions = -Xmx32g -Xms32g
114,119c114,119
<     javaOptions = "-XX:+UnlockExperimentalVMOptions"
<     javaOptions = "-XX:G1NewSizePercent=15"
<     javaOptions = "-XX:+UseStringDeduplication"
<     javaOptions = "-XX:+HeapDumpOnOutOfMemoryError"
<     javaOptions = "-XX:+ExitOnOutOfMemoryError"
<     javaOptions = "-XX:HeapDumpPath=/srv/gerrit"
---
>     javaOptions = -XX:+UnlockExperimentalVMOptions
>     javaOptions = -XX:G1NewSizePercent=15
>     javaOptions = -XX:+UseStringDeduplication
>     javaOptions = -XX:+HeapDumpOnOutOfMemoryError
>     javaOptions = -XX:+ExitOnOutOfMemoryError
>     javaOptions = -XX:HeapDumpPath=/srv/gerrit
254d253
<     smtpEncryption = none
257c256
<     listenAddress = 208.80.153.107:29418
---
>     listenAddress = [2620:0:860:4:208:80:153:107]:29418

The last one is sshd.listenAddress. It used to be set to gerrit.wikimedia.org probably to workaround the faulty detection by gerrit init.

The configuration should not vary.

Details

SubjectRepoBranchLines +/-
gerrit: explicitly set `sshd.listenAddress`operations/puppetproduction+1 -0
gerrit: listen on all address with iptables ruleoperations/puppetproduction+3 -6
gerrit: remove unused container.javaOptions valuesoperations/puppetproduction+9 -8
gerrit: remove unused settings from [container]operations/puppetproduction+0 -5
Revert "gerrit: daemon option in gerrit.config"operations/puppetproduction+2 -2
gerrit: config values do not need double quotesoperations/puppetproduction+3 -3
gerrit: remove SMTP encryption optionoperations/puppetproduction+0 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Jul 21 2021, 9:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2021, 9:02 PM
gerritbot added a comment.Jul 21 2021, 9:05 PM

Change 706042 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: config values do not need double quotes

https://gerrit.wikimedia.org/r/706042

gerritbot added a project: Patch-For-Review.Jul 21 2021, 9:05 PM
brennen renamed this task from resolve gerrit.config dispredancy between managed config and gerrit init to resolve gerrit.config disprepancy between managed config and gerrit init.Jul 21 2021, 9:10 PM
gerritbot added a comment.Jul 21 2021, 9:10 PM

Change 706043 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: remove SMTP encryption option

https://gerrit.wikimedia.org/r/706043

hashar added a subscriber: Dzahn.Jul 21 2021, 9:13 PM

For sshd.listenAddress, I am afraid we have to dig into Gerrit core to find out what kind of logic it uses. @Dzahn suggested that we used a fqdn as the first listenAddress to work around gerrit init to insert a dupe ipv6. But that was a fragile/undocumented hack.

I think I might switch to have Gerrit to listen on all address then add some firewall rules to only allow connections from the service IPs.

hashar mentioned this in T278990: Upgrade Gerrit to 3.2.11.Jul 21 2021, 10:38 PM
gerritbot added a comment.Jul 21 2021, 10:38 PM

Change 706049 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: listen on all address with iptables rule

https://gerrit.wikimedia.org/r/706049

gerritbot added a comment.Jul 23 2021, 10:17 AM

Change 706042 merged by Dzahn:

[operations/puppet@production] gerrit: config values do not need double quotes

https://gerrit.wikimedia.org/r/706042

gerritbot added a comment.Jul 23 2021, 11:21 AM

Change 706043 merged by Dzahn:

[operations/puppet@production] gerrit: remove SMTP encryption option

https://gerrit.wikimedia.org/r/706043

dancy updated the task description. (Show Details)Jul 23 2021, 3:21 PM
thcipriani subscribed.Jul 23 2021, 4:09 PM

After doing some grepping, it seems like the current on-disk config matches except for the listenAddress. Does gerrit init get rid of the ip4 listening address?

hashar moved this task from INBOX to Doing on the Release-Engineering-Team board.Jul 25 2021, 9:05 AM
hashar edited projects, added Release-Engineering-Team (Doing); removed Release-Engineering-Team.
hashar added a subtask: T287360: gerrit.config values having # or ; should be double quoted.Jul 26 2021, 11:06 AM
gerritbot added a comment.Jul 26 2021, 12:30 PM

Change 708102 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] Revert \"gerrit: daemon option in gerrit.config\"

https://gerrit.wikimedia.org/r/708102

gerritbot added a comment.Jul 26 2021, 12:30 PM

Change 708103 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: remove unused settings from [container]

https://gerrit.wikimedia.org/r/708103

gerritbot added a comment.Jul 26 2021, 12:30 PM

Change 708104 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: remove unused container.javaOptions values

https://gerrit.wikimedia.org/r/708104

gerritbot added a comment.Jul 26 2021, 3:01 PM

Change 708102 merged by Jbond:

[operations/puppet@production] Revert \"gerrit: daemon option in gerrit.config\"

https://gerrit.wikimedia.org/r/708102

Stashbot added a comment.Jul 26 2021, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-26T15:29:18Z] <hashar> Restarted gerrit replica on gerrit2001.wikimedia.org # T287122

gerritbot added a comment.Jul 28 2021, 12:05 PM

Change 708103 merged by Dzahn:

[operations/puppet@production] gerrit: remove unused settings from [container]

https://gerrit.wikimedia.org/r/708103

gerritbot added a comment.Jul 28 2021, 12:08 PM

Change 708104 merged by Dzahn:

[operations/puppet@production] gerrit: remove unused container.javaOptions values

https://gerrit.wikimedia.org/r/708104

hashar closed subtask T287360: gerrit.config values having # or ; should be double quoted as Resolved.Aug 2 2021, 10:00 AM
gerritbot added a comment.Aug 2 2021, 12:36 PM

Change 706049 merged by Dzahn:

[operations/puppet@production] gerrit: listen on all address with iptables rule

https://gerrit.wikimedia.org/r/706049

hashar added a comment.Aug 2 2021, 1:10 PM

After the last puppet change got deployed, on gerrit2001 I have:

  • stopped Gerrit
  • java -jar bin/gerrit.war init --batch
  • run puppet

There are still two differences:

Notice: /Stage[main]/Gerrit::Jetty/File[/var/lib/gerrit2/review_site/etc/gerrit.config]/mode: mode changed '0644' to '0444' (corrective)

@@ -251,7 +251,6 @@
     mac = -hmac-md5-96
     threads = 28
     batchThreads = 4
-	listenAddress = *:29418
 [theme]
     backgroundColor = fff
     topMenuColor = fff
gerritbot added a comment.Aug 2 2021, 1:14 PM

Change 709469 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: explicitly set `sshd.listenAddress`

https://gerrit.wikimedia.org/r/709469

gerritbot added a comment.Aug 2 2021, 1:19 PM

Change 709469 merged by Dzahn:

[operations/puppet@production] gerrit: explicitly set `sshd.listenAddress`

https://gerrit.wikimedia.org/r/709469

hashar closed this task as Resolved.Aug 2 2021, 1:22 PM
hashar claimed this task.

Solved and verified with @Dzahn! Danke!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL