In /var/log/gerrit we have:

The systemd journal shows nothing.

Gerrit has built-in support to write:

httpd_log and error_log do not show up, I guess because we overwrite the default log4j configuration via -Dlog4j.configuration.

I briefly tested log.jsonLogging for the 3.2 upgrade. According to my notes, the format for @timestamp is slightly different (Z as timezone vs. +0000. That's more of a pain on the Java side than one wants to), the @version differs, and IIRC exceptions are slightly different.

But one can surely work around all these with a bit of hacking.

What made me stick with jsonevent-layout is that:

• it was what we had there and the upgrade was big enough already (which is now of course moot :-) )
• logging in Java is a bit messy aleady and Gerrit was not the best citizen here in the past. So I'd not rely on anything that lives inside Gerrit, and instead would go with massaging log messages outside of Gerrit (like jsonevent-layout did and does).

But there is actually nothing requiring us to have the jar in the gerrit war file. One could put the jsonevent-layout jar $SOMEWHERE on the gerrit hosts and then simply add them to classpath (i.e.: in the systemd service file) and it would get picked up accordingly. That would allow to use the upstream vanilla gerrit.war and still keep the massaging of log messages outside of Gerrit.

(Obviously, none of the above are a blocker for using log.jsonLogging nonetheless)

And surely passing -Dlog4j.configuration disables the Gerrit builtin logs which have support for log.jsonLogging :-\

public static boolean shouldConfigure() {
  return Strings.isNullOrEmpty(System.getProperty(LOG4J_CONFIGURATION));
}

Thank you @QChris for your input, it is very helpful to have all the context. My intent for this task is to drop the cherry pick. I will look at shipping the jsonevent-layout jar outside of the gerrit.war :]

In T268020#6627706, @hashar wrote:

Thank you @QChris for your input, it is very helpful to have all the context. My intent for this task is to drop the cherry pick. I will look at shipping the jsonevent-layout jar outside of the gerrit.war :]

jsonevent-layout has its own dependencies, most of those dependencies are inside gerrit already with the exception of json-smart, so we'll need both of those in the class path.

Also, if we start using upstream packages (which I think is probably the purpose of this task) we should move to openjdk-11 per: https://www.gerritcodereview.com/3.2.html#native-packaging

From Luca, we can put the extra .jar in $GERRIT_SITE/lib like we did previously for the mysql connector. Gerrit adds that path to CLASSPATH and it thus discovers jar in there automagically. Or in short: no need to mess with systemd configuration / environment variables.

In T268020#6631592, @hashar wrote:

From Luca, we can put the extra .jar in $GERRIT_SITE/lib like we did previously for the mysql connector. Gerrit adds that path to CLASSPATH and it thus discovers jar in there automagically. Or in short: no need to mess with systemd configuration / environment variables.

I did not test this, so I might be totally off-track here. And maybe that also changed with one of Gerrit's logging backend switches, but loading from $GERRIT_SITE/lib used to happen only after the logging got set up. As most relevant classes still have their logger in a static field, logging would get initialized very early on, before the jar from $GERRIT_SITE/lib got loaded. But then again Flogger is trying to do a few things lazily these days, so that might help or mitigate.

If $GERRIT_SITE/lib works, it's certainly nice. But if it doesn't, I'd not bother much and go for adding the class path in the systemd service file, as WMF has been sunk quite some time with $GERRIT_SITE/lib issues in the past already.

Thank you for mentioning load order might be an issue and actually has hit us in the past. I will give it a try at least for educational purpose.

I have looked at the systemd unit, it sources an environment file which is generated by Puppet. Adding CLASSPATH=/var/lib/gerrit2/review_site/lib/* is thus straightforward.

I will look at fetching the .jar we need and have them included in our deploy/wmf/stable-3.2 branch under a lib directory. Then have Puppet to symlink /var/lib/gerrit2/review_site/lib at /srv/deployment/gerrit/gerrit/lib and I guess we will be set :]