After looking at the H2 Database source code 'see below) we might be able to set MAX_COMPACT_TIME via a system setting: -Dh2.maxCompactTime. It is not explicitly mentioned in the documentation but the code seems to look up the properties:

/**
 * Get the setting for the given key.
 *
 * @param key the key
 * @param defaultValue the default value
 * @return the setting
 */
protected String get(String key, String defaultValue) {
    StringBuilder buff = new StringBuilder("h2.");
    boolean nextUpper = false;
    for (char c : key.toCharArray()) {
        if (c == '_') {
            nextUpper = true;
        } else {
            // Character.toUpperCase / toLowerCase ignores the locale
            buff.append(nextUpper ? Character.toUpperCase(c) : Character.toLowerCase(c));
            nextUpper = false;
        }
    }
    String sysProperty = buff.toString();
    String v = settings.get(key);
    if (v == null) {
        v = Utils.getProperty(sysProperty, defaultValue);
        settings.put(key, v);
    }
    return v;
}

Indeed I have restarted Zuul after I got ping on IRC about some merge failures occurring. It is indeed the same as T309376.

I have posted to wikitech-l how one can try out the plugin locally: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/3ULF5NPVC4MSVABZBSXAMDODLZUKFXHS/ . That should provide a bit of feedback and bug reporting before deploying it live.

In T321350#8435239, @Nikerabbit wrote:

L10n-bot doesn't do a direct push in Gerrit, so we would need the default l10n-bot(-watcher) permissions and i18n-check on this repo.

We could also look into making it into a direct push, but I think former is preferable.

I gave it a try using a JavaScript plugin that would crawl the DOM for reported message, parse the messages emitted by Zuul and replace the inner HTML by a nicer version. I had several issue cause I could not reliably catch the elements being rendered to hijack the content and I could not find out how to catch an event such as one clicking on a comment to collapse/uncollapse it. Essentially the code was fragile.

(I filed that one before I forget, I haven't looked into upstream issue tracker)

@Jelto thank you for the excellent analysis about monitoring. I will look at integrating that to the deployment runbook we have at https://wikitech.wikimedia.org/wiki/Gerrit/Upgrade#Deploying

Sorry for lack of update. I did dig into the Gerrit cache which are backed up by H2 Database. Some finding are at T323754 and it looks like the easiest path is to empty up the cache and let Gerrit regenerate them. That would include deleting the web session cache which would logout every single users, which I don't think it is a problem granted some announcement is made before the operation. I will look at scheduling that at some point next week.

Daniel and I talked about it this morning.

The reason was that all 3 containers were added in the hosts file with 127.0.0.1. So it tries to ssh to itself and that broke the world. Solved by updating podman compose to a more recent (with a new way to do networking) and installing dnsmasq locally to enable a DNS resolver local to the pod. Another couple fixes needed:

For Gerrit, I have made the announcement on wikitech-l: Gerrit now forbids plain HTTP.

In T225730#8370890, @Krinkle wrote:

@hashar Afaik for PHPUnit we only install (and thus test) dependency extensions, based on the dependency map in CI config. Is this not the case for the Selenium job? If there's a default list applied also, perhaps we can opt-out from that for the Selenium job.

Nice finding @Reedy :)

We have gerrit.serverId set to e9e9afe9-4712-486d-8885-f54b72dd1951 since December 2016 commit 15bc9b80a7a91c54567d3fb63442ac13c955e603.

maxCompactTime
Database setting MAX_COMPACT_TIME (default: 200).

And to get a SQL prompt:

java -cp h2-1.3.176.jar org.h2.tools.Shell -url 'jdbc:h2:file:./git_file_diff;IFEXISTS=TRUE'

I did a few wrong things:

Ideally Zuul would report immediately when it knows that a change is not going to pass. As you found out that is T248531 which probably should be declined given I don't want to make any change to the legacy forked Zuul code were are using.

As a teaser I found a way to load the plugin against our own instance (using incognito mode to be sure and the Chromium extension [[ chrome://extensions/?id=jimgomcnodkialnpmienbomamgomglkd | Gerrit FE Dev Helper ]].

Made against https://gerrit.wikimedia.org/r/c/mediawiki/core/+/859146/4 side loading the plugin https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/859083/4

It is merely a proof of concept:

This script has been used to migrate Gerrit accounts in the gerrit: scheme to be all lower case which was done for T216605.

We have upgraded to Gerrit 3.5.4 (T307334) which upgrades the gitiles plugin which does have @QChris fix.

That has been addressed and the Gerrit 3.5 obsolete @apply statement has been removed.

Still happening on our Gerrit 3.5.4.

After upgrading to Gerrit 3.5.4 (T307334), the triple clicking on the first line of the commit message still does NOT work on Firefox. A workaround is the change title can be copied to the clipboard via a paperclip icon.

Fixed after I have upgraded to Gerrit 3.5.4 this morning (T307334):

We are on Gerrit 3.5.4 there are a few UI changes which would probably trigger some discussions here and there but overall everything else is working as far as I can tell.

I have marked with Sustainability (Incident Followup) and SRE-OnFire based on the incident report template.

@jcrespo I am writing the incident report at https://wikitech.wikimedia.org/wiki/Incidents/2022-11-17_Gerrit_3.5_upgrade ;)

Largest H2 files:

-rw-r--r--  1 gerrit2 gerrit2 8.2G Nov 17 11:47 git_file_diff.h2.db
-rw-r--r--  1 gerrit2 gerrit2  12G Nov 17 11:47 gerrit_file_diff.h2.db

The indexing completed at roughlly 11:05 UTC. That caused Gerrit to be automatically started either by Puppet or systemd. It had some more disk space issue.

I have started a full offline reindexing at 9:56 UTC.

The update had some issues:

I have asked the upstream authors and the plugin is indeed legacy. It would need a complete rewrite of the UI part and probably some addition to Gerrit UI itself. It was not wide spread used by them and they have choose to discontinue it.

I have scheduled the upgrade for tomorrow Thursday November 17th at 9:00 UTC which is just after the backport window. https://wikitech.wikimedia.org/wiki/Deployments#Thursday%2C_November_17

Gerrit now has the attention set feature ( https://gerrit.wikimedia.org/r/Documentation/user-attention-set.html ) which should help keeping track of changes one has to act on.

In T228084#5549908, @Paladox wrote:

You can either configure the inbuilt image server or use an external one, see https://gerrit.googlesource.com/plugins/imagare/+/refs/heads/master/src/main/resources/Documentation/about.md

Upstream script contrib/find-duplicate-usernames.sh no more reports any duplicate against username:, gerrit: or external: scheme \o/

And this morning I thought: what about upper case unicode characters? There are three such accounts found by matching against unicode characters having the property letter + uppercase which in pcre is \p{Lu}.

[externalId "gerrit:Ál"]  # LDAP account
	accountId = 2172
	email = al@blogmail.cc

I have pushed a commit to All-Users.git to adjust all those accounts (cecd04c3ccd7b40097b753d0c7226b6861c6d5a2).

Most of the 199 accounts having an upper case letter are the same as the lower case version (checked with diff -i). I though 7 accounts that had a mismatch:

I have verified all of the Gerrit external ids for LDAP (scheme gerrit:) having an upper case letter have an equivalent all lower case. The only differences I have spotted is some email addresses have been updated.

The reason is we have ldap.localUsernameToLowerCase=true. The upstream script to do the conversion had issue and instead @thcipriani crafted a shell script for that:

There is no more any upper case letter in Gerrit accounts (scheme username:):

$ git grep -P 'username:.*[A-Z]' refs/meta/external-ids --
$

I have reindexed all accounts with ssh -p 29418 gerrit.wikimedia.org -- gerrit index start accounts --force then queried again the three accounts via the REST API and ran a diff:

diff -u99 -r before/78 after/78
--- before/78	2022-11-15 16:17:21.000980424 +0100
+++ after/78	2022-11-15 16:17:55.937418442 +0100
@@ -1,6 +1,6 @@
 {
   "_account_id": 78,
   "name": "Kaldari",
   "email": "kaldari@gmail.com",
-  "username": "Kaldari"
+  "username": "kaldari"
 }
diff -u99 -r before/381 after/381
--- before/381	2022-11-15 16:17:30.761102834 +0100
+++ after/381	2022-11-15 16:18:02.741503701 +0100
@@ -1,6 +1,6 @@
 {
   "_account_id": 381,
   "name": "Fran",
   "email": "fran@dumetella.net",
-  "username": "Fran McCrory"
+  "username": "fran mccrory"
 }
diff -u99 -r before/3327 after/3327
--- before/3327	2022-11-15 16:17:40.485224760 +0100
+++ after/3327	2022-11-15 16:18:10.313598566 +0100
@@ -1,6 +1,6 @@
 {
   "_account_id": 3327,
   "name": "SamanthaNguyen",
   "email": "codynguyen1116@gmail.com",
-  "username": "SamanthaNguyen"
+  "username": "samanthanguyen"
 }

I have converted SamanthaNguyen to all lower case:

commit f8b7cc2dc1844518cdd92495d8df1cf5b94bc3d5 (HEAD -> meta/external-ids, origin/meta/external-ids, refs/meta/external-ids)
Author: Antoine Musso <hashar@free.fr>
Date:   Tue Nov 15 15:14:12 2022 +0000

I have converted username:Fran McCrory to be all lower case:

I have manually deleted username:Kaldari in All-Users.git:

commit 495ec6989a771ca345d89d6df8055c66dd91e888 (HEAD -> meta/external-ids, origin/meta/external-ids, refs/meta/external-ids)
Author: Antoine Musso <hashar@free.fr>
Date:   Tue Nov 15 14:44:13 2022 +0000

My plan:

The reason for those accounts originates from when we turned on ldap.localUsernameToLowerCase T152640, T197083, T197257 etc. The upstream script to handle that LDAP username change failed apparently and left some accounts broken.

The major change which affects us is normalizing username to be all lower case. I initially gave it a try with a local checkout of the All-Users.git database but it lacks a copy of refs/users/* and thus the script was essentially doing nothing.

I think it is good now :-]

Part of this task relocated the Castor instance and I took that opportunity to also change the directory layout for the service. As a result we have been serving stall caches for quite a while. T323051 fixes it.

I ran Tyler's script from /home/thcipriani/elapsed_gc_time.py

gerrit1001:~$ python /home/thcipriani/elapsed_gc_time.py|sort -g|tail -n 10
0:00:27	operations/software/netbox-deploy
0:00:30	scoring/ores/editquality
0:00:35	analytics/superset/deploy
0:00:35	wikimedia/production
0:00:42	mediawiki/extensions/Wikibase
0:01:17	All-Users
0:01:31	mediawiki/extensions
0:03:49	operations/puppet
0:04:01	mediawiki/core
0:31:26	TOTAL

This is most probably a representation bug in the Web interface (PolyGerrit).

Even in Phabricator, vanish @ori comments!

For later reference, we restarted Gerrit a couple time on November 10th 2022 and some users reported being logged out after the restart(s).

Done! I have confirmed the JavaScript plugins are properly loaded and gerrit-theme.js is gone :-]

I have published them to gerrit2002 (the replica). Gerrit automatically reloaded plugins, from error log:

[2022-11-09T12:17:00.676Z] [PluginScanner] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin wm-app-theme, version app-theme
[2022-11-09T12:17:00.677Z] [PluginScanner] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin wm-custom-links, version custom-links
[2022-11-09T12:17:00.677Z] [PluginScanner] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin wm-test-result-table, version test-result-table

And I had to reinstall the builtin plugins with:

sudo -u gerrit2 java -jar /var/lib/gerrit2/review_site/bin/gerrit.war init --batch --install-all-plugins

On gerrit1001 and gerrit2002:

Info: Applying configuration version '(0ff3c7b7bc) Filippo Giunchedi - gerrit: remove gerrit-theme.js'
Notice: Applied catalog in 31.94 seconds

Most dependencies on Pypi now provides wheels and our build download them. It is not really optimal but it is good enough. Maybe I will revisit this again later if I need it, meanwhile I am declining the task.