hrm. I can recreate the test failures in a container on the CI infra.

In T221026#5206626, @hashar wrote:

Regarding the SendEmail thread, it took a while to remember about it but T131189 was about SendEmail having stuck tcp connections eventually blocking the task and thus the thread pool. Alexandros used gdb to close the sockets "manually" :D The task is worth reading.

hrm, well I can say nothing changed with that specific function:

It seems that the cassandra subchart already exists for cask (via https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/509102/ ); however, to use that in the pipeline we would have to override some values at deployment time.

In T222539#5169148, @thcipriani wrote:

In T222539#5169146, @gerritbot wrote:

Change 508488 merged by jenkins-bot:
[mediawiki/tools/scap@master] Clear MessageBlobStore after syncing i18n data

https://gerrit.wikimedia.org/r/508488

Merged in scap master, will have to create a new version of the scap deb and upload it to get it to production. I can do that next week if no one beats me to it.

I think I have a reasonable explanation for this over in: T217830#5009234

In T222015#5159988, @Nikerabbit wrote:

I am not sure I am allowed to do that per the new privilege policy.

In T222539#5169146, @gerritbot wrote:

Change 508488 merged by jenkins-bot:
[mediawiki/tools/scap@master] Clear MessageBlobStore after syncing i18n data

https://gerrit.wikimedia.org/r/508488

@MisterSynergy sorry I missed your username on T220867 :(

In T141324#5168214, @Dzahn wrote:

Deployed the change above and restarted Gerrit. The new file /var/log/gerrit/gerrit.json has been created now and logs are written to it. Works! Thanks for the change, Paladox.

@mmodell and I talked about this a bit during our pairing, assigning to him to work on.

As of 1.34.0-wmf.4 this is working! https://www.mediawiki.org/wiki/MediaWiki_1.34/wmf.4/Changelog was generated by jenkins with no manual steps other than cutting the branch.

assigning to @dduvall based on hangout discussion

I did update the file by manually replacing it with the version in master.

Updated your account in the DB, please reopen if that does not resolve your issue.

Updated your account in the DB, please reopen if that does not resolve your issue.

Updated your account in the DB, please reopen if that does not resolve your issue.

In T222621#5164600, @alaa_wmde wrote:

@thcipriani yes it has been reverted already

I notice that this is still blocking the train, but no one has addressed this yet.

In T141324#5122978, @herron wrote:

In T141324#5122806, @Gehel wrote:

• structured logging from log4j can be exposed in a number of ways. The easier is probably to continue logging to file, use syslog for transport, but have messages in a structured format. A json layout could be used for that (here is one from Jetbrains, which I haven't tested, but I tend to trust Jetbrains). This would allow to output all logging context and not just a few basic information.

This sounds like a solid option taking into account the ongoing migration towards shipping to logstash via the local rsyslogd, which produces to the Kafka logging pipeline. Can this output json formatted messages to syslog prefixed by an @cee cookie?

Someone whose local environment is running node 6.11.0 will need to regenerate this, as I can't.

In T222324#5153358, @sbassett wrote:

@thcipriani - these are the two security patches that were deployed on Tuesday: T222036#5142596, T222038#5142604 (though not the -formatter patch.) These should only affect granular view permissions for certain revdel logs.

Looking at logstash, I'm seeing this happen on mwdebug, so maybe folks are already looking into it. Anyway, here's a stacktrace if it's helpful

In T222324#5151758, @Bawolff wrote:

Note security did some minor adjustments to the revdel process on tuesday but nothing that should cause this.

In T222329#5152446, @Aklapper wrote:

Tentatively setting T220728 as parent task. Please feel free to correct me.

In T222195#5150981, @kostajh wrote:

@thcipriani the patch above should fix it; if you create a separate task please comment here and I'll update the patch.

Seems as though this is still happening in 1.34.0-wmf.3 -- stacktrace is a little different though, so maybe a different path to the same problem? This seems to be coming from api.php

This is currently the largest producers of errors in logstash. It doesn't sound like this is actually anything to be concerned about, but fixing logging would be nice since it looks like 3 separate train deploys have been worried enough to comment on this task.

This is not done until it has been removed from make-wmf-branch. I don't know the status of JADE so I don't know what the consequences are of renaming it in terms of localization updates for new branch deployments. I, unfortunately, discovered this in the middle of cutting a branch and consequently don't have the cycles to look more deeply at this space. Release-Engineering-Team can look at this this coming week.

I have noticed 2 problems:

It seems like we've tuned a good number of gerrit parameters at this point and we're still experiencing GC thrashing (although less than previous) which means we ought to start looking at JVM GC tuning.

In T221026#5121121, @thcipriani wrote:

I managed to capture a threaddump from the moment tasks started piling up: https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDQvMTcvLS1qc3RhY2stMTktMDQtMTctMjAtNTgtMDIuZHVtcC0tMjEtNDctNA==

I'll post this upstream to see if it reveals any insights.

[0]. https://groups.google.com/d/msg/repo-discuss/pBMh09-XJsw/vuhDiuTWAAAJ

changeid_projects cache is, today, looking like it's in poor shape:

In T218750#5125231, @akosiaris wrote:

Before we move forward and enable this, let's make sure we have understood the security repercussions and have mitigated them (and if we find it impossible to do so, avoid it).

We don't actually sync the cdb files https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/scap/tasks.py#58

Other thing to note in gerrit show-caches output:

I upgraded Gerrit to 2.15.12 in preparation for plugins still in development. I'd like to not change too many things at once, but I am a bit stuck

In T221026#5119712, @hashar wrote:

Requests	IP	DNS PTR
69921	2620:0:861:102:10:64:16:8	phab1001.eqiad.wmnet.

In T221026#5116925, @thcipriani wrote:

Here's the GCEasy report from around the time gerrit started thrashing today:
https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDQvMTYvLS1qdm1fZ2MuZ2Vycml0LmxvZy4xLS0yMC01NS0zMQ==&channel=WEB

A threaddump from right before and right after the first GC at 10UTC:
https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDQvMTYvLS1qc3RhY2stMTktMDQtMTYtMDktNTAtMDMuZHVtcC0tMjItMjUtMjE7Oy0tanN0YWNrLTE5LTA0LTE2LTEwLTAwLTAyLmR1bXAtLTIyLTI1LTIx

The notable change I see in the threaddumps is there some blocking on reading a packfile in the second dump. That lock clears in subsequent dumps. That kind of BLOCKING doesn't seem uncommon -- 13% of the threaddump files I have (currently about 550 files from 5 or so days).

In T221026#5117178, @mmodell wrote:

From looking at http requests per minute in javamelody, over 1 year, I see that traffic has increased a lot:

	https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=httpHitsRate

Each simultaneous request allocates a significant chunk of ram. I think we need to increase the heap size a bit and possibly reduce the size of the thread pool to limit memory use.

Here's the GCEasy report from around the time gerrit started thrashing today:
https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDQvMTYvLS1qdm1fZ2MuZ2Vycml0LmxvZy4xLS0yMC01NS0zMQ==&channel=WEB

Once again, around 10UTC JVM GC started kicking in and tread use is up and has not dropped back down:

Reopening following rollback

From the "Scaling Gerrit" link above:

In T221026#5112672, @LarsWirzenius wrote:

Just to clarify, when you say GC, do you mean Java virtual machine garbage collection or git repository object garbage collection?

Can we add back both @Legoktm and @QChris to Gerrit Administrators? Seems like they were doing work that needs Administrator privilege and exercising their privilege judiciously. This was discussed in a higher-bandwidth convo with me, @greg, and @hashar which I think are the folks who needed to agree for this ticket to be resolved.

In T218783#5099959, @greg wrote:

Since these are vms, can their disk be expanded easily? I ask because they're the oddballs in the mw fleet :) I know Tyler wants to fix this issue this week, though.