Latest graph, FYI:

To add some more data/justification for this task: the number of Gerrit repositories has grown:

I do see the gerrit apache logs in logstash; however, I don't see the log4j logs for gerrit in logstash (https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/manifests/gerrit/server.pp#110) I do see it in the syslog though.

Also, I should say, feel free to yell at us on the talk page for the architecture document – it's a long document, but it encapsulates a lot of thought on CI.

Nice proposal! This dovetails with a lot of our current thinking about CI. We really would like to expose developers to JJB as little as possible—it's not healthy for anyone :)

In T222820#5504304, @Bstorm wrote:

If this project ends up integrating with WMCS-managed stuff at all (Beta cluster -- does that mean deployment-prep?), I'd at least be interested in being a fly on the wall. I'm kind of curious what people come up with in general for our use or understanding, but if we are doing any peering or VPN with things in Cloud, I definitely would like to know to see how it impacts things.

In T230024#5502211, @jijiki wrote:

@thcipriani is it ok if we update php7 to 7.2.22 on deploy* servers? Do you know if there are any dependencies ?

In T231928#5464522, @Mathew.onipe wrote:

Not sure but seems we are missing some configs in our config.yaml patch

Patches are moving, looks like this is currently in progress, doing the workboard fiddling accordingly

posted upstream as https://groups.google.com/forum/#!topic/repo-discuss/Ah6ftyVe39c

In T232538#5491215, @ssastry wrote:

In T232538#5490768, @gerritbot wrote:

Change 536598 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[operations/puppet@production] beta cluster: Make deployment-parsoid09 a Mediawiki appserver as well
https://gerrit.wikimedia.org/r/536598

CPT folks, I've done a very preliminary patch here. I don't yet know what else is involved there but, if @jeena's code review indicates it is simple to push it over the hump since I've already licked the cookie, so to speak. But, if any of you want to drive this to completion, I appreciate the help. Please take it over.

In T232661#5485794, @MarcoAurelio wrote:

I've tried to force the update via HTTP and via SSH. None of them do work. I'm mentioning @mmodell and @thcipriani so they can take a look at the Phabricator account on Gerrit, as well K18/K19.

In T232619#5483743, @hashar wrote:

I think I have created an instance based on the same name earlier but using Stretch, that was to prepare the migration of the instances to Stretch. Not a big loss though.
What I suspect is there is a cert mismatch between the client (which would use the labs puppetmaster cert) and the CI puppetmaster. That is a known issue. Potentially clearing the cert on the puppetmaster might solve the issue, else we will have to recreate the instance.

In T231872#5478912, @hashar wrote:

On cobalt from lsof -p $(pidof java) (F30290036), I see a LOT of pack files opened:

Restarted Gerrit.

In T232195#5474581, @kostajh wrote:

Ah, it's only failing for extensions which have no unit tests, and which have 0% coverage in the junit.xml. These then show up as having 100% coverage in doc.wikimedia.org/cover-extensions.
We didn't intend to change the doc.wikimedia.org/cover-extensions report, but by switching to using unit tests and not integration tests for coverage calculation, we have.

In T218844#5171654, @gerritbot wrote:

Change 507787 merged by Dzahn:
[operations/puppet@production] Gerrit: Redirect /p/(.+)/info/(.+) to /$1/info/$2
https://gerrit.wikimedia.org/r/507787

In T224857#5463992, @Joe wrote:

@thcipriani should we create a new package/release?

In T224857#5462415, @gerritbot wrote:

Change 533782 merged by jenkins-bot:
[mediawiki/tools/scap@master] Fix restart_php
https://gerrit.wikimedia.org/r/533782

In T230138#5404114, @CDanis wrote:

Aren't we already alerting on Gerrit responding to HTTP requests in a timely fashion? Shouldn't catching that symptom that be enough?

I saw this email as well. Puppet is working there now and haven't gotten an email in a while.

Also relevant is the GC timing chart from the train window. The branch cut for train today began at 12:32UTC: https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?orgId=1&panelId=14&fullscreen&from=1567513099983&to=1567516699983&var-Application=&var-Window=30m

This should be complete now.

This has been re-enabled since August 8th. I have been waiting to see if this change, once again, caused T224448 to increase in frequency before resolving this task.

In T231198#5437902, @Ladsgroup wrote:

The issue actually started in Friday:

• https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/27634/consoleFull
• https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/27859/consoleFull

Before Friday it took this for phpunit excluding database group:

Time: 3.66 minutes, Memory: 1716.30MB

In Friday it took:

Time: 9.1 minutes, Memory: 2710.30MB

No test got added or removed (looking at slow tests logs) every single test just got slower.

In T226240#5421656, @Paladox wrote:

@hashar phabricator has been migrated to use the replica :)

This looks like @Urbanecm 's umask maybe set wrong:

In T217008#5296569, @kostajh wrote:

@thcipriani wondering if the code health group could move ahead with this. We have the polling script approach working but we would like to use a bot so that we can post a small summary of the specific codehealth issues (rather than just a link to sonarcloud). I know you had also mentioned triggering a job build directly from the webhook. Please let us know if you have thoughts on a preferred approach.

In D1153#23102, @MarcoAurelio wrote:

What's the next step? arc land?

In T224857#5413024, @gerritbot wrote:

Change 530014 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] scap: set flag for check-and-restart-php
https://gerrit.wikimedia.org/r/530014

In T229756#5390681, @thcipriani wrote:

hrm. I wrote a little shell script that checks all wmf-deployed extensons:

check-gerrit-branch.sh

#!/usr/bin/env bash
set -euo pipefail
# Path to local copy of mediawiki/tools/release
TOOLS_REPO_PATH="$1"
BASE_PATH="https://gerrit-slave.wikimedia.org/r/mediawiki"
BRANCH='wmf/1.34.0-wmf.16'
for ext in $(jq -r '.extensions | .[]' "$TOOLS_REPO_PATH"/make-wmf-branch/config.json); do
    printf 'CHECKING %s....' "$ext"
    if ! git ls-remote --heads "${BASE_PATH}/${ext}" | grep -q "$BRANCH"; then
        printf '\n\t[ERRROR] no branch "%s" found for "%s"!\n' "$BRANCH" "$ext"
        exit 1
    fi
    printf 'Found!\n'
done

@Viztor this should be resolved for you, please re-open if that is not the case.