I have rollbacked due to T228436

Indeed the rollback got rid of the spam of errors. To be investigated now is what kind of db transactions are taking a while in Wikimedia\Rdbms\LBFactory->commitMasterChanges() :-\ There might be informations in logstash log if MediaWiki logs anything for databases.

That happens mostly on enwiki. Over one hour 4k occurrences out of 4400 total matches.

@SBisson definitely :-]

Same on enwiki https://en.wikipedia.org/w/api.php?format=json&action=query&list=recentchanges&rctoken=patrol

Can be reproduced using logged in or not with:

Thanks, I can confirm the component is around and it addresses the concern of mixing up upgrades with Toolforge. However that imports 18.09.7 but we need the previous version 18.06.x for now :-\

Too many changes to backport for REL1_32 or REL1_31 so I just skip those. It is not a huge win anyway :-]

Done on All-Projects.git via https://gerrit.wikimedia.org/r/#/c/All-Projects/+/524206/

Hurrah :]

Yes that looks good now. Thank you!

That is with systemd 215-17+deb8u13 from Jessie. systemd.service (5) states:

Given PageForms is supposedly usable without SMW, would it makes sense to have run TWO phan analysis? One with SMW and another one without it?

integration/config.git has:

- name: translatewiki
  test:
    - translatewiki-rake-docker
    - translatewiki-composer-hhvm-docker
  gate-and-submit:
    - translatewiki-rake-docker
    - translatewiki-composer-hhvm-docker

@Volans sounds good. I guess my concern was to potentially have a wrong state, but the daily crontab would indeed align the debmonitor database with the reality. So that addresses my concern :-]

The XML configuration for a job looks like:

<project>
  <properties>
    <jenkins.model.BuildDiscarderProperty>
      <strategy class="hudson.tasks.LogRotator">
        <daysToKeep>30</daysToKeep>
        <numToKeep>-1</numToKeep>
        <artifactDaysToKeep>-1</artifactDaysToKeep>
        <artifactNumToKeep>-1</artifactNumToKeep>
      </strategy>
    </jenkins.model.BuildDiscarderProperty>
...

Hurrah! Thank you to have remaindered us about this task :-]

I have quickly talked with @Paladox about it. He has tried the metrics-reporter-prometheus plugin and it does exposes all metrics to another endpoint. So we should add that plugin to our deployment, configure it, add a prometheus server and potentially we would have bunch of new metrics to build a dashboard with.

The job has produced https://gerrit.wikimedia.org/r/#/c/wikidata/query/gui-deploy/+/523962

The job happens after a change has been merged (Zuul pipeline: postmerge).

That job got copied from the one that builds wikimedia/portals assets. It suffered from the exact same issue: T227448

Another related one Serialization of 'Closure' is not allowed. reqId XS8cxQpAIDAAAF2FuqQAAABB

Seems WMF production database had some schema changes recently for the oauth_users database table:

• T225643: Schema change to oathauth_users
• T226826: Drop old oathauth_users columns

In T220739#5340812, @TheDJ wrote:

T228250: PHP Notice: Undefined property: stdClass::$module in OATHAuth/src/OATHUserRepository.php on line 193 could be a potential blocker ?

There is now a Javamelody prometheus exporter at https://gerrit.wikimedia.org/r/monitoring?format=prometheus . It reports JVM related metrics which are rendered on Grafana at https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody . That solely report Java internal metrics.

Erik Bernhardson has hit the same issue to add symfony/yaml for the CirrusSearch extension. T220723

Train blockers are unbreak now usually, then I am not sure how bad it is. Over 7 days there have been just 6 occurrences and they seem like end users altering the query manually. Hardly qualifies as a blocker to me?

The retention policy for a job shows up as:

In T228056#5336385, @hashar wrote:

So I guess lets just unset base::resolving::labs_additional_domains ?

So I guess lets just unset base::resolving::labs_additional_domains ?

Looking at compiler1004, it seems the metadata retrieval done by puppet has a 1 minute timeout of some sort:

Ah indeed on compiler1002.puppet-diffs.eqiad.wmflabs:

https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler-test/ now has NUM_THREADS=4.

*facepalm* I was debugging using the production branch catalog instead of the change catalog. Thank you !!!!! :-]

Have you planned the cloudvirt yet? I guess that is a bit more challenging since instances would have to be moved ahead of time, but I am genuinely interested in seeing whether that improves the bad CPU experience I have noticed.

I would guess that the new tag would be kind of a super set of ci-test-error isn't it? Regardless yes please be bold :-]

Puppet fails due to /etc/puppetdb being a directory when it tries to make it a symlink. Eventually I found we use the default puppetdb:

# apt-cache policy puppetdb
puppetdb:
  Installed: 2.3.8-1~wmf1+stretch
  Candidate: 4.4.0-1~wmf2
  Version table:
     4.4.0-1~wmf2 1001
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/component/puppetdb4 amd64 Packages
 *** 2.3.8-1~wmf1+stretch 1001
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

In hiera config:

- profile::puppetdb::master: compiler1002.puppet-diffs.eqiad.wmflabs
+ profile::puppetdb::master: compiler1003.puppet-diffs.eqiad.wmflabs

On compiler1003 I have applied role::puppet_compiler and the hiera config. Postgre fails though:

Notice: /Stage[main]/Postgresql::Slave/Exec[pg_basebackup-compiler1002.puppet-diffs.eqiad.wmflabs]/returns: pg_basebackup: could not connect to server: FATAL:  no pg_hba.conf entry for replication connection from host "172.16.2.57", user "replication", SSL on
Notice: /Stage[main]/Postgresql::Slave/Exec[pg_basebackup-compiler1002.puppet-diffs.eqiad.wmflabs]/returns: FATAL:  no pg_hba.conf entry for replication connection from host "172.16.2.57", user "replication", SSL off
Error: /usr/bin/pg_basebackup -X stream -D /srv/postgres/9.6/main -h compiler1002.puppet-diffs.eqiad.wmflabs -U replication -w returned 1 instead of one of [0]
Error: /Stage[main]/Postgresql::Slave/Exec[pg_basebackup-compiler1002.puppet-diffs.eqiad.wmflabs]/returns: change from notrun to 0 failed: /usr/bin/pg_basebackup -X stream -D /srv/postgres/9.6/main -h compiler1002.puppet-diffs.eqiad.wmflabs -U replication -w returned 1 instead of one of [0]
Notice: /Stage[main]/Postgresql::Slave/File[/srv/postgres/9.6/main/recovery.conf]: Dependency Exec[pg_basebackup-compiler1002.puppet-diffs.eqiad.wmflabs] has failures: true
Warning: /Stage[main]/Postgresql::Slave/File[/srv/postgres/9.6/main/recovery.conf]: Skipping because of failed dependencies
Notice: /Stage[main]/Puppetdb::App/File[/etc/puppetdb]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Puppetdb::App/File[/etc/puppetdb]: Not removing directory; use 'force' to override
Error: Could not remove existing file
Error: /Stage[main]/Puppetdb::App/File[/etc/puppetdb]/ensure: change from directory to link failed: Could not remove existing file
...

Eventually the instance console has shown the login prompt after 2300 seconds (38 minutes). Running puppet again:

Notice: Applied catalog in 7.76 seconds

I tried provisioning a new instance but puppet takes age / fail retrieiving files. I have filled T228056 for that.

On releases1001.eqiad.wmnet in /var/lib/puppet/state/state.yaml there is a Service[bacula-fd] but no Service[bacula]. Then puppet seems to work just fine on the host.

I have made Quibble to populate the localization cache, so that is good enough for now.

Seems we would want to tag 0068b08e120d43b09a41115aa08b1043deb58f4b

commit 0068b08e120d43b09a41115aa08b1043deb58f4b
Author: Marko Obrovac <mobrovac@wikimedia.org>
Date:   Tue Apr 30 17:36:30 2019 -0700

@Hexmode on test/gerrit-ping.git can you try creating a reference in a sandbox and then deleting it? https://gerrit.wikimedia.org/r/522462 should enable that. Then I guess we can enable the feature by default via All-Projects.git.

Hurrah. Thank you @CDanis & @godog :]

No one working on it, but can be reopened or filled again if there is interest later on.

The Jenkins logger ( https://integration.wikimedia.org/ci/log/Jenkins%20Queue/ ) missed the FINEST log level. The reasons are the same as shown in the web gui, for example:

In the Jenkins log bucket hudson.model.Queue.

Another threaddump P8736
https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMTEvLS10aHJlYWRkdW1wLnR4dC0tOS0zMC0zMg==

For T72597

I am not sure discernatron is still used / maintained. Eventually there is an open change to migrate to a different library:

The selenium-wikibase-chrome job was based on ruby mediawiki_selenium. This task was to run that test suite on patch submission but it never has been done since the suite was so long. The job eventually got deleted a month ago: