https://github.com/jenkinsci/gearman-plugin/pull/13 updates the code to a bit more modern interface :]

I have send a tentative patch upstream to url encode POST made to conduit: https://gerrit-review.googlesource.com/c/plugins/its-phabricator/+/305522 But I don't know how to write tests in Java and I don't have any idea how to test the code I wrote :-\

its-phabricator code to send to conduit is:

/**
 * Calls a conduit method with some parameters
 *
 * @param method The name of the method that should get called
 * @param params A map of parameters to pass to the call
 * @return The call's result, if there has been no error
 * @throws ConduitException
 */
JsonElement call(String method, Map<String, Object> params, String token)
    throws ConduitException {
  String methodUrl = apiUrlBase + method;

Done in January via a721b6c0b7468df793b03d7e42e8c09a02884f52 which referenced the parent task T266777: integration instances suffer from high IO latency due to Ceph

Argh. So I looked at pending open patches:

It seems to be solely for the Analytics team.

We have indeed upgraded to Gerrit 3.2 (June 2020) which comes solely with the new UI. So I guess that makes this task obsolete ;)

As per my previous comment, I could not find why the tag would not have replicated. Afterward the creation of a tag got replicated properly. We also have since upgraded from Gerrit 2.15 to Gerrit 3.2 so I don't think this task is relevant nowadays.

Assuming that got solved by the last patch.

Should be good. It is gone from CI and read only in Gerrit.

The authentication related callers are to switch the thread to have the Jenkins instance privileges (ACL.SYSTEM instead of some user privileges). The StopJobWorker and SetDescriptionWorker threads oth call GearmanPluginUtil.findBuild() which does invoke the authentication check:

/**
 * Function to finds the build with the unique build id.
 *
 * @param jobName
 *      The jenkins job or project name without folder name
 * @param buildNumber
 *      The jenkins build number
 * @return
 *      the build Run if found, otherwise return null
 */
public static Run<?,?> findBuild(String jobName, int buildNumber) {

And that happened again yesterday:

May 04, 2021 7:13:03 PM SEVERE hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException

Great, thank you @EBernhardson . Will do the archival with @Gehel :]

If I abandon a change in Gerrit with the content %28%2B%29 it is shown as is in Gerrit but in Phabricator the comment has somehow been decoded and show up as (+). So the comment got url decoded, either by Gerrit its-phabricator / its-base plugin due to the Soy templates being too smart, or Phabricator conduit expect the payload to be urlencoded (that sounds weird).

From the Jenkins code in core/src/main/java/hudson/model/Computer.java for getHostName(), and playing it in https://integration.wikimedia.org/ci/script:

import java.net.NetworkInterface

The code:

// constructor
private GearmanProxy() {
    gewtHandles = Collections.synchronizedList(new ArrayList<ExecutorWorkerThread>());
    gmwtHandles = Collections.synchronizedList(new ArrayList<ManagementWorkerThread>());

The graph will remain high until the Gearman server is restarted to flush the backlog. That has to be done by restarting the Zuul scheduler which I will do tomorrow morning when CI is less busy.

After restarting, the manager is:

$ zuul-gearman.py workers|cut -b-90|grep manager
21 ::ffff:127.0.0.1 172.17.0.1_manager : set_description:172.17.0.1 stop:172.17.0.1

The stop* and set_description* functions are handled by ManagementWorkerThread in the Gearman plugin. It has a single worker:

$ zuul-gearman.py workers|grep stop:contint2001
30 ::ffff:127.0.0.1 contint2001.wikimedia.org_manager : stop:contint2001.wikimedia.org set_description:contint2001.wikimedia.org

The JJB config for the search/mjolnir has a comment referring to search/xgboost so maybe it is still used. Though most probably it is an artifact from the past and the code has since been updated to no more rely on xgboost.

My message had Abandonning ... , the template thus has %%%Abandonning and I guess that %A is interpreted as well as other %. I am not quite sure I understand what is going, but I suspect it might that its-phabricator does not url encode the payload when sending to Phabricator conduit. I am wondering what would happen if one abandon a change with some urlencoded string.

Tried again and there is a server side error:

Rendered template /var/lib/gerrit2/review_site/etc/its/templates/PatchSetAbandoned.soy to:
Change 683803 **abandoned** by Hashar:
%%%[test/gerrit-ping@master] __version__%20%7C%20https%3A%2F%2Fexample.org%2F%2B%2Ffoo.html%%%
Reason:
%%%Abandonning%20after%20https%3A%2F%2Fgerrit.wikimedia.org%2Fr%2Fc%2Foperations%2Fpuppet%2F%2B%2F683810%20got%20deployed%%%
https://gerrit.wikimedia.org/r/683803

Fun thing, @Paladox raised that on Google Closure Template mailing list at https://groups.google.com/g/closure-templates-discuss/c/lXOqqmkO7fE :)

I went to elevate a bunch of logging level in Gerrit. Loggers can be found using:

com.googlesource.gerrit.plugins.its.base.its.ItsConfig: INFO
com.googlesource.gerrit.plugins.its.base.util.CommitMessageFetcher: INFO
com.googlesource.gerrit.plugins.its.base.util.IssueExtractor: INFO
com.googlesource.gerrit.plugins.its.base.validation.ItsValidateComment: INFO
com.googlesource.gerrit.plugins.its.base.workflow.ActionController: INFO
com.googlesource.gerrit.plugins.its.base.workflow.ActionExecutor: INFO
com.googlesource.gerrit.plugins.its.base.workflow.AddSoyComment: INFO
com.googlesource.gerrit.plugins.its.base.workflow.ItsRulesProjectCacheImpl: INFO
com.googlesource.gerrit.plugins.its.base.workflow.ItsRulesProjectCacheRefresher: INFO
com.googlesource.gerrit.plugins.its.base.workflow.RuleBase: INFO
com.googlesource.gerrit.plugins.its.phabricator.PhabricatorItsFacade: INFO
com.googlesource.gerrit.plugins.its.phabricator.PhabricatorModule: INFO
com.googlesource.gerrit.plugins.its.phabricator.conduit.ConduitConnection: INFO

The change for T93331 is to wrap the abandoning reason with a Phabricator literal block marker: %%% . The code at https://gerrit.wikimedia.org/r/c/operations/puppet/+/675479/2/modules/gerrit/templates/its/PatchSetAbandoned.soy.erb can be summarized as:

Reason:{\n}
- {$reason}
+ %%%{$reason}%%%{\n}

I wrote a blog post summarizing this adventure: Blog Post: Tracking memory issue in a Java application

I have nuked the cache, but it still pick up the "wrong" flake8:

Collecting flake8
  Downloading flake8-3.9.1-py2.py3-none-any.whl (73 kB)

+ @Volans due to overall knowledge about python linters / tox etc.

I note that the MediaWiki core code coverage run went from a few hours run down to just 13 minutes! https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/ Well done @Daimona

The job is https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/ , it indeed used to take a few hours to complete and is now completing in 13 minutes! The reason is @Daimona switched from xdebug to pcov: T234020 :]

I have no idea whether that is still an issue nor how to even reproduce it. I think the original intent was to run the PHPUnit tests with all Wikimedia extensions installed. Then that was like 6 years ago... ;)

The job passes now, I apparently just forgot to mark this resolved.

The theory is we can disable the account in wikitech/ldap. There is an account in wikitech https://wikitech.wikimedia.org/wiki/Special:ListUsers?username=Mdhollo :

And that should be good now! :]

Revisiting a decade old bug.

I have confirmed that eatmydata is now included in all cowbuilder images. When jenkins debian glue invokes the build, it does try to include eatmydata and show it is already available:

@Addshore could it be that the MediaWiki api is slow to respond? We have some debug logs available and attached to the build as mw-debug-www.log.gz. Also, the jobs use the php built-in server which process requests serially so that might cause issues. There is an Apache based flavor of the job which I think can be triggered via the experimental pipeline, though that surfaces some race conditions in the tests :-/

In T281122#7038656, @hashar wrote:

TLDR: looks like there is not enough memory, at least on integration-agent-docker-1006

Log 2 and Log 3 fail at the same time on the same integration node (02:44:07 vs 02:44:15)

Grafana for the node running the job https://grafana-labs.wikimedia.org/d/000000590/instance-details?orgId=1&from=1619053200000&to=1619056800000&var-project=integration&var-job=node&var-node=integration-agent-docker-1006

From that dashboard under Memory Detail Meminfo we can see the memory committed exceeded the limit:

Maybe that in turns trigger the memory compaction system which seems to be a Linux feature to "defrag" the memory.

I note that integration-agent-docker-1006 seems to only have 8G of RAM!!!!!!!!!

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7721        2310        3353         572        2057        4562
Swap:          7947         452        7495

I am pretty sure we had them with 32G since they can run up to 4 jobs concurrently. I can't reach out to Horizon right now to confirm the VM memory.

TLDR: looks like there is not enough memory, at least on integration-agent-docker-1006

Great. The only issue I have encountered was with the Token Macro plugin on the CI Jenkins which I have somehow forgot to upgrade.

The CI Jenkins upgrade apparently went fine \o/

The note about the LDAP plugin is that we must have 1.26 installed which is the latest still compatible with the 2.263.3 Jenkins we are running and has:

Perfect, thank you @Dzahn and @elukey ;)

That is because the job Docker image moved from Stretch to Buster and thus it upgrades ruby from 2.3 to 2.5. In ruby 2.4 integer got changed hence why we get:

generator.c:861:25: error: ‘rb_cFixnum’ undeclared (first use in this function);
did you mean ‘mFixnum’?
     } else if (klass == rb_cFixnum) {
                         ^~~~~~~~~~

Danke Schon @Jakob_WMDE !

Neat! Looks like that requires Winstone-Jetty is configured to handle SSL/TLS connections but on our setup we use plain HTTP (and envoy / the cache infra for encryption. So I am guessing we are not affected.

@Ladsgroup thank you so much for fixing this test!

It is not an issue with Jenkins. The parser test suite creates a few files for testing purpose and delete them on completion. This task is that the test suite always referred to /tmp/Foobar.svg so when one runs tests in parallel there is an "opportunity" for the file to be deleted by one of the suite while the other has not completed yet and might then fail cause the file got deleted.

I have looked at the Apache scoreboard and Gerrit thread pool and they seem fine now. Gerrit is at less than 40 threads out of a total pool of 60. Apache does have some spikes, but they are below the 150 workers limit. So that part is solved indeed.

SoothGallery no more shows up at https://gerrit.wikimedia.org/r/plugins/gitiles/translatewiki/+/refs/heads/master/groups/MediaWiki/mediawiki-extensions.txt , maybe that is the only thing that needs to be done to remove it from Translatewiki?

It seems this task is now completed :]

I guess that was a glitch on Travis side. If it happens again, I guess the best is to reach out to them directly, though I have no idea whether they offer support for Free plan.

We deleted the faulty refs/master directly on disk and that was not noticed by Gerrit. The fix is to request a full replication:

ssh -p 29418 gerrit.wikimedia.org replication start --now mediawiki/extensions/AutoCreateCategoryPages

Thank you @Paladox !

The puppet bits are preparation work to let us "trivially" switch from java 8 to java 11.

K-and-R looks like an organization and is unrelated. If I look at https://travis-ci.com/organizations/wikimedia/plan it states:

Not much we can do ourselves but that has been discussed this week on the Gerrit mailing list ( https://groups.google.com/g/repo-discuss/c/UUvDYv2qWqY ).