@thcipriani now deserves some well deserved time off after all the madness that task has caused :-]

Some time

The way we debug those is usually by looking at the Jenkins build console between the latest good build and the first bad build. Luckily there is a periodic job https://integration.wikimedia.org/ci/job/mobileapps-periodic-test/ which makes it easy to find the good/bad builds.

Eventually I found a MediaWiki application server with a CPU older than the one on cloudvirt1006: mw2139.codfw.wmnet has a Xeon CPU E5-2450 0 based on Sandy Bridge.

Changing priority, that has been worked for now. Still have to figure out the root cause though :-(

Ran it on a couple production host and two of my machines at home:

Thanks! So the benchmarks on the real hardware looks similar to what I get on the instance. I guess that rules out kvm/qemu/labvirt etc.

Looks like 1.34.0-wmf.6 is mostly fine.

From https://www.php.net/eval eval beahves differently on PHP7:

That seems to be the feature https://www.mediawiki.org/wiki/Growth/Focus_on_help_desk/Help_panel Apparently that only happens on the kowiki. There is no specific referrer URL to blame :/

Apparently the exception bubbles up and cause a message of: Calculating version for "ext.growthExperiments.Help" failed: BadMethodCallException: Sessions are disabled for this entry point

[13:25:28] <logmsgbot> !log hashar@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.6

@SBisson sorry I have missed T223741 hotfix ( https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Echo/+/512070/ ). I think I had it has a blocker, you found it to be a duplicate and eventually I have not let it be a blocker since that occurred previously.

Here is an explanation of the issue @Smalyshev encountered on T219749 as I posted it on https://gerrit.wikimedia.org/r/#/c/integration/config/+/511308/

In T223971#5206979, @bd808 wrote:

• cloudvirt1006.eqiad.wmnet has 2x 12 cores/24 threads, 384G of ram, and 15000rpm spinning disks on an Ivy Bridge (3rd gen) architecture.
• cloudvirt1025.eqiad.wmnet has 2x 18 cores/36 threads, 512G of ram, and SSD disks on a Skylake (6th gen) architecture.
  
  @hashar what is your hoped for outcome here? I'm not sure I understand why you are confused that 5 year old servers (cloudvirt1006) are slower than servers purchased in the last year (cloudvirt1025).

In T223971#5207016, @Krenair wrote:

I wonder if there should be two separate sets of flavours, one for each type of host. Probably wouldn't want an instance set up on one type migrated to the other. It sounds like right now if you see docs/examples that say a particular flavour should be used (perhaps on the basis of VCPUs), it's useless due to it actually coming down to the luck of what host you get scheduled on?

Thank you @Lucas_Werkmeister_WMDE

Yes that is an internal function that uses to be in /Collections.php has a global function. https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Collection/+/332929/ moved it to a class static method.

And I forgot T224093: [Collections] Fatal error: Call to undefined function wfAjaxCollectionGetItemList()

(and since that happened last week already, it is imho not a blocker for this week ;))

Ah indeed that is a duplicate! I blame myself for not properly searching in Phabricator. I filled it as a child task of the deployment blocker since I just clicked a link that automatically set the blocker task as a parent.

Thanks to @SBisson / @Reedy and @Jdforrester-WMF , FlaggedRevs faulty changes have been reverted and I have upgraded group 1 to 1.34.0-wmf.6.

Fixed by @SBisson that reverted FlaggedRevs extension.json for T224116
@Jdforrester-WMF also took care of rebuilding i18n messages.

Fixed by @SBisson / reverting extension.json

@Volans wrote:
I'm assuming that in cases in which the rebase fails because of conflicts or the CI fails after the rebase Jenkins would vote -1 and the patch would be out of the merging queue. Correct me if I'm wrong.

I should have searched in Phabricator before filling another task. Thank you @SBisson to have spotted it!

@jbond those OpenStack .original files, I am assuming they are copied verbatim from the upstream project. So one would assume that we would never modify them and that they already have been validated. So I guess it is fine to miss them or eventually explicitly exclude them. We can surely afford a few holes here and there!

Regarding the SendEmail thread, it took a while to remember about it but T131189 was about SendEmail having stuck tcp connections eventually blocking the task and thus the thread pool. Alexandros used gdb to close the sockets "manually" :D The task is worth reading.

Status update:

https://en.wikinews.org/wiki/Special:ProblemChanges and https://pl.wiktionary.org/wiki/Specjalna:W%C4%85tpliwe_zmiany no more shows the issue after moving back to MediaWiki 1.34.0-wmf.5

That might be related to T224116. There is also bab8ea77c7020795cd813cc390e3cbb03ccbb9fd

A spike of those, apparently only on meta.wikimedia.org so far.

The error is gone from https://logstash.wikimedia.org/goto/85e59a7e13a40e11c9595c444ea90375 and https://ca.wikinews.org/ is back up.

There are various code path from extensions or mediawiki/core that ends up hitting the error. It has been added in mediawiki/core by https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/504000/

Not a blocker since that predate 1.34.0-wmf.6.

Thank you @Aklapper , I have transfered the informations to the task.

I have rolled back cawikinews:

And https://ca.wikinews.org/ get us a Wikimedia error page. Time for a rollback!

Only got it on cawikinews so far.

https://gerrit.wikimedia.org/r/#/c/integration/config/+/511866 switches CI to only use composer install / composer merge plugin instead of relying on mediawiki/vendor.git

I am not sure whether that one should be a blocker given it already got shipped with the previous version last week (1.34.0-wmf.5).

Not a train blocker, that happened previously and the notice is only found for queries toward test.wikidata.org / test.wikipedia.org

That is emitted for lines 32 33 34 and 35 of JCApi.php:

30         if ( isset( $conf->store ) ) {
31             $res['store'] = [
32                 'cacheNewValue' => $conf->store->cacheNewValue,
33                 'notifyUrl' => $conf->store->notifyUrl,
34                 'notifyUsername' => $conf->store->notifyUsername !== '', // true or false
35                 'notifyPassword' => $conf->store->notifyPassword !== '', // true or false
36             ];
37         }

And on the same URL we also have:

As follow up, we now have: T224092: [Collections] Call to a member function getLocalUrl() on a non-object (null) :)