Fatal error: Call to undefined function Wikibase\Client\Tests\RecentChanges\both() - on jenkins
Closed, ResolvedPublic

Description

Wikidata build tests are failing because hamcrest functions are not being autoloaded (not included in mediawiki's vendor/composer/autoload_files.php)

https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/37857/console

These are being autoloaded for Wikibase tests (alone, outside the build). The issue seems to be just with the build. It works locally for me.

10:02:33 Fatal error: Call to undefined function Wikibase\Client\Tests\RecentChanges\both() in /srv/jenkins-workspace/workspace/mwext-testextension-hhvm/src/extensions/Wikidata/extensions/Wikibase/client/tests/phpunit/includes/RecentChanges/ChangeLineFormatterTest.php on line 177

The slave script bin/mw-fetch-composer-dev.sh apparently manage to inject the dev dependencies and they get installed:

00:00:48.043   - Installing hamcrest/hamcrest-php (v2.0.0)
00:00:48.045     Loading from cache
00:00:48.045     Extracting archive
00:00:48.123 
00:00:48.137   - Installing wmde/hamcrest-html-matchers (v0.1.0)
00:00:48.138     Loading from cache
00:00:48.138     Extracting archive

vendor/composer/autoload_files.php and or composer.lock might end up missing them though :(

Reproduction

mediawiki/core
mediawiki/vendor cloned in the vendor subdirectory
integration/jenkins

The CI script mw-composer-fetch-dev.sh grabs the list of packages from require-dev and then require them in the vendor subdirectory.

The dev dependencies:

(cd vendor; ~/projects/integration/jenkins/tools/composer-dev-args.js ~/projects/mediawiki/core/composer.json)
/Users/amusso/projects/mediawiki/core/vendor
composer/spdx-licenses=1.1.4
jakub-onderka/php-parallel-lint=0.9.2
justinrainbow/json-schema=~3.0
mediawiki/mediawiki-codesniffer=0.7.2
jetbrains/phpstorm-stubs=dev-master#1b9906084d6635456fcf3f3a01f0d7d5b99a578a
monolog/monolog=~1.18.2
nikic/php-parser=2.1.0
nmred/kafka-php=0.1.5
phpunit/phpunit=4.8.31
wikimedia/avro=1.7.7
hamcrest/hamcrest-php=^2.0
wmde/hamcrest-html-matchers=^0.1.0

For each we invoke composer require --dev --ansi --no-progress --prefer-dist -v . The result is available as https://gerrit.wikimedia.org/r/339404

Then (cd vendor && composer dump-autoload --optimize) which apparently is a noop.

Given a dummy test file:

tests/phpunit/BlaTest.php
<?php

use Hamcrest\Matchers;

class BlaTest extends MediaWikiTestCase {

    function testHamcrest() {
        both();
    }

}

That fails to load one of Hamcrest function:

$ php tests/phpunit/phpunit.php tests/phpunit/BlaTest.php
PHPUnit 4.8.31 by Sebastian Bergmann and contributors.

PHP Fatal error:  Call to undefined function both() in /Users/amusso/projects/mediawiki/core/tests/phpunit/BlaTest.php on line 8

Case that works

Instead of trying to lookup the static method via namespace, if we refer to the canonical class path it seems to work:

<?php
class CanonicalPathTest extends MediaWikiTestCase {

    function testHamcrest() {
        \Hamcrest\Matchers::both();
    }
}

Global functions

One has to explicitly register them with Hamcrest\Util::registerGlobalFunctions();:

<?php
Hamcrest\Util::registerGlobalFunctions();
class BlaTest extends MediaWikiTestCase {
    function testHamcrest() {
        assertThat();
    }
}

Other findings

vendor/composer/autoload_files.php refers to $vendorDir . '/mediawiki/at-ease/src/Functions.php'. That is because at-ease composer.json has:

"autoload": {
    "files": [
        "src/Functions.php"
    ]
}
aude created this task.Feb 21 2017, 4:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 21 2017, 4:53 PM

I invoke @Addshore and @Aleksey_WMDE since that is ham crest related. Gotta reproduce and figure out why ham crest is not loaded when running wikidata build with mediawiki/vendor although CI is supposed to fetch the dev dependencies :/

hashar updated the task description. (Show Details)Feb 21 2017, 10:12 PM

Change 339051 had a related patch set uploaded (by Hashar):
Capture composer.lock after mw-fetch-composer-dev

https://gerrit.wikimedia.org/r/339051

aude added a comment.Feb 21 2017, 10:14 PM

hamcrest-php and hamcrest-html-matchers are dev dependencies of mediawiki core.

Jenkins uses mediawiki/vendor (https://github.com/wikimedia/mediawiki-vendor/) and then dev dependencies of core are added with a script:

https://github.com/wikimedia/integration-jenkins/blob/master/bin/mw-fetch-composer-dev.sh

the jenkins workspace is shared / reused across different builds/jobs, so the issue might be related to that.

Jenkins is using a specific (older) version of composer: https://github.com/wikimedia/integration-composer

I tried locally with the same version of composer and the build (and with mediawiki/vendor) and couldn't reproduce the problem. Also, in other test runs on jenkins (e.g. Wikibase extension), hamcrest is installed and works without any issues.

aude added a comment.Feb 22 2017, 2:39 PM

think the issue is just that the script https://github.com/wikimedia/integration-jenkins/blob/master/tools/composer-dev-args.js only copies over the require-dev section from mediawiki's composer.json (piped to composer require)

this doesn't include autoload-dev. Think we need the script to also copy these over to mediawiki-vendor's composer.json

Change 339202 had a related patch set uploaded (by Aude):
Add script to copy composer require-dev and autoload-dev to mw vendor

https://gerrit.wikimedia.org/r/339202

hashar updated the task description. (Show Details)Feb 23 2017, 1:33 PM
hashar updated the task description. (Show Details)Feb 23 2017, 1:55 PM
hashar updated the task description. (Show Details)Feb 23 2017, 2:03 PM
hashar updated the task description. (Show Details)Feb 23 2017, 2:19 PM
hashar updated the task description. (Show Details)

So I don't think the PHP namespace lookup works for static methods. Eg both() would not end up being resolved as \HamCrest\Matchers::both(). A lame test case:

A namespaced class definition:

<?php
namespace namespaced {
    class StaticClass {
        public static function hello() {
            return "Hello";
        }
    }
}

Script using it:

<?php
use namespaced\StaticClass;

spl_autoload_register( function () { 
    include 'namespaced.php';
} );

print "namespaced\StaticClass::hello():\n";
print ">>> " . namespaced\StaticClass::hello() . "\n";  // works
print "hello():\n";
print ">>> " . hello() . "\n";  // Call to undefined function hello() 

Output with PHP 5.6.29:

$ php exec.php
namespaced\StaticClass::hello():
>>> Hello
hello():
PHP Fatal error:  Call to undefined function hello() in /tmp/exec.php on line 11
PHP Stack trace:
PHP   1. {main}() /tmp/exec.php:0

Fatal error: Call to undefined function hello() in /tmp/exec.php on line 11

Call Stack:
    0.0002     231576   1. {main}() /tmp/exec.php:0

That might why Hamcrest has a Hamcrest\Util::registerGlobalFunctions();: which include a file defining a set of global functions which in turns invoke the static method using the full canonical path. Extracted from the file:

vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest.php
<?php
if (!function_exists('both')) {
    function both(\Hamcrest\Matcher $matcher)
    {
        return \Hamcrest\Core\CombinableMatcher::both($matcher);
    }
}

So potentially in our tests, we can explicitly register the global functions using:

\Hamcrest\Util::registerGlobalFunctions();:

And can then use both().

Which works around the following PHP code that fails to lookup up the both() static method:

use Hamcrest\Matchers;
both();  // fails

Change 339051 abandoned by Hashar:
Capture composer.lock after mw-fetch-composer-dev

Reason:
The script knows about LOG_DIR , so I will archive the files directly from it in https://gerrit.wikimedia.org/r/#/c/339202/

https://gerrit.wikimedia.org/r/339051

Zppix added a subscriber: Zppix.Feb 23 2017, 10:14 PM

Change 339202 merged by jenkins-bot:
Option to merge mw composer into vendor

https://gerrit.wikimedia.org/r/339202

Mentioned in SAL (#wikimedia-releng) [2017-02-24T13:52:35Z] <hashar> deployed slave script update to be able to merge mediawiki/composer.json into vendor/composer.json 6527f49..a7728a5 https://gerrit.wikimedia.org/r/#/c/339202/ T158674

Mentioned in SAL (#wikimedia-releng) [2017-02-24T13:56:03Z] <hashar> Log refresh Nodepool instances to deploy slave script update to be able to merge mediawiki/composer.json into vendor/composer.json 6527f49..a7728a5 https://gerrit.wikimedia.org/r/#/c/339202/ T158674

Change 339634 had a related patch set uploaded (by Hashar):
Set MW_COMPOSER_MERGE_MW_IN_VENDOR for Wikidata build

https://gerrit.wikimedia.org/r/339634

Change 339634 merged by jenkins-bot:
Set MW_COMPOSER_MERGE_MW_IN_VENDOR for Wikidata build

https://gerrit.wikimedia.org/r/339634

Gave it a try on a Wikidata change and the job fails https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/38196/console

00:00:38.703 Generating optimized autoload files
00:00:45.768   [InvalidArgumentException]                                                             
00:00:45.768   Setting extra.merge-plugin.include does not exist or is not supported by this command  
00:00:45.769                                                                                          
00:00:45.769

CI uses composer 1.0.3 which does not support setting arbitrary keys.

1.1.0-RC hints at it though:

  • Added support for editing all top-level properties (name, minimum-stability, ...) as well as extra values via the config command

But we can't update composer since we need PHP 5.5 support :-( Next idea?

WMDE-leszek moved this task from Proposed to Backlog on the Wikidata-Sprint board.

But we can't update composer since we need PHP 5.5 support :-( Next idea?

The issue is/was that composer generates a composer/autoload_static.php file that is for PHP 5.6+ and that cause the php55lint job to choke on it: T135161 . Got fixed with a hack to always filer out that file ( T136021 ).

So most probably we can upgrade composer on CI T125343 . But I have no real clue about all the side effects that will end up having :(

I got composer upgraded to 1.1.0 which let us merge the extra section from mw/core into vendor and thus enable merging of the autoload-dev section.

That is only enabled on the mediawiki/extensions/Wikidata repo for now. The build that happened few minutes ago https://gerrit.wikimedia.org/r/#/c/340698/ fails with some ham crest mismatch. Ie both() is actually found.

Build: https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/38654/console

So I guess almost fixed.

Bad news: other composer based jobs end up being broken :-(

Change 340975 had a related patch set uploaded (by hashar):
[integration/config] Merge mw/core autoload-dev in vendor.git

https://gerrit.wikimedia.org/r/340975

Change 340975 merged by jenkins-bot:
[integration/config] Merge mw/core autoload-dev in vendor.git

https://gerrit.wikimedia.org/r/340975

Change 340976 had a related patch set uploaded (by hashar):
[integration/config] We want to match the project repo name

https://gerrit.wikimedia.org/r/340976

Change 340976 merged by Hashar:
[integration/config] We want to match the project repo name

https://gerrit.wikimedia.org/r/340976

hashar closed this task as Resolved.Mar 3 2017, 2:00 PM
hashar claimed this task.

So that one is fixed. The root cause was that autoload-dev was not merged. Had to upgrade composer meanwhile and resort of some hack in CI config but it is now working.

wikidata build has landed https://gerrit.wikimedia.org/r/340944

Lydia_Pintscher moved this task from Backlog to Done on the Wikidata-Sprint board.Mar 6 2017, 12:59 PM