Page MenuHomePhabricator

Fatal error: Call to undefined function Wikibase\Client\Tests\RecentChanges\both() - on jenkins
Closed, ResolvedPublic

Description

Wikidata build tests are failing because hamcrest functions are not being autoloaded (not included in mediawiki's vendor/composer/autoload_files.php)

https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/37857/console

These are being autoloaded for Wikibase tests (alone, outside the build). The issue seems to be just with the build. It works locally for me.

10:02:33 Fatal error: Call to undefined function Wikibase\Client\Tests\RecentChanges\both() in /srv/jenkins-workspace/workspace/mwext-testextension-hhvm/src/extensions/Wikidata/extensions/Wikibase/client/tests/phpunit/includes/RecentChanges/ChangeLineFormatterTest.php on line 177

The slave script bin/mw-fetch-composer-dev.sh apparently manage to inject the dev dependencies and they get installed:

00:00:48.043   - Installing hamcrest/hamcrest-php (v2.0.0)
00:00:48.045     Loading from cache
00:00:48.045     Extracting archive
00:00:48.123 
00:00:48.137   - Installing wmde/hamcrest-html-matchers (v0.1.0)
00:00:48.138     Loading from cache
00:00:48.138     Extracting archive

vendor/composer/autoload_files.php and or composer.lock might end up missing them though :(

Reproduction

mediawiki/core
mediawiki/vendor cloned in the vendor subdirectory
integration/jenkins

The CI script mw-composer-fetch-dev.sh grabs the list of packages from require-dev and then require them in the vendor subdirectory.

The dev dependencies:

(cd vendor; ~/projects/integration/jenkins/tools/composer-dev-args.js ~/projects/mediawiki/core/composer.json)
/Users/amusso/projects/mediawiki/core/vendor
composer/spdx-licenses=1.1.4
jakub-onderka/php-parallel-lint=0.9.2
justinrainbow/json-schema=~3.0
mediawiki/mediawiki-codesniffer=0.7.2
jetbrains/phpstorm-stubs=dev-master#1b9906084d6635456fcf3f3a01f0d7d5b99a578a
monolog/monolog=~1.18.2
nikic/php-parser=2.1.0
nmred/kafka-php=0.1.5
phpunit/phpunit=4.8.31
wikimedia/avro=1.7.7
hamcrest/hamcrest-php=^2.0
wmde/hamcrest-html-matchers=^0.1.0

For each we invoke composer require --dev --ansi --no-progress --prefer-dist -v . The result is available as https://gerrit.wikimedia.org/r/339404

Then (cd vendor && composer dump-autoload --optimize) which apparently is a noop.

Given a dummy test file:

tests/phpunit/BlaTest.php
<?php

use Hamcrest\Matchers;

class BlaTest extends MediaWikiTestCase {

    function testHamcrest() {
        both();
    }

}

That fails to load one of Hamcrest function:

$ php tests/phpunit/phpunit.php tests/phpunit/BlaTest.php
PHPUnit 4.8.31 by Sebastian Bergmann and contributors.

PHP Fatal error:  Call to undefined function both() in /Users/amusso/projects/mediawiki/core/tests/phpunit/BlaTest.php on line 8

Case that works

Instead of trying to lookup the static method via namespace, if we refer to the canonical class path it seems to work:

<?php
class CanonicalPathTest extends MediaWikiTestCase {

    function testHamcrest() {
        \Hamcrest\Matchers::both();
    }
}

Global functions

One has to explicitly register them with Hamcrest\Util::registerGlobalFunctions();:

<?php
Hamcrest\Util::registerGlobalFunctions();
class BlaTest extends MediaWikiTestCase {
    function testHamcrest() {
        assertThat();
    }
}

Other findings

vendor/composer/autoload_files.php refers to $vendorDir . '/mediawiki/at-ease/src/Functions.php'. That is because at-ease composer.json has:

"autoload": {
    "files": [
        "src/Functions.php"
    ]
}

Event Timeline

I invoke @Addshore and @Aleksey_WMDE since that is ham crest related. Gotta reproduce and figure out why ham crest is not loaded when running wikidata build with mediawiki/vendor although CI is supposed to fetch the dev dependencies :/

Change 339051 had a related patch set uploaded (by Hashar):
Capture composer.lock after mw-fetch-composer-dev

https://gerrit.wikimedia.org/r/339051

hamcrest-php and hamcrest-html-matchers are dev dependencies of mediawiki core.

Jenkins uses mediawiki/vendor (https://github.com/wikimedia/mediawiki-vendor/) and then dev dependencies of core are added with a script:

https://github.com/wikimedia/integration-jenkins/blob/master/bin/mw-fetch-composer-dev.sh

the jenkins workspace is shared / reused across different builds/jobs, so the issue might be related to that.

Jenkins is using a specific (older) version of composer: https://github.com/wikimedia/integration-composer

I tried locally with the same version of composer and the build (and with mediawiki/vendor) and couldn't reproduce the problem. Also, in other test runs on jenkins (e.g. Wikibase extension), hamcrest is installed and works without any issues.

think the issue is just that the script https://github.com/wikimedia/integration-jenkins/blob/master/tools/composer-dev-args.js only copies over the require-dev section from mediawiki's composer.json (piped to composer require)

this doesn't include autoload-dev. Think we need the script to also copy these over to mediawiki-vendor's composer.json

Change 339202 had a related patch set uploaded (by Aude):
Add script to copy composer require-dev and autoload-dev to mw vendor

https://gerrit.wikimedia.org/r/339202

hashar updated the task description. (Show Details)

So I don't think the PHP namespace lookup works for static methods. Eg both() would not end up being resolved as \HamCrest\Matchers::both(). A lame test case:

A namespaced class definition:

<?php
namespace namespaced {
    class StaticClass {
        public static function hello() {
            return "Hello";
        }
    }
}

Script using it:

<?php
use namespaced\StaticClass;

spl_autoload_register( function () { 
    include 'namespaced.php';
} );

print "namespaced\StaticClass::hello():\n";
print ">>> " . namespaced\StaticClass::hello() . "\n";  // works
print "hello():\n";
print ">>> " . hello() . "\n";  // Call to undefined function hello() 

Output with PHP 5.6.29:

$ php exec.php
namespaced\StaticClass::hello():
>>> Hello
hello():
PHP Fatal error:  Call to undefined function hello() in /tmp/exec.php on line 11
PHP Stack trace:
PHP   1. {main}() /tmp/exec.php:0

Fatal error: Call to undefined function hello() in /tmp/exec.php on line 11

Call Stack:
    0.0002     231576   1. {main}() /tmp/exec.php:0

That might why Hamcrest has a Hamcrest\Util::registerGlobalFunctions();: which include a file defining a set of global functions which in turns invoke the static method using the full canonical path. Extracted from the file:

vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest.php
<?php
if (!function_exists('both')) {
    function both(\Hamcrest\Matcher $matcher)
    {
        return \Hamcrest\Core\CombinableMatcher::both($matcher);
    }
}

So potentially in our tests, we can explicitly register the global functions using:

\Hamcrest\Util::registerGlobalFunctions();:

And can then use both().

Which works around the following PHP code that fails to lookup up the both() static method:

use Hamcrest\Matchers;
both();  // fails

Change 339051 abandoned by Hashar:
Capture composer.lock after mw-fetch-composer-dev

Reason:
The script knows about LOG_DIR , so I will archive the files directly from it in https://gerrit.wikimedia.org/r/#/c/339202/

https://gerrit.wikimedia.org/r/339051

Change 339202 merged by jenkins-bot:
Option to merge mw composer into vendor

https://gerrit.wikimedia.org/r/339202

Mentioned in SAL (#wikimedia-releng) [2017-02-24T13:52:35Z] <hashar> deployed slave script update to be able to merge mediawiki/composer.json into vendor/composer.json 6527f49..a7728a5 https://gerrit.wikimedia.org/r/#/c/339202/ T158674

Mentioned in SAL (#wikimedia-releng) [2017-02-24T13:56:03Z] <hashar> Log refresh Nodepool instances to deploy slave script update to be able to merge mediawiki/composer.json into vendor/composer.json 6527f49..a7728a5 https://gerrit.wikimedia.org/r/#/c/339202/ T158674

Change 339634 had a related patch set uploaded (by Hashar):
Set MW_COMPOSER_MERGE_MW_IN_VENDOR for Wikidata build

https://gerrit.wikimedia.org/r/339634

Change 339634 merged by jenkins-bot:
Set MW_COMPOSER_MERGE_MW_IN_VENDOR for Wikidata build

https://gerrit.wikimedia.org/r/339634

Gave it a try on a Wikidata change and the job fails https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/38196/console

00:00:38.703 Generating optimized autoload files
00:00:45.768   [InvalidArgumentException]                                                             
00:00:45.768   Setting extra.merge-plugin.include does not exist or is not supported by this command  
00:00:45.769                                                                                          
00:00:45.769

CI uses composer 1.0.3 which does not support setting arbitrary keys.

1.1.0-RC hints at it though:

  • Added support for editing all top-level properties (name, minimum-stability, ...) as well as extra values via the config command

But we can't update composer since we need PHP 5.5 support :-( Next idea?

But we can't update composer since we need PHP 5.5 support :-( Next idea?

The issue is/was that composer generates a composer/autoload_static.php file that is for PHP 5.6+ and that cause the php55lint job to choke on it: T135161 . Got fixed with a hack to always filer out that file ( T136021 ).

So most probably we can upgrade composer on CI T125343 . But I have no real clue about all the side effects that will end up having :(

I got composer upgraded to 1.1.0 which let us merge the extra section from mw/core into vendor and thus enable merging of the autoload-dev section.

That is only enabled on the mediawiki/extensions/Wikidata repo for now. The build that happened few minutes ago https://gerrit.wikimedia.org/r/#/c/340698/ fails with some ham crest mismatch. Ie both() is actually found.

Build: https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/38654/console

So I guess almost fixed.

Bad news: other composer based jobs end up being broken :-(

Change 340975 had a related patch set uploaded (by hashar):
[integration/config] Merge mw/core autoload-dev in vendor.git

https://gerrit.wikimedia.org/r/340975

Change 340975 merged by jenkins-bot:
[integration/config] Merge mw/core autoload-dev in vendor.git

https://gerrit.wikimedia.org/r/340975

Change 340976 had a related patch set uploaded (by hashar):
[integration/config] We want to match the project repo name

https://gerrit.wikimedia.org/r/340976

Change 340976 merged by Hashar:
[integration/config] We want to match the project repo name

https://gerrit.wikimedia.org/r/340976

hashar claimed this task.

So that one is fixed. The root cause was that autoload-dev was not merged. Had to upgrade composer meanwhile and resort of some hack in CI config but it is now working.

wikidata build has landed https://gerrit.wikimedia.org/r/340944