Page MenuHomePhabricator

Migrate DonationInterface CI job to Docker
Closed, ResolvedPublic

Description

For mediawiki/extensions/DonationInterface we have specific jobs. They use specific branches and specially run patchsets against REL1_27 branch of MediaWiki (among other tricks).

Moreover, the tests are running with PHP Zend 5.6.

Jobs to migrate:

mwext-donationinterfacecore-REL1_27-zend56-jessie
mwext-donationinterfacecore-REL1_28-zend56-jessie

Dummy test change: https://gerrit.wikimedia.org/r/#/c/324066

Repro simulating a CI job for a DonationInterface patch:

ZUUL_PROJECT=mediawiki/extensions/DonationInterface quibble \
  --packages-source composer \
  --db mysql \
  --run=phpunit \
  --branch REL1_27 \
  --project-branch mediawiki/extensions/DonationInterface=master \
  --project-branch mediawiki/vendor=fundraising/REL1_27 \
  mediawiki/vendor \
  mediawiki/extensions/DonationInterface \
  mediawiki/extensions/FundraisingEmailUnsubscribe \
  mediawiki/extensions/ContributionTracking \
  mediawiki/extensions/ParserFunctions \
  mediawiki/extensions/cldr

Note: the job clones mediawiki/vendor but also run composer install from the root of mediawiki/core which generates a file for the composer merge plugin:

composer.local.json
{
   "extra" : {
      "merge-plugin" : {
         "include" : [
            "extensions/DonationInterface/composer.json",
            "extensions/FundraisingEmailUnsubscribe/composer.json",
            "extensions/ContributionTracking/composer.json",
            "extensions/ParserFunctions/composer.json",
            "extensions/cldr/composer.json"
         ]
      }
   }
}

Details

Related Gerrit Patches:
integration/config : masterDonationInterface job is now passing fine
mediawiki/extensions/DonationInterface : masterTest merge plugin prepend-repositories
mediawiki/vendor : fundraising/REL1_27Sync composer.json/composer.lock
mediawiki/vendor : fundraising/REL1_27Update wikimedia/composer-merge-plugin to 1.4.1
mediawiki/core : fundraising/REL1_27Use composer-merge-plugin 1.4.1 to prepend VCS repos
integration/config : masterMigrate DonationInterface to Docker job (non voting)
integration/config : mastermediawiki/core has fundraising/RELxxx branches
integration/config : masterFix DonationInterface skip-if rule
integration/config : masterAdd experimental DonationInterface job mediawiki
integration/config : masterDocker jobs for DonationInterface
integration/config : masterdocker: quibble-jessie-php56 fix package name
integration/config : masterdocker: Quibble container with PHP 5.6

Event Timeline

hashar triaged this task as Medium priority.Aug 29 2018, 1:26 PM
hashar created this task.

Change 456141 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: Quibble container with PHP 5.6

https://gerrit.wikimedia.org/r/456141

Change 456141 merged by jenkins-bot:
[integration/config@master] docker: Quibble container with PHP 5.6

https://gerrit.wikimedia.org/r/456141

Change 456148 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Docker jobs for DonationInterface

https://gerrit.wikimedia.org/r/456148

Change 456152 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: quibble-jessie-php56 fix package name

https://gerrit.wikimedia.org/r/456152

Change 456152 merged by jenkins-bot:
[integration/config@master] docker: quibble-jessie-php56 fix package name

https://gerrit.wikimedia.org/r/456152

Change 456148 merged by jenkins-bot:
[integration/config@master] Docker jobs for DonationInterface

https://gerrit.wikimedia.org/r/456148

hashar updated the task description. (Show Details)Aug 29 2018, 2:47 PM

I have deployed the new container that ships Zend php 5.6. I have added the new job to Zuul and it can be triggered on any change by commenting check experimental. I did that on the dummy change #324066 and it fails:

https://integration.wikimedia.org/ci/job/quibble-donationinterface-REL1_27-zend56-docker/2/console

1) AmazonApiTest::testDoPaymentSuccess
ReflectionException: Class PayWithAmazon\Mocks\MockPaymentsClient does not exist

/workspace/src/vendor/wikimedia/smash-pig/Core/Configuration.php:178
/workspace/src/extensions/DonationInterface/amazon_gateway/amazon.adapter.php:164
/workspace/src/extensions/DonationInterface/amazon_gateway/amazon.adapter.php:134
/workspace/src/extensions/DonationInterface/amazon_gateway/amazon.api.php:50
/workspace/src/includes/api/ApiMain.php:1362
/workspace/src/includes/api/ApiMain.php:437
/workspace/src/tests/phpunit/includes/api/ApiTestCase.php:112
/workspace/src/extensions/DonationInterface/tests/phpunit/Adapter/Amazon/AmazonApiTest.php:43
/workspace/src/tests/phpunit/MediaWikiTestCase.php:156
/workspace/src/tests/phpunit/phpunit.php:262

2) AmazonApiTest::testDoPaymentErrors
ReflectionException: Class PayWithAmazon\Mocks\MockPaymentsClient does not exist

/workspace/src/vendor/wikimedia/smash-pig/Core/Configuration.php:178
/workspace/src/extensions/DonationInterface/tests/phpunit/Adapter/Amazon/AmazonApiTest.php:81
/workspace/src/tests/phpunit/MediaWikiTestCase.php:156
/workspace/src/tests/phpunit/phpunit.php:262

3) DonationInterface_Adapter_Amazon_Test::testDoPaymentSuccess
ReflectionException: Class PayWithAmazon\Mocks\MockPaymentsClient does not exist

/workspace/src/vendor/wikimedia/smash-pig/Core/Configuration.php:178
/workspace/src/extensions/DonationInterface/amazon_gateway/amazon.adapter.php:164
/workspace/src/extensions/DonationInterface/amazon_gateway/amazon.adapter.php:134
/workspace/src/extensions/DonationInterface/tests/phpunit/Adapter/Amazon/AmazonTest.php:154
/workspace/src/tests/phpunit/MediaWikiTestCase.php:156
/workspace/src/tests/phpunit/phpunit.php:262

4) DonationInterface_Adapter_Amazon_Test::testDoPaymentDeclined
ReflectionException: Class PayWithAmazon\Mocks\MockPaymentsClient does not exist

/workspace/src/vendor/wikimedia/smash-pig/Core/Configuration.php:178
/workspace/src/extensions/DonationInterface/tests/phpunit/Adapter/Amazon/AmazonTest.php:183
/workspace/src/tests/phpunit/MediaWikiTestCase.php:156
/workspace/src/tests/phpunit/phpunit.php:262

5) DonationInterface_Adapter_Amazon_Test::testFailOnAmazonRejected
ReflectionException: Class PayWithAmazon\Mocks\MockPaymentsClient does not exist

/workspace/src/vendor/wikimedia/smash-pig/Core/Configuration.php:178
/workspace/src/extensions/DonationInterface/tests/phpunit/Adapter/Amazon/AmazonTest.php:210
/workspace/src/tests/phpunit/MediaWikiTestCase.php:156
/workspace/src/tests/phpunit/phpunit.php:262

6) DonationInterface_Adapter_Amazon_Test::testTransactionTimedOut
ReflectionException: Class PayWithAmazon\Mocks\MockPaymentsClient does not exist

/workspace/src/vendor/wikimedia/smash-pig/Core/Configuration.php:178
/workspace/src/extensions/DonationInterface/tests/phpunit/Adapter/Amazon/AmazonTest.php:235
/workspace/src/tests/phpunit/MediaWikiTestCase.php:156
/workspace/src/tests/phpunit/phpunit.php:262

7) DonationInterface_Adapter_Amazon_Test::testClientException
ReflectionException: Class PayWithAmazon\Mocks\MockPaymentsClient does not exist

/workspace/src/vendor/wikimedia/smash-pig/Core/Configuration.php:178
/workspace/src/extensions/DonationInterface/tests/phpunit/Adapter/Amazon/AmazonTest.php:261
/workspace/src/tests/phpunit/MediaWikiTestCase.php:156
/workspace/src/tests/phpunit/phpunit.php:262

8) DonationInterface_Adapter_Amazon_Test::testDoRecurringPaymentSuccess
ReflectionException: Class PayWithAmazon\Mocks\MockPaymentsClient does not exist

/workspace/src/vendor/wikimedia/smash-pig/Core/Configuration.php:178
/workspace/src/extensions/DonationInterface/amazon_gateway/amazon.adapter.php:164
/workspace/src/extensions/DonationInterface/amazon_gateway/amazon.adapter.php:134
/workspace/src/extensions/DonationInterface/tests/phpunit/Adapter/Amazon/AmazonTest.php:290
/workspace/src/tests/phpunit/MediaWikiTestCase.php:156
/workspace/src/tests/phpunit/phpunit.php:262

Seems like an issue with composer and autoloading :-(

hashar updated the task description. (Show Details)Aug 29 2018, 3:38 PM
hashar updated the task description. (Show Details)

The original job relies on mediawiki/vendor and then run composer install inside the DonationInterface extension:

cd $WORKSPACE/src/extensions/DonationInterface
composer install

Then the MediaWiki install is run.

Quibble only supports using the composer merge plugin to ship dependencies (via composer update).

I guess I will need a hack to handle DonationInterface as a special edge case.

Based on a discussion with @Ejegg , the issues would be because of amzn/login-and-pay-with-amazon-sdk-php. DonationInterface composer.json has:

"require": {
    "amzn/login-and-pay-with-amazon-sdk-php": "dev-master",
},
"repositories": [
    {
        "type": "git",
        "url": "https://github.com/ejegg/login-and-pay-with-amazon-sdk-php"
    }
],

And the composer.lock has:

{
    "name": "amzn/login-and-pay-with-amazon-sdk-php",
    "version": "dev-master",
    "source": {
        "type": "git",
        "url": "https://github.com/ejegg/login-and-pay-with-amazon-sdk-php",
        "reference": "e3f8d1082556104526999740beaf55b481aaf323"
    },

But when installing with the composer merge plugin, the repository is apparently ignored and it ends up doing:

Installing amzn/login-and-pay-with-amazon-sdk-php (dev-master 272469d): [257.2MB/29.88s]

That is dev-master on https://packagist.org/packages/amzn/login-and-pay-with-amazon-sdk-php#dev-master states 272469d62dfe8e027fa091442825c7d9ce449af4

That is apparently not the last commit of upstream lib https://github.com/amzn/amazon-pay-sdk-php/commits/master

Change 459762 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@fundraising/REL1_27] Composer merge plugin now prepends VCS repositories

https://gerrit.wikimedia.org/r/459762

Change 459765 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/extensions/DonationInterface@master] Test merge plugin prepend-repositories

https://gerrit.wikimedia.org/r/459765

Change 459767 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Add experimental DonationInterface job mediawiki

https://gerrit.wikimedia.org/r/459767

Change 459767 merged by jenkins-bot:
[integration/config@master] Add experimental DonationInterface job mediawiki

https://gerrit.wikimedia.org/r/459767

So that will be solved by bumping wikimedia/composer-merge-plugin from 1.3.1 to 1.4.1. It would prepend the repositories before packagist which thus cause composer to prefer the VCS version over packagist.

I gave it a try locally and it installs:

Installing amzn/login-and-pay-with-amazon-sdk-php (dev-master e3f8d10)

Which is the master branch of https://github.com/ejegg/login-and-pay-with-amazon-sdk-php

https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/459762 Use composer-merge-plugin 1.4.1 to prepend VCS repos

I could not figure out how to update mediawiki/vendor@fundraising/REL1_27. composer always install unrelated dependencies. @Ejegg could update vendor please? :]

And to confirm, I gave it a try with:

mediawiki/corefundraising/REL1_27Gerrit 459762Bump composer-merge-plugin to 1.4.1
mediawiki/vendorfundraising/REL1_26Gerrit 459792BAD change to update merge plugin
DonationInterfacemasterGerrit 459765Dummy with Depends-On to two previous changes

A check experimental on the DonationInterface change triggered https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/DonationInterface/+/459765/ which is passing since it did:

  • Installing amzn/login-and-pay-with-amazon-sdk-php (dev-master e3f8d10)

So I guess we want to merge the mediawiki/core patch https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/459762/ and then one has to craft an update for mediawiki/vendor. Once done, I will switch CI to use the Docker based job :]

Change 459792 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/vendor@fundraising/REL1_27] Update wikimedia/composer-merge-plugin to 1.4.1

https://gerrit.wikimedia.org/r/459792

Change 462888 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/vendor@fundraising/REL1_27] Sync composer.json/composer.lock

https://gerrit.wikimedia.org/r/462888

Change 467682 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Fix DonationInterface skip-if rule

https://gerrit.wikimedia.org/r/467682

Change 467682 merged by jenkins-bot:
[integration/config@master] Fix DonationInterface skip-if rule

https://gerrit.wikimedia.org/r/467682

Change 468018 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] mediawiki/core has fundraising/RELxxx branches

https://gerrit.wikimedia.org/r/468018

Change 468018 merged by jenkins-bot:
[integration/config@master] mediawiki/core has fundraising/RELxxx branches

https://gerrit.wikimedia.org/r/468018

quibble-donationinterface-REL1_27-zend56-docker SUCCESS in 3m 33s \o/

That is using a dummy change https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/DonationInterface/+/459765/ that has the required dependencies.

Change 473181 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate DonationInterface to Docker job (non voting)

https://gerrit.wikimedia.org/r/473181

Change 473181 merged by jenkins-bot:
[integration/config@master] Migrate DonationInterface to Docker job (non voting)

https://gerrit.wikimedia.org/r/473181

hashar removed hashar as the assignee of this task.Mar 19 2019, 8:39 PM

There are patch aboves, including one to update the composer merge plugin. I haven't had any review or feedback on those though.

Anyway I have too many things to do and I am not actively working on this task.

Change 459762 abandoned by Reedy:
Use composer-merge-plugin 1.4.1 to prepend VCS repos

Reason:
No one cares about fundraising/REL1_27 anymore :)

https://gerrit.wikimedia.org/r/459762

hashar closed this task as Resolved.Apr 11 2019, 7:06 PM
hashar claimed this task.

REL1_27 and the outdated composer-merge-plugin are gone.

DonationInterface is now using REL1_31 / Quibble / php7 on Docker https://gerrit.wikimedia.org/r/#/c/integration/config/+/499334/

\o/

Change 459792 abandoned by Hashar:
Update wikimedia/composer-merge-plugin to 1.4.1

Reason:
No more needed. Got migrated to fundraising/REL1_31 https://gerrit.wikimedia.org/r/#/c/integration/config/ /499334/

https://gerrit.wikimedia.org/r/459792

Change 462888 abandoned by Hashar:
Sync composer.json/composer.lock

Reason:
No more needed. Got migrated to fundraising/REL1_31 https://gerrit.wikimedia.org/r/#/c/integration/config/ /499334/

https://gerrit.wikimedia.org/r/462888

Change 459765 abandoned by Hashar:
Test merge plugin prepend-repositories

Reason:
No more needed. Got migrated to fundraising/REL1_31 https://gerrit.wikimedia.org/r/#/c/integration/config/ /499334/

https://gerrit.wikimedia.org/r/459765

Change 533205 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] DonationInterface job is now passing fine

https://gerrit.wikimedia.org/r/533205

Change 533205 merged by jenkins-bot:
[integration/config@master] DonationInterface job is now passing fine

https://gerrit.wikimedia.org/r/533205