quibble-fundraising-cldr-REL1_39-php74-docker failure on CLDR master
Closed, ResolvedPublic
Actions

Description

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/cldr/+/884114/

https://integration.wikimedia.org/ci/job/quibble-fundraising-cldr-REL1_39-php74-docker/4/console

00:30:15 [10.9MiB/0.20s] > pre-update-cmd: Wikimedia\Composer\Merge\V2\MergePlugin->onInstallUpdateOrDump
00:30:15 [10.9MiB/0.20s] > pre-update-cmd: ComposerHookHandler::onPreUpdate
00:30:15 [12.2MiB/0.21s] Script ComposerHookHandler::onPreUpdate handling the pre-update-cmd event terminated with an exception
00:30:15 
00:30:15 In autoload_real.php line 59:
00:30:15                                                                                
00:30:15   [ErrorException]                                                             
00:30:15   require(/workspace/src/vendor/composer/../../extensions/DonationInterface/g  
00:30:15   ateway_common/WmfFramework.php): failed to open stream: No such file or dir  
00:30:15   ectory                                                                       
00:30:15                                                                                
00:30:15 
00:30:15 Exception trace:
00:30:15   at /workspace/src/vendor/composer/autoload_real.php:59
00:30:15  Composer\Util\ErrorHandler::handle() at /workspace/src/vendor/composer/autoload_real.php:59
00:30:15  require() at /workspace/src/vendor/composer/autoload_real.php:59
00:30:15  composerRequire_mediawiki_vendor() at /workspace/src/vendor/composer/autoload_real.php:42
00:30:15  ComposerAutoloaderInit_mediawiki_vendor::getLoader() at /workspace/src/vendor/autoload.php:25
00:30:15  require_once() at /workspace/src/includes/AutoLoader.php:327
00:30:15  require_once() at /workspace/src/includes/composer/ComposerHookHandler.php:15
00:30:15  ComposerHookHandler::startAutoloader() at /workspace/src/includes/composer/ComposerHookHandler.php:20
00:30:15  ComposerHookHandler::onPreUpdate() at phar:///usr/bin/composer/src/Composer/EventDispatcher/EventDispatcher.php:439
00:30:15  Composer\EventDispatcher\EventDispatcher->executeEventPhpScript() at phar:///usr/bin/composer/src/Composer/EventDispatcher/EventDispatcher.php:257
00:30:15  Composer\EventDispatcher\EventDispatcher->doDispatch() at phar:///usr/bin/composer/src/Composer/EventDispatcher/EventDispatcher.php:129
00:30:15  Composer\EventDispatcher\EventDispatcher->dispatchScript() at phar:///usr/bin/composer/src/Composer/Installer.php:277
00:30:15  Composer\Installer->run() at phar:///usr/bin/composer/src/Composer/Command/UpdateCommand.php:247
00:30:15  Composer\Command\UpdateCommand->execute() at phar:///usr/bin/composer/vendor/symfony/console/Command/Command.php:298
00:30:15  Symfony\Component\Console\Command\Command->run() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:1040
00:30:15  Symfony\Component\Console\Application->doRunCommand() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:301
00:30:15  Symfony\Component\Console\Application->doRun() at phar:///usr/bin/composer/src/Composer/Console/Application.php:377
00:30:15  Composer\Console\Application->doRun() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:171
00:30:15  Symfony\Component\Console\Application->run() at phar:///usr/bin/composer/src/Composer/Console/Application.php:141
00:30:15  Composer\Console\Application->run() at phar:///usr/bin/composer/bin/composer:88
00:30:15  require() at /usr/bin/composer:29

Details

Subject	Repo	Branch	Lines +/-
Remove fundraising jobs for ParserFunctions and cldr	integration/config	master	+0 -13
Drop composer autoloader listing for WmfFramework	mediawiki/extensions/DonationInterface	master	+0 -3
jjb: [quibble-fundraising-{ext}…] Directly clone each extension dependency	integration/config	master	+9 -1
zuul: Disable fr/REL1_39 gate-and-submit for ParserFunctions	integration/config	master	+3 -2
layout.yaml: Disable FR cldr gate-and-submit job	integration/config	master	+9 -9

Customize query in gerrit

Event Timeline

Reedy created this task.Jan 30 2023, 12:56 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 30 2023, 12:56 AM

Reedy triaged this task as High priority.Jan 30 2023, 12:56 AM

Reedy added a subscriber: Ejegg.Jan 30 2023, 1:00 AM

Wow, that's kind of bizarre - what is composer doing looking for a DonationInterface file on the preUpdate hook? @Jdforrester-WMF do you suppose this is an artifact of the fundraising test suites? Perhaps CLDR should just run the normal tests and not the fundraising-specific ones?

00:30:00 + exec docker run --entrypoint=quibble-with-supervisord --tmpfs /workspace/db:size=320M --volume /srv/jenkins/workspace/quibble-fundraising-cldr-REL1_39-php74-docker/src:/src --volume /srv/jenkins/workspace/quibble-fundraising-cldr-REL1_39-php74-docker/cache:/cache --volume /srv/jenkins/workspace/quibble-fundraising-cldr-REL1_39-php74-docker/log:/workspace/log --volume /srv/git:/srv/git:ro --security-opt seccomp=unconfined --init --rm --label jenkins.job=quibble-fundraising-cldr-REL1_39-php74-docker --label jenkins.build=4 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/quibble-buster-php74:1.4.7-s3 --packages-source composer --db mysql --db-dir /workspace/db --run=phpunit --branch REL1_39 --project-branch mediawiki/extensions/cldr=master --project-branch mediawiki/core=fundraising/REL1_39 --project-branch mediawiki/vendor=fundraising/REL1_39 mediawiki/vendor mediawiki/extensions/cldr

As it's checking out mediawiki/core with fundraising/REL1_39, that does mean DonationInterface is brought in as a git submodule.

The file is referenced in https://github.com/wikimedia/mediawiki-extensions-DonationInterface/blob/9e2ea5f/composer.json#L22

As part of the CI:

00:30:15 INFO:quibble.commands:>>> Start: Create composer.local.json with dependencies ['mediawiki/core', 'mediawiki/extensions/cldr', 'mediawiki/skins/Vector', 'mediawiki/vendor']
00:30:15 INFO:quibble.commands:composer.local.json for merge plugin
00:30:15 INFO:quibble.commands:Created composer.local.json
00:30:15 INFO:quibble.commands:<<< Finish: Create composer.local.json with dependencies ['mediawiki/core', 'mediawiki/extensions/cldr', 'mediawiki/skins/Vector', 'mediawiki/vendor'], in 0.001 s

If we look at the code in quibble..

https://github.com/wikimedia/quibble/blob/6ef19d8902259fb56b9ffccc658e0446d17fe17d/quibble/commands.py#L256-L275

def execute(self):
    log.info('composer.local.json for merge plugin')
    composer_local = os.path.join(
        self.mw_install_path, 'composer.local.json'
    )
    with open(composer_local, 'w') as f:
        json.dump(
            {
                "extra": {
                    "merge-plugin": {
                        "include": [
                            "extensions/*/composer.json",
                            "skins/*/composer.json",
                        ]
                    }
                }
            },
            f,
        )
    log.info('Created composer.local.json')

Which makes Create composer.local.json with dependencies ['mediawiki/core', 'mediawiki/extensions/cldr', 'mediawiki/skins/Vector', 'mediawiki/vendor'] seem misleading.

All that being said, it's then definitely odd that if it's reading a composer.json file from DonationInterface extension, how can extensions/DonationInterface/gateway_common/WmfFramework.php then not exist? The file is in master and REL1_39, and on the same commit...

matmarex subscribed.Feb 13 2023, 2:06 PM

Setting to medium for now, please let us know if it needs higher action while we focus on a short term goal needed for FR.

In case it wasn't clear, this problem is currently blocking all merges in the CLDR extension.

Nikerabbit subscribed.Feb 20 2023, 9:46 AM

Traditionally merge blockers have been triaged as UBN!, though this one is only affecting one extension to my knowledge.

Nikerabbit moved this task from Quarter Backlog to Blocked on the Language-Team (Language-2023-January-March) board.Feb 20 2023, 9:49 AM

In T328240#8616273, @greg wrote:

Setting to medium for now, please let us know if it needs higher action while we focus on a short term goal needed for FR.

- name: mediawiki/extensions/cldr
  template:
    - name: in-wikimedia-production
    - name: in-wikimedia-fundraising-production
    - name: extension-quibble
    - name: extension-coverage
    - name: extension-gate
  # Special extra jobs because Fundraising uses this on PHP74/REL1_39
  test:
    - quibble-fundraising-cldr-REL1_39-php74-docker
  gate-and-submit:
    - quibble-fundraising-cldr-REL1_39-php74-docker

Are we ok to disable the gate-and-submit job here until someone has any time to look at this properly? The test one will still run, and (presumably) fail, but shouldn't block unrelated patches being merged into master of cldr...

As we've got a few patches starting to build up that we can't merge (unless we C+2/V+2 and force)

Change 891281 had a related patch set uploaded (by Reedy; author: Reedy):

[integration/config@master] layout.yaml: Disable FR cldr gate-and-submit job

https://gerrit.wikimedia.org/r/891281

gerritbot added a project: Patch-For-Review.Feb 22 2023, 2:36 PM

Change 891281 merged by jenkins-bot:

[integration/config@master] layout.yaml: Disable FR cldr gate-and-submit job

https://gerrit.wikimedia.org/r/891281

Mentioned in SAL (#wikimedia-releng) [2023-02-22T14:48:01Z] <Reedy> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/891281 T328240

In T328240#8637433, @gerritbot wrote:

Change 891281 merged by jenkins-bot:

[integration/config@master] layout.yaml: Disable FR cldr gate-and-submit job

https://gerrit.wikimedia.org/r/891281

This can be reverted at will. I've left the job (failing) on the test pipeline, which should help/make debugging a bit easier. Jenkins will -1 on test, but if people ignore that job and +2 it will merge the patch, if the other tests pass...

Maintenance_bot removed a project: Patch-For-Review.Feb 22 2023, 3:10 PM

The job quibble-fundraising-cldr-REL1_39-php74-docker triggers for the cldr MediaWiki extension master branch to ensure it is working with DonationInterface. That was done several years ago because DonationInterface depended on the master branch of cldr, FundraisingEmailUnsubscribe and ParserFunctions.

The issue is PHP is unable to find a file:

require(/workspace/src/vendor/composer/../../extensions/DonationInterface/gateway_common/WmfFramework.php)

It is thus looking for a file provided by DonationInterface, but the repository is NOT cloned by the job. That is because CI intentionally does not process submodules when it clones mediawiki/core (which in fundraising/REL1_39 has git submodules), instead it is expected CI clones each repositories individually and to run the tests. There is also a large mix up of branches being used to compose the MediaWiki deployment for fundraising, mediawiki/core fundraising/REL1_39 has:

CodeEditor	REL1_39
ParserFunctions	REL1_39
SyntaxHighlight_GeSHi	REL1_39
WikiEditor	REL1_39
Vector	REL1_39
vendor	fundraising/REL1_39
cldr	REL1_39
DonationInterface	deployment
FundraisingEmailUnsubscribe	master

In the ideal world:

all those repositories would use the same branch (fundraising/REL1_39)
for each of those repositories CI would trigger a common job which clones each of those repositories and run the tests (the same way we have the wmf-quibble-* jobs being triggered by a set of a repositories to ensure they work well together)

Then we can get Gerrit to automatically update the submodules from mediawiki/core fundraising/REL1_39 which is what we are doing for the wmf/* branches.

Or in short, I think the quibble-fundraising-* jobs are broken cause they don't trigger for all repositories or are missing some repository (such as DonationInterface).

Change 897068 had a related patch set uploaded (by Majavah; author: Majavah):

[integration/config@master] zuul: Disable fr/REL1_39 gate-and-submit for ParserFunctions

https://gerrit.wikimedia.org/r/897068

Change 897068 merged by jenkins-bot:

[integration/config@master] zuul: Disable fr/REL1_39 gate-and-submit for ParserFunctions

https://gerrit.wikimedia.org/r/897068

Mentioned in SAL (#wikimedia-releng) [2023-03-12T10:23:01Z] <taavi> reload zuul for https://gerrit.wikimedia.org/r/897068 T328240

Maintenance_bot removed a project: Patch-For-Review.Mar 12 2023, 10:30 AM

jhsoby subscribed.Mar 17 2023, 10:21 AM

XenoRyet edited projects, added Fundraising-Backlog; removed Fundraising Tech - Chaos Crew.Mar 21 2023, 1:55 AM

AKanji-WMF moved this task from Triage to Blocked or not fr-tech on the Fundraising-Backlog board.Mar 27 2023, 8:35 PM

Pginer-WMF edited projects, added Language-Team (Language-2023-April-June); removed Language-Team (Language-2023-January-March).Mar 31 2023, 1:24 PM

Pginer-WMF moved this task from Quarter Backlog to Blocked on the Language-Team (Language-2023-April-June) board.

Noting this is still a problem...

Fails on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/cldr/+/915801 / https://integration.wikimedia.org/ci/job/quibble-fundraising-cldr-REL1_39-php74-docker/43/console

Pginer-WMF edited projects, added Language-Team (Language-2023-July-September); removed Language-Team (Language-2023-April-June).Jun 30 2023, 11:45 AM

Pginer-WMF moved this task from Quarter Backlog to Blocked on the Language-Team (Language-2023-July-September) board.

AKanji-WMF moved this task from Blocked or not fr-tech to DRI Backlog on the Fundraising-Backlog board.Jul 5 2023, 4:30 PM

AKanji-WMF added a project: Fundraising Tech - Chaos Crew.

Winston_Sung subscribed.Aug 9 2023, 10:06 AM

Change 948653 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[integration/config@master] zuul: Disable quibble-fundraising-*-REL1_39-php74-docker from test runs

https://gerrit.wikimedia.org/r/948653

gerritbot added a project: Patch-For-Review.Aug 14 2023, 7:45 PM

In T328240#8573141, @Reedy wrote:

As it's checking out mediawiki/core with fundraising/REL1_39, that does mean DonationInterface is brought in as a git submodule.

The submodules are not cloned, at least the logs does not show that.

The file is referenced in https://github.com/wikimedia/mediawiki-extensions-DonationInterface/blob/9e2ea5f/composer.json#L22

No, it is also in the branch of mediawiki/vendor - https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/vendor/+/refs/heads/fundraising/REL1_39/composer/autoload_classmap.php#3395

All that being said, it's then definitely odd that if it's reading a composer.json file from DonationInterface extension, how can extensions/DonationInterface/gateway_common/WmfFramework.php then not exist? The file is in master and REL1_39, and on the same commit...

It needs to clone the submodules or the dependencies must be explicit set up.

greg raised the priority of this task from Medium to High.Sep 1 2023, 4:47 PM

jgleeson moved this task from Backlog to In Progress on the Fundraising Tech - Chaos Crew board.Sep 5 2023, 1:53 PM

Michael subscribed.Sep 14 2023, 12:02 PM

The usage of CLDR in DonationInterface is fairly limited - We use LanguageNames::getName() on an email preference form, and for payments forms we're getting the name of the country in the user's language for a message in the footer that is only shown in France.

"Notre statut d’exonération varie selon la législation de chaque pays. Les dons à la Wikimedia Foundation ne sont pas déductibles des impôts en France."

I would be fine with just dropping the fundraising branch tests for commits in the cldr extension, and depending on tests run against the DonationInterface commits to ensure that cldr updates haven't broken compatibility there.

Change 948653 abandoned by Umherirrender:

[integration/config@master] jjb: [quibble-fundraising-{ext}…] Directly clone each extension dependency

Reason:

https://gerrit.wikimedia.org/r/948653

Maintenance_bot removed a project: Patch-For-Review.Sep 17 2023, 6:10 PM

Change 959251 had a related patch set uploaded (by Ejegg; author: Ejegg):

[integration/config@master] Remove fundraising jobs for ParserFunctions and cldr

https://gerrit.wikimedia.org/r/959251

gerritbot added a project: Patch-For-Review.Sep 20 2023, 2:41 PM

Ejegg moved this task from In Progress to Ready for Review on the Fundraising Tech - Chaos Crew board.Sep 20 2023, 2:43 PM

Change 961146 had a related patch set uploaded (by Ejegg; author: Ejegg):

[mediawiki/extensions/DonationInterface@master] Drop composer autoloader listing for WmfFramework

https://gerrit.wikimedia.org/r/961146

Change 961146 merged by jenkins-bot:

[mediawiki/extensions/DonationInterface@master] Drop composer autoloader listing for WmfFramework

https://gerrit.wikimedia.org/r/961146

Oh hey, looks like it was enough to just get that file out of DonationInterface's composer.json. Thanks so much for the suggestion @jgleeson ! This cldr patch just passed the fundraising branch tests:
https://integration.wikimedia.org/ci/job/quibble-fundraising-cldr-REL1_39-php74-docker/103/console