Page MenuHomePhabricator

composer stops quibble and requires input
Closed, ResolvedPublicBUG REPORT

Description

I'm using the included file to reproduce some errors seen in jenkins+quibble but, instead of running without requiring intervention (as I would expect), quibble stops part way through because of a composer prompt. Pressing <enter> twice gets past the prompt and continues. Only after it continues do I see the following that indicate the problem:

Do you want to move these requirements? [no]? Do you want to re-run the command without --dev? [yes]? ./composer.json has been updated

If I redirect stdin so that it reads from /dev/null (in the vain hope that it composer would see that it isn't on an interactive terminal and not prompt), it still pauses but pressing <enter> no longer works.

I would suggest adding --no-interaction to the composer invocations.

#!/bin/sh -e

DOCKER_IMG=docker-registry.wikimedia.org/releng/quibble-buster-php74
DOCKER_VERSION=1.6.0
DEPENDENCIES="mediawiki/extensions/SocialProfile mediawiki/skins/Vector"

cat <<-EOF > quibble.env
	ZUUL_VOTING=1
	MW_COMPOSER_MERGE_MW_IN_VENDOR=1
	EXT_NAME=CommentStreams
	ZUUL_PIPELINE=test
	ZUUL_BRANCH=master
	ZUUL_URL=git://contint2001.wikimedia.org
	ZUUL_PROJECT=mediawiki/extensions/CommentStreams
	ZUUL_REF=refs/zuul/master/Z98a5d9b0bc254f5b8a9ae3db2270f09b
	ZUUL_COMMIT=80c4478ecb513ca6556fc9e79a462ad54300e62f
EOF

podman run -it --rm \
  --env-file ./quibble.env \
  -v "$(pwd)"/cache:/cache \
  -v "$(pwd)"/cache:/nonexistent \
  -v "$(pwd)"/log:/log \
  -v "$(pwd)"/ref:/srv/git:ro \
  -v "$(pwd)"/docker-src:/workspace/src \
  "${DOCKER_IMG}":"${DOCKER_VERSION}" \
  $DEPENDENCIES

Event Timeline

Change 986672 had a related patch set uploaded (by MarkAHershberger; author: MarkAHershberger):

[integration/quibble@master] Make config non-interactive

https://gerrit.wikimedia.org/r/986672

Change 986672 abandoned by Hashar:

[integration/quibble@master] Make config non-interactive

Reason:

Lets first find out why composer asks a question, I guess something is broken earlier in the chain and using that non interactive flag merely hide the error instead of addressing it :-]

https://gerrit.wikimedia.org/r/986672

May you past the composer output? Reading the composer code in src/Composer/Command/RequireCommand.php it should shows some warnings:

'%s is currently present in the %s key and you ran the command %s the --dev flag, which will move it to the %s key.',

It says the composer.json got updated, I guess there are some differences being introduced? At quick glance, that prompt got added as part of https://github.com/composer/composer/issues/9516:

Warn about making dev dependency prod dependency

What I've seen happen in our projects several times now is that when people upgrade a package to a newer version they use composer require to get the latest version. This is fine, but sometimes they forget --dev for packages that were in dev dependencies, or sometimes --dev is added out of habit and it shouldn't be (it should be a production dependency).

Is there a way that composer can at least warn when a package is moved from dev dependency to prod dependency or vice versa?

And one of the comment stating:

Sounds to me like their habit should be to run composer update foo/bar rather than require..

And I think the prompt comes from when Quibble injects composer dev dependencies on top of mediawiki/vendor ones. The code being:

quibble/commands.py
class VendorComposerDependencies:
    def __init__(self, mw_install_path, log_dir):
        self.mw_install_path = mw_install_path
        self.log_dir = log_dir

    def execute(self):
        log.info('mediawiki/vendor is used, add require-dev dependencies')
        mw_composer_json = os.path.join(self.mw_install_path, 'composer.json')
        vendor_dir = os.path.join(self.mw_install_path, 'vendor')
        with open(mw_composer_json, 'r') as f:
            composer = json.load(f)

        reqs = [
            '='.join([dependency, version])
            for dependency, version in composer['require-dev'].items()
        ]

        log.debug('composer require --dev %s', ' '.join(reqs))
        composer_require = [
            'composer',
            'require',
            '--dev',
            '--ansi',
            '--no-progress',
            '--prefer-dist',
            '-v',
        ]
        composer_require.extend(reqs)

        run(composer_require, cwd=vendor_dir)

Which does a composer require --dev of mediawiki/core has in require-dev

Looking at mediawiki/core:

composer.json
"require-dev": {
     "composer/spdx-licenses": "1.5.7",
     "doctrine/dbal": "3.4.2",
     "doctrine/sql-formatter": "1.1.1",
     "ext-simplexml": "*",
     "giorgiosironi/eris": "^0.14.0",
     "hamcrest/hamcrest-php": "^2.0",
     "johnkary/phpunit-speedtrap": "^4.0",
     "mediawiki/mediawiki-codesniffer": "42.0.0",
     "mediawiki/mediawiki-phan-config": "0.13.0",
     "nikic/php-parser": "^4.10.2",
     "php-parallel-lint/php-console-highlighter": "1.0.0",
     "php-parallel-lint/php-parallel-lint": "1.3.2",
     "phpunit/phpunit": "9.5.28",
     "psy/psysh": "^0.11.1",
     "seld/jsonlint": "1.8.3",
     "wikimedia/alea": "1.0.0",
     "wikimedia/langconv": "^0.4.2",
     "wikimedia/testing-access-wrapper": "^3.0.0",
     "wmde/hamcrest-html-matchers": "^1.0.0"
 },

From within mediawiki/vendor I can reproduce the interactive prompt with wikimedia/langconv:

$  composer require --dev 'wikimedia/langconv=^0.4.2'
wikimedia/langconv is currently present in the require key and you ran the command with the --dev flag, which will move it to the require-dev key.
Do you want to move this requirement? [no]? 

langconv is a dev requirement due to https://gerrit.wikimedia.org/r/c/mediawiki/core/+/833727

mediawiki/vendor has langconv under require since it is a dependency of Parsoid :)

The composer warning is about langconv being moved from require to require-dev.

The output I was looking for:

INFO:quibble.commands:>>> Start: Install composer dev-requires for vendor.git
INFO:quibble.commands:mediawiki/vendor is used, add require-dev dependencies
DEBUG:quibble.commands:composer require --dev composer/spdx-licenses=1.5.8 doctrine/dbal=3.7.2 doctrine/sql-formatter=1.1.3 ext-simplexml=* giorgiosironi/eris=^0.14.0 hamcrest/hamcrest-php=^2.0 johnkary/phpunit-speedtrap=^4.0 mediawiki/mediawiki-codesniffer=42.0.0 mediawiki/mediawiki-phan-config=0.13.0 nikic/php-parser=^4.10.2 php-parallel-lint/php-console-highlighter=1.0.0 php-parallel-lint/php-parallel-lint=1.3.2 phpunit/phpunit=9.5.28 psy/psysh=^0.12.0 seld/jsonlint=1.10.1 wikimedia/alea=1.0.0 wikimedia/langconv=^0.4.2 wikimedia/testing-access-wrapper=^3.0.0 wmde/hamcrest-html-matchers=^1.0.0
> init: Wikimedia\Composer\Merge\V2\MergePlugin->onInit
nikic/php-parser is currently present in the require key and you ran the command with the --dev flag, which will move it to the require-dev key.
psy/psysh is currently present in the require key and you ran the command with the --dev flag, which will move it to the require-dev key.
wikimedia/langconv is currently present in the require key and you ran the command with the --dev flag, which will move it to the require-dev key.
wikimedia/testing-access-wrapper is currently present in the require key and you ran the command with the --dev flag, which will move it to the require-dev key.

Then press return twice and you get:

Do you want to move these requirements? [no]?
Do you want to re-run the command without --dev? [yes]?
./composer.json has been updated

:)

Change 986672 restored by Hashar:

[integration/quibble@master] Make config non-interactive

https://gerrit.wikimedia.org/r/986672

Change 986672 merged by jenkins-bot:

[integration/quibble@master] Let composer require move reqs to dev reqs

https://gerrit.wikimedia.org/r/986672

Change #1014014 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] release: Quibble 1.7.0

https://gerrit.wikimedia.org/r/1014014

Change #1014014 merged by jenkins-bot:

[integration/quibble@master] release: Quibble 1.7.0

https://gerrit.wikimedia.org/r/1014014

Change #1014039 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: update Quibble to 1.7.0

https://gerrit.wikimedia.org/r/1014039

Change #1014039 merged by jenkins-bot:

[integration/config@master] dockerfiles: update Quibble to 1.7.0

https://gerrit.wikimedia.org/r/1014039