Page MenuHomePhabricator

Geocoder composer issues
Open, Needs TriagePublic

Description

If I try to update geocoder (to deal with another issue) it pulls in some "replace" statements that cause the packages that in our main vendor to be removed - see log

Note - replace is a composer command which can be used to prevent a package installing packages elsewhere in use by the main package (e.g the extension + civicrm) - which is an alternate solution of sorts to our use of composer-merge-plugin - the combination of the are causing the problem - also recorded in https://github.com/wikimedia/composer-merge-plugin/issues/221

http://www.darwinbiler.com/how-does-the-replace-property-work-in-composer/
Argument for adding the replace to geocoder https://github.com/eileenmcnaughton/org.wikimedia.geocoder/issues/26

docker@civicrm:/srv/civi-sites/wmff$ composer update guzzlehttp/promises
Loading composer repositories with package information
Updating dependencies                                 
Lock file operations: 0 installs, 0 updates, 9 removals
  - Removing guzzlehttp/guzzle (6.5.6)
  - Removing guzzlehttp/promises (v1.3.1)
  - Removing guzzlehttp/psr7 (1.8.5)
  - Removing psr/http-message (1.0.1)
  - Removing ralouphie/getallheaders (3.0.3)
  - Removing symfony/polyfill-intl-idn (v1.18.1)
  - Removing symfony/polyfill-intl-normalizer (v1.18.1)
  - Removing symfony/polyfill-php70 (v1.18.1)
  - Removing symfony/polyfill-php72 (v1.18.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 0 installs, 0 updates, 9 removals
  - Removing symfony/polyfill-php72 (v1.18.1)
  - Removing symfony/polyfill-php70 (v1.18.1)
  - Removing symfony/polyfill-intl-normalizer (v1.18.1)
  - Removing symfony/polyfill-intl-idn (v1.18.1)
  - Removing ralouphie/getallheaders (3.0.3)
  - Removing psr/http-message (1.0.1)
  - Removing guzzlehttp/psr7 (1.8.5)
  - Removing guzzlehttp/promises (v1.3.1)
  - Removing guzzlehttp/guzzle (6.5.6)
Package webmozart/path-util is abandoned, you should avoid using it. Use symfony/filesystem instead.
Package phpunit/php-token-stream is abandoned, you should avoid using it. No replacement was suggested.
Generating autoload files
3 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
docker@civicrm:/srv/civi-sites/wmff$

Event Timeline

Note I tried switching replace to 'self.version' per the docs

image.png (320×1 px, 38 KB)

but wind up with

docker@civicrm:/srv/civi-sites/wmff$ composer update -W
Loading composer repositories with package information Updating dependencies
Your requirements could not be resolved to an installable set of packages.

Problem 1
  - Root composer.json requires guzzlehttp/guzzle ~6.0, found guzzlehttp/guzzle[6.0.0, ..., 6.5.x-dev] but these were not loaded, likely because it conflicts with another require.
Problem 2
  - Root composer.json requires php-http/guzzle6-adapter ^1.1 -> satisfiable by php-http/guzzle6-adapter[v1.1.0, v1.1.1].
  - php-http/guzzle6-adapter[v1.1.0, ..., v1.1.1] require guzzlehttp/guzzle ^6.0 -> found guzzlehttp/guzzle[6.0.0, ..., 6.5.x-dev] but these were not loaded, likely because it conflicts with another require.
Problem 3
  - php-http/httplug[v1.0.0-RC1, ..., v1.1.0] require psr/http-message ^1.0 -> found psr/http-message[dev-master, 1.0, 1.0.1, 1.0.x-dev (alias of dev-master)] but these were not loaded, likely because it conflicts with another require.
  - omnimail/omnimail[0.3.0, ..., 0.3.5.1] require php-http/httplug ^1.1 -> satisfiable by php-http/httplug[v1.1.0].
  - Root composer.json requires omnimail/omnimail ^0.3 -> satisfiable by omnimail/omnimail[0.3.0, ..., 0.3.5.1].

Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.
docker@civicrm:/srv/civi-sites/wmff$

@Reedy - what is your feeling on https://github.com/wikimedia/composer-merge-plugin/issues/221 & making the merging of 'replace' configurable per that github issue - which would help us here

It feels like you're misusing replace.

I understand it's to stop bringing in guzzle (etc) that Civi includes itself. But it does feels like "doing it wrong" when you're "replacing" numerous other random packages.

Map of packages that are replaced by this package. This allows you to fork a package, publish it under a different name with its own version numbers, while packages requiring the original package continue to work with your fork because it replaces the original package.

http://www.darwinbiler.com/how-does-the-replace-property-work-in-composer/ does show the other example, but in this case, geocoder isn't the thing providing guzzle etc, so shouldn't be the thing replacing it out.

We do have similar issues with some MW extensions, but we don't generally ship the vendor folder (which looks to be being done here... Is there any reason? Does it need to be? Especially if you're also using composer-merge-plugin), and suggest people use composer-merge-plugin, because running composer in random extension folders can result in similar conflicts.

We don't install CiviCRM via composer (when we get off Drupal 7 we will) - so we have the entire tarball of CiviCRM on our site (and we just manage it to avoid conflicts with the other packages we have on the Drupal site). CiviCRM runs with Wordpress, Drupal and Joomla which all have different relationships with composer....

Then CiviCRM has extensions that ship in it - some are managed by us and others are not - these sometimes ALSO come with a vendor directory. The one in question is one we created but it has community users outside WMF (and the composer replace patch came from one of them).

So for the broader community the situation is

  • the extension will only ever be installed when CiviCRM is installed
  • there are some packages (Guzzle) that are in CiviCRM so they don't want a second copy but
  • there are other packages that are in the extension and NOT CiviCRM that need to be there - this sitution is described in the docs as a secondary usage of replace
  • many of the users will be installing both CiviCRM and the extension using a tarball.
  • even though in this extension we do have ultimate control and could remove the replace statement I don't think it's the only one since CiviCRM has been battling with how to deal with vendor directories with the different needs of Wordpress, Joomla and Drupal and I suspect others might use 'replace' in this way

In Fundraising tech we use composer-merge-plugin on the extension directories but it isn't broadly used in the CiviCRM world. So, this might be a mis-use but it's one that is very hard for us to get away from. Whereas, making it possible to opt out from merging in 'replace' seems technically 'not that hard' and would empower us / users of the merge-plugin