Page MenuHomePhabricator

composer update command installs PHP client for Elasticsearch v6.8.0 where some classes were removed
Open, Needs TriagePublic

Description

How to reproduce:
install MediaWiki 1.35.1 with ElasticSearch extension from git, run composer update and then run php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php

you would get an error:

php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php 
indexing namespaces...
        Indexing namespaces...done
content index...
        Fetching Elasticsearch version...6.8.13...ok
        Scanning available plugins...none
        Picking analyzer...english
        Inferring index identifier...mediawiki_content_first
        Creating index...ok
                Validating number of shards...ok
                Validating replica range...ok
                Validating shard allocation settings...done
                Validating max shards per node...ok
        Validating analyzers...ok
        Validating mappings...
                Validating mapping...different...corrected
        Validating aliases...
                Validating mediawiki_content alias...alias is free...[6e4df9b2173aee354e984b31] [no req]   Error from line 451 of /var/www/html/w/extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Index.php: Class 'Elasticsearch\Endpoints\Indices\Aliases\Update' not found
Backtrace:
#0 /var/www/html/w/extensions/CirrusSearch/includes/Maintenance/Validators/SpecificAliasValidator.php(137): Elastica\Index->addAlias()
#1 /var/www/html/w/extensions/CirrusSearch/includes/Maintenance/Validators/SpecificAliasValidator.php(79): CirrusSearch\Maintenance\Validators\SpecificAliasValidator->updateFreeIndices()
#2 /var/www/html/w/extensions/CirrusSearch/includes/Maintenance/Validators/IndexAliasValidator.php(98): CirrusSearch\Maintenance\Validators\SpecificAliasValidator->updateIndices()
#3 /var/www/html/w/extensions/CirrusSearch/maintenance/UpdateOneSearchIndexConfig.php(458): CirrusSearch\Maintenance\Validators\IndexAliasValidator->validate()
#4 /var/www/html/w/extensions/CirrusSearch/maintenance/UpdateOneSearchIndexConfig.php(411): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateSpecificAlias()
#5 /var/www/html/w/extensions/CirrusSearch/maintenance/UpdateOneSearchIndexConfig.php(269): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateAlias()
#6 /var/www/html/w/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php(61): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->execute()
#7 /var/www/html/w/maintenance/doMaintenance.php(107): CirrusSearch\Maintenance\UpdateSearchIndexConfig->execute()
#8 /var/www/html/w/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php(70): require_once(string)
#9 {main}

It is because composer update installs elasticsearch/elasticsearch version 6.8.0 (since 2021-03-01 18:54 UTC)
and compared to previous version 6.7.2 some classes were removed, see http://comparabl.com/upgrade/elasticsearch-elasticsearch/v6.7.2/v6.8.0

Elastica extension requires "ruflin/elastica": "6.1.1" which requires elasticsearch/elasticsearch: ^6.0 and composer installs elasticsearch-elasticsearch version 6.8.0.

I fixed it adding to composer.local.json file:

"require": {
		"elasticsearch/elasticsearch": "6.7.2"
}

It seems without the fix CirrusSearch works fine but you face the problem when you run the maintenance scripts.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Reedy added a subscriber: Reedy.

Yeah, I've already reported it upstream to ruflin/elastica at https://github.com/ruflin/Elastica/issues/1908 and mentioned it on the commit in question - https://github.com/elastic/elasticsearch-php/commit/2da5f9337a269113954a84fe8e79ea6843f0e7c3

The classes were removed, but aliases were supposed to be left behind. Something got messed up whilst doing that.

This is kind of a dupe of T276320: Unbreak elastic search tests, though that is a more specific instance of this

Change 668691 had a related patch set uploaded (by Reedy; owner: Tobias Andersson):
[mediawiki/extensions/Elastica@REL1_35] Explicitly set elasticsearch/elasticsearch dependency

https://gerrit.wikimedia.org/r/668691

Change 669796 had a related patch set uploaded (by Reedy; owner: Tobias Andersson):
[mediawiki/extensions/Elastica@master] Explicitly set elasticsearch/elasticsearch dependency

https://gerrit.wikimedia.org/r/669796

Reedy renamed this task from composer update command installs PHP client for Elasticsearch v6.8.0 where some classes were removed (2021-03-01 18:54 UTC) to composer update command installs PHP client for Elasticsearch v6.8.0 where some classes were removed.Mar 8 2021, 9:40 PM

Change 668691 merged by jenkins-bot:
[mediawiki/extensions/Elastica@REL1_35] Explicitly set elasticsearch/elasticsearch dependency

https://gerrit.wikimedia.org/r/668691

Change 669796 merged by jenkins-bot:
[mediawiki/extensions/Elastica@master] Explicitly set elasticsearch/elasticsearch dependency

https://gerrit.wikimedia.org/r/669796

Thanks for the report.

I've put the fixes into master and REL1_35; to be reverted out later. elasticsearch/elasticsearch did indeed bring in a break as per semver, but it was unintentional it seems. Hopefully 6.8.1 should follow "soon".

Is this fixed now, since 6.8.1 has been released?

Thank you for clarifying what I meant to say, but I would like for someone to actually confirm this problem has been fixed on MediaWiki. I cannot do so because I am still not able to get CirrusSearch working on AWS Elasticsearch.

This still doesn't seem to work. I'm on MediaWiki 1.35.3, trying to get AWS Elasticsearch v6.5.4 working with CirrusSearch. I've got Elastica 6.1.3 (de9fe84) and elasticsearch/elasticsearch 6.8.1 but when I run the maintenance script I get the error in T267106:

# php ./extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php 
indexing namespaces...
PHP Fatal error:  Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getPar
amWhitelist(): array in $IP/extensions/Elastica/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/Indices/Exists.php on line 45

This still doesn't seem to work. I'm on MediaWiki 1.35.3, trying to get AWS Elasticsearch v6.5.4 working with CirrusSearch. I've got Elastica 6.1.3 (de9fe84) and elasticsearch/elasticsearch 6.8.1 but when I run the maintenance script I get the error in T267106:

# php ./extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php 
indexing namespaces...
PHP Fatal error:  Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getPar
amWhitelist(): array in $IP/extensions/Elastica/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/Indices/Exists.php on line 45

But that error is completely different (it's not from missing classes), and is detailed in T267106: PHP fatal error while searching index: Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getParamWhitelist():

This still doesn't seem to work. I'm on MediaWiki 1.35.3, trying to get AWS Elasticsearch v6.5.4 working with CirrusSearch. I've got Elastica 6.1.3 (de9fe84) and elasticsearch/elasticsearch 6.8.1 but when I run the maintenance script I get the error in T267106:

# php ./extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php 
indexing namespaces...
PHP Fatal error:  Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getPar
amWhitelist(): array in $IP/extensions/Elastica/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/Indices/Exists.php on line 45

But that error is completely different (it's not from missing classes), and is detailed in T267106: PHP fatal error while searching index: Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getParamWhitelist():

My bad, it looked the same, I didn't read closely enough. But it's similar and still a problem. Happy to help be a guinea pig to figure this out.

Also, 6.8 isn't listed as a supported version on REL1_35 either - https://github.com/wikimedia/mediawiki-extensions-Elastica/blob/REL1_35/composer.json#L16

~6.5.1||~6.7.2 doesn't allow 6.8.

I'm seeing the same problem in the original post where Elastica requires ruflin/elastica 6.1.5, which requires elasticsearch/elasticsearch: ^6.0 and conflicts with 6.8.1. However, I also just noticed that Semantic MediaWiki requires elasticsearch/elasticsearch ^5.3|^6.0, so I think that's why I'm getting 6.8.1.

If you're letting it install based on the SMW version constraints alone, probably.

If you're combining (using composer-merge-plugin) your composer.json files, it should try and make sure it validates and meets all constraints

If you're letting it install based on the SMW version constraints alone, probably.

If you're combining (using composer-merge-plugin) your composer.json files, it should try and make sure it validates and meets all constraints

Ok, I was using composer-merge-plugin in the main composer.json file but only for a single extension. After updating that and adding a bit more config to it, including hard-coding elasticsearch/elasticsearch 6.7.* in it, I've finally gotten UpdateSearchIndexConfig.php to run successfully! Now whether the rest of the ES setup goes well remains to be seen since I am attempting to use AWS Elasticsearch.