Page MenuHomePhabricator
Create Task
Maniphest T276854

composer update command installs PHP client for Elasticsearch v6.8.0 where some classes were removed
Closed, ResolvedPublic
Actions

Description

How to reproduce:
install MediaWiki 1.35.1 with ElasticSearch extension from git, run composer update and then run php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php

you would get an error:

php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php 
indexing namespaces...
        Indexing namespaces...done
content index...
        Fetching Elasticsearch version...6.8.13...ok
        Scanning available plugins...none
        Picking analyzer...english
        Inferring index identifier...mediawiki_content_first
        Creating index...ok
                Validating number of shards...ok
                Validating replica range...ok
                Validating shard allocation settings...done
                Validating max shards per node...ok
        Validating analyzers...ok
        Validating mappings...
                Validating mapping...different...corrected
        Validating aliases...
                Validating mediawiki_content alias...alias is free...[6e4df9b2173aee354e984b31] [no req]   Error from line 451 of /var/www/html/w/extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Index.php: Class 'Elasticsearch\Endpoints\Indices\Aliases\Update' not found
Backtrace:
#0 /var/www/html/w/extensions/CirrusSearch/includes/Maintenance/Validators/SpecificAliasValidator.php(137): Elastica\Index->addAlias()
#1 /var/www/html/w/extensions/CirrusSearch/includes/Maintenance/Validators/SpecificAliasValidator.php(79): CirrusSearch\Maintenance\Validators\SpecificAliasValidator->updateFreeIndices()
#2 /var/www/html/w/extensions/CirrusSearch/includes/Maintenance/Validators/IndexAliasValidator.php(98): CirrusSearch\Maintenance\Validators\SpecificAliasValidator->updateIndices()
#3 /var/www/html/w/extensions/CirrusSearch/maintenance/UpdateOneSearchIndexConfig.php(458): CirrusSearch\Maintenance\Validators\IndexAliasValidator->validate()
#4 /var/www/html/w/extensions/CirrusSearch/maintenance/UpdateOneSearchIndexConfig.php(411): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateSpecificAlias()
#5 /var/www/html/w/extensions/CirrusSearch/maintenance/UpdateOneSearchIndexConfig.php(269): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateAlias()
#6 /var/www/html/w/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php(61): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->execute()
#7 /var/www/html/w/maintenance/doMaintenance.php(107): CirrusSearch\Maintenance\UpdateSearchIndexConfig->execute()
#8 /var/www/html/w/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php(70): require_once(string)
#9 {main}

It is because composer update installs elasticsearch/elasticsearch version 6.8.0 (since 2021-03-01 18:54 UTC)
and compared to previous version 6.7.2 some classes were removed, see http://comparabl.com/upgrade/elasticsearch-elasticsearch/v6.7.2/v6.8.0

Elastica extension requires "ruflin/elastica": "6.1.1" which requires elasticsearch/elasticsearch: ^6.0 and composer installs elasticsearch-elasticsearch version 6.8.0.

I fixed it adding to composer.local.json file:

"require": {
		"elasticsearch/elasticsearch": "6.7.2"
}

It seems without the fix CirrusSearch works fine but you face the problem when you run the maintenance scripts.

Details

SubjectRepoBranchLines +/-
Explicitly set elasticsearch/elasticsearch dependencymediawiki/extensions/Elasticamaster+1 -0
Explicitly set elasticsearch/elasticsearch dependencymediawiki/extensions/ElasticaREL1_35+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Pastakhov created this task.Mar 8 2021, 6:43 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptMar 8 2021, 6:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Reedy added a project: Upstream.Mar 8 2021, 7:21 PM
Reedy subscribed.

Yeah, I've already reported it upstream to ruflin/elastica at https://github.com/ruflin/Elastica/issues/1908 and mentioned it on the commit in question - https://github.com/elastic/elasticsearch-php/commit/2da5f9337a269113954a84fe8e79ea6843f0e7c3

The classes were removed, but aliases were supposed to be left behind. Something got messed up whilst doing that.

This is kind of a dupe of T276320: Unbreak elastic search tests, though that is a more specific instance of this

Reedy added a comment.Mar 8 2021, 7:26 PM

https://github.com/elastic/elasticsearch-php/issues/1112 filed too

Reedy added a parent task: T276320: Unbreak elastic search tests.Mar 8 2021, 7:26 PM
Reedy added a project: MW-1.35-release.
tosfos subscribed.Mar 8 2021, 9:17 PM
gerritbot added a comment.Mar 8 2021, 9:37 PM

Change 668691 had a related patch set uploaded (by Reedy; owner: Tobias Andersson):
[mediawiki/extensions/Elastica@REL1_35] Explicitly set elasticsearch/elasticsearch dependency

https://gerrit.wikimedia.org/r/668691

gerritbot added a project: Patch-For-Review.Mar 8 2021, 9:37 PM

Change 669796 had a related patch set uploaded (by Reedy; owner: Tobias Andersson):
[mediawiki/extensions/Elastica@master] Explicitly set elasticsearch/elasticsearch dependency

https://gerrit.wikimedia.org/r/669796

Reedy renamed this task from composer update command installs PHP client for Elasticsearch v6.8.0 where some classes were removed (2021-03-01 18:54 UTC) to composer update command installs PHP client for Elasticsearch v6.8.0 where some classes were removed.Mar 8 2021, 9:40 PM
gerritbot added a comment.Mar 8 2021, 9:54 PM

Change 668691 merged by jenkins-bot:
[mediawiki/extensions/Elastica@REL1_35] Explicitly set elasticsearch/elasticsearch dependency

https://gerrit.wikimedia.org/r/668691

gerritbot added a comment.Mar 8 2021, 10:19 PM

Change 669796 merged by jenkins-bot:
[mediawiki/extensions/Elastica@master] Explicitly set elasticsearch/elasticsearch dependency

https://gerrit.wikimedia.org/r/669796

Reedy moved this task from Blocker to Not a blocker on the MW-1.35-release board.Mar 8 2021, 10:31 PM
ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.34; 2021-03-09).Mar 8 2021, 11:00 PM
Reedy added a comment.Mar 8 2021, 11:10 PM

Thanks for the report.

I've put the fixes into master and REL1_35; to be reverted out later. elasticsearch/elasticsearch did indeed bring in a break as per semver, but it was unintentional it seems. Hopefully 6.8.1 should follow "soon".

Maintenance_bot removed a project: Patch-For-Review.Mar 8 2021, 11:10 PM
Pastakhov added a comment.Mar 9 2021, 8:42 AM

Thank you!

Ciencia_Al_Poder subscribed.Mar 9 2021, 6:41 PM
MPhamWMF moved this task from needs triage to watching / waiting on the Discovery-Search board.Mar 15 2021, 3:12 PM
JeffreyWang mentioned this in T267106: PHP fatal error while searching index: Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getParamWhitelist(): .Jun 10 2021, 12:39 AM
JeffreyWang subscribed.Jun 29 2021, 3:09 AM

Is this fixed now, since 6.8.1 has been released?

Ciencia_Al_Poder added a comment.Jun 29 2021, 7:19 AM

Looking at the comment on https://github.com/elastic/elasticsearch-php/issues/1112#issuecomment-804420738 it should be fixed

JeffreyWang added a comment.Jun 29 2021, 7:21 AM

Thank you for clarifying what I meant to say, but I would like for someone to actually confirm this problem has been fixed on MediaWiki. I cannot do so because I am still not able to get CirrusSearch working on AWS Elasticsearch.

Justin_C_Lloyd subscribed.Jul 21 2021, 2:46 PM
Justin_C_Lloyd added a comment.EditedJul 21 2021, 2:49 PM

This still doesn't seem to work. I'm on MediaWiki 1.35.3, trying to get AWS Elasticsearch v6.5.4 working with CirrusSearch. I've got Elastica 6.1.3 (de9fe84) and elasticsearch/elasticsearch 6.8.1 but when I run the maintenance script I get the error in T267106:

# php ./extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php 
indexing namespaces...
PHP Fatal error:  Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getPar
amWhitelist(): array in $IP/extensions/Elastica/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/Indices/Exists.php on line 45
Reedy added a comment.Jul 21 2021, 2:53 PM
In T276854#7227486, @Justin_C_Lloyd wrote:

This still doesn't seem to work. I'm on MediaWiki 1.35.3, trying to get AWS Elasticsearch v6.5.4 working with CirrusSearch. I've got Elastica 6.1.3 (de9fe84) and elasticsearch/elasticsearch 6.8.1 but when I run the maintenance script I get the error in T267106:

# php ./extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php 
indexing namespaces...
PHP Fatal error:  Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getPar
amWhitelist(): array in $IP/extensions/Elastica/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/Indices/Exists.php on line 45

But that error is completely different (it's not from missing classes), and is detailed in T267106: PHP fatal error while searching index: Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getParamWhitelist():

Justin_C_Lloyd added a comment.Jul 21 2021, 2:55 PM
In T276854#7227492, @Reedy wrote:
In T276854#7227486, @Justin_C_Lloyd wrote:

This still doesn't seem to work. I'm on MediaWiki 1.35.3, trying to get AWS Elasticsearch v6.5.4 working with CirrusSearch. I've got Elastica 6.1.3 (de9fe84) and elasticsearch/elasticsearch 6.8.1 but when I run the maintenance script I get the error in T267106:

# php ./extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php 
indexing namespaces...
PHP Fatal error:  Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getPar
amWhitelist(): array in $IP/extensions/Elastica/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/Indices/Exists.php on line 45

But that error is completely different (it's not from missing classes), and is detailed in T267106: PHP fatal error while searching index: Declaration of Elasticsearch\Endpoints\Indices\Exists::getParamWhitelist() must be compatible with Elasticsearch\Endpoints\AbstractEndpoint::getParamWhitelist():

My bad, it looked the same, I didn't read closely enough. But it's similar and still a problem. Happy to help be a guinea pig to figure this out.

Reedy added a comment.Jul 21 2021, 2:55 PM

Also, 6.8 isn't listed as a supported version on REL1_35 either - https://github.com/wikimedia/mediawiki-extensions-Elastica/blob/REL1_35/composer.json#L16

~6.5.1||~6.7.2 doesn't allow 6.8.

Justin_C_Lloyd added a comment.Jul 21 2021, 3:44 PM

I'm seeing the same problem in the original post where Elastica requires ruflin/elastica 6.1.5, which requires elasticsearch/elasticsearch: ^6.0 and conflicts with 6.8.1. However, I also just noticed that Semantic MediaWiki requires elasticsearch/elasticsearch ^5.3|^6.0, so I think that's why I'm getting 6.8.1.

Reedy added a comment.Jul 21 2021, 4:20 PM

If you're letting it install based on the SMW version constraints alone, probably.

If you're combining (using composer-merge-plugin) your composer.json files, it should try and make sure it validates and meets all constraints

Justin_C_Lloyd added a comment.Jul 21 2021, 8:11 PM
In T276854#7227840, @Reedy wrote:

If you're letting it install based on the SMW version constraints alone, probably.

If you're combining (using composer-merge-plugin) your composer.json files, it should try and make sure it validates and meets all constraints

Ok, I was using composer-merge-plugin in the main composer.json file but only for a single extension. After updating that and adding a bit more config to it, including hard-coding elasticsearch/elasticsearch 6.7.* in it, I've finally gotten UpdateSearchIndexConfig.php to run successfully! Now whether the rest of the ES setup goes well remains to be seen since I am attempting to use AWS Elasticsearch.

Reedy closed this task as Resolved.Dec 16 2022, 6:07 PM
Reedy assigned this task to • toan.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL