Page MenuHomePhabricator
Create Task
Maniphest T271777

Bump rufin/elastica (and related libraries) to versions that support PHP 8.0
Closed, ResolvedPublic
Actions

Description

Right now we use:

  • ruflin/elastica on 6.1.5; 6.2.0 supports PHP 8.0
  • elasticsearch/elasticsearch on 6.5.1; >=6.8.0 supports PHP 8.0 but not PHP < 7.3

For ruflin/Elastica, the major version of the release matches the ES major release:

Screenshot 2021-02-15 at 21.12.28.png (658×1 px, 115 KB)

For elasticsearch/elasticsearch:

Screenshot 2021-02-15 at 21.12.46.png (722×968 px, 78 KB)

Details

SubjectRepoBranchLines +/-
Allow elasticsearch/elasticsearch ~6.8.2mediawiki/extensions/ElasticaREL1_35+1 -1
Allow elasticsearch/elasticsearch ~6.8.2mediawiki/extensions/ElasticaREL1_36+1 -1
Allow elasticsearch/elasticsearch ~6.8.2mediawiki/extensions/ElasticaREL1_37+1 -1
Allow elasticsearch/elasticsearch ~6.8.2mediawiki/extensions/Elasticamaster+1 -1
Upgrading ruflin/elastica (6.1.5 => 6.2.0)mediawiki/extensions/Elasticamaster+1 -1
Upgrading ruflin/elastica (6.1.5 => 6.2.0)mediawiki/vendormaster+259 -249
Upgrading ruflin/elastica (6.1.5 => 6.2.0)mediawiki/extensions/ElasticaREL1_36+1 -1
Upgrading ruflin/elastica (6.1.5 => 6.2.0)mediawiki/extensions/ElasticaREL1_35+1 -1
Upgrading ruflin/elastica (6.1.5 => 6.2.0)mediawiki/extensions/ElasticaREL1_37+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT248925 Make MediaWiki release tarball compatible with PHP 8.0
 ResolvedJdforrester-WMFT300463 Make PHP 8.0 voting on MW master
 ResolvedNoneT283275 Make MW master tests pass on PHP 8.0
 ResolvedReedyT268861 CirrusSearch uses Elastica's Match class
 ResolvedReedyT268863 Translate uses Elastica's Match class
 ResolvedmatthiasmullieT268866 WikibaseMediaInfo uses Elastica's Match class
 InvalidNoneT268864 WikibaseCirrusSearch uses Elastica's Match class
 ResolvedReedyT268865 WikibaseLexemeCirrusSearch uses Elastica's Match class
 ResolvedEBernhardsonT271777 Bump rufin/elastica (and related libraries) to versions that support PHP 8.0
 ResolvedGehelT263142 [EPIC] Upgrade Elasticsearch to version 7.10
 ResolvedEBernhardsonT218994 Deprecation warning on elasticsearch 6
ResolvedEBernhardsonT218995 re-enable deprecation warning logger on elasticsearch once issues are solved
 ResolveddcausseT219265 Elasticsearch 6: _retry_on_conflict is deprecated
 ResolveddcausseT219266 Elasticsearch 6 nested_path and nested_filter are deprecated
 ResolveddcausseT219267 Elasticsearch 6: auto_generate_phrase_queries is deprecated and ineffective
 ResolveddcausseT219268 Elasticsearch 6: the classic similarity is deprecated
 ResolveddcausseT219533 Reindex space less languages wikis to use BM25
 OpenNoneT219534 Test MLR models for zhwiki, jawiki and kowiki
 ResolvedEBernhardsonT219269 Elasticsearch 6: silence deprecation warnings to avoid logspam
 DuplicateNoneT215615 Stop using negative scores for deboosting statements
 ResolvedEBernhardsonT288764 Set include_type_name in all get requests
 Resolved• EJosephT288765 Always provide minimum_should_match in bool queries
 ResolvedEBernhardsonT306123 Ensure mjolnir models have positive scores
 DuplicateNoneT273814 Create ElasticSearch 7.10 image in releng/dev-images
 ResolvedGehelT295666 Upgrade Cirrus elasticsearch clusters to 6.8.23
 Resolved• EJosephT294499 Upgrade Cirrus elasticsearch plugins to 6.8.23
 ResolvedbkingT301954 Upgrade deployment-prep to elasticsearch 6.8.23
 ResolvedbkingT301955 Upgrade relforge to elasticsearch 6.8.23
 ResolvedRKemperT278378 Pull Elasticsearch config out of Spicerack
 ResolvedRKemperT301956 Upgrade cloudelastic to elasticsearch 6.8.23
 ResolvedRKemperT301958 Upgrade Search elasticsearch cluster / codfw to elasticsearch 6.8.23
 ResolvedRKemperT301959 Upgrade Search elasticsearch cluster / eqiad to elasticsearch 6.8.23
 ResolvedGehelT304570 elasticsearch: make rolling upgrade proceed to second host after shards are stable
 ResolvedbkingT289135 Upgrade Cirrus Elasticsearch clusters to Debian Bullseye
 ResolvedbkingT299797 Deploy new bullseye elastic cluster nodes on deployment-prep
 ResolvedbkingT301408 Enable search team SRE access to deployment-prep VMs
 ResolvedbkingT307510 Update tls proxy config for Bullseye
 ResolvedbkingT308770 Reimage relforge elastic hosts from Stretch to Bullseye
 ResolvedbkingT309343 Upgrade cloudelastic clusters to Debian Bullseye
 ResolvedbkingT312298 Update NIC firmware on all Elastic PowerEdge R440 elastic hosts
 ResolvedEBernhardsonT295736 Evaluate deployed elasticsearch plugins for 7.10 compatability
 ResolvedEBernhardsonT295734 Bring up two copies of the CirrusSearch browser integration env in cloud
 ResolvedTJonesT300302 Test Elastic 6.8 language analyzers
 Resolved• ZbyszkoT300833 Address "Log4shell" vulnerability in Elasticsearch 7.10.2
 DeclinedNoneT301724 Sunset ApiFeatureUsage
 DeclinedNoneT302638 Sunset ApiFeatureUsage (TDMP)
 ResolvedTJonesT301131 Test Elastic 7.10 language analyzers
 ResolvedEBernhardsonT317200 Reindex all wikis to fix nnbsp regression
 DeclinedNoneT303443 Ensure that all applications depending on the Search Elasticsearch cluster are compatible with Elasticsearch 7.10
 Resolvedbd808T303444 Ensure Toolhub is compatible with Elasticsearch 7.10
 DeclinedNoneT303445 Ensure Phabricator is compatible with Elasticsearch 7.10
 ResolvedEBernhardsonT304290 Create docker elasticsearch images with wmf search plugins
 DeclinedNoneT304569 Remove row/rack awareness in Elasticsearch config
 ResolveddcausseT305428 Upgrade the Translate TTM Elasticsearch implementation to elasticsearch 6.8 and onwards
ResolvedEBernhardsonT306644 re-run wbsearchentities optimization process
ResolvedEBernhardsonT307931 Run CirrusSearch test suite against Elasticsearch 7.10
 ResolvedEBernhardsonT308676 Elasticsearch 7.10.2 rollout plan
 ResolvedbkingT315604 Upgrade relforge cluster to 7.10.2
 ResolvedGehelT316159 Upgrade cloudelastic cluster to 7.10.2
 ResolvedGehelT316240 Upgrade deployment-prep to Elasticsearch 7.10.2
 ResolvedRKemperT316719 Upgrade codfw cluster to Elasticsearch 7.10.2
 ResolvedBUG REPORTEBernhardsonT317381 Reduction in helpfulness and quantity of autocomplete search results
 ResolvedEBernhardsonT317374 Search now caps total results count at 10k because of elasticsearch 7 upgrade
 ResolvedRKemperT317686 Upgrade eqiad cluster to Elasticsearch 7.10.2
 ResolvedEBernhardsonT313434 Migrate apifeatureusage to elasticsearch 7
 ResolvedkostajhT317513 Flow is incompatible with Elastica - Remove ElasticSearch integration / Flow search

Event Timeline

Jdforrester-WMF created this task.Jan 12 2021, 1:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 12 2021, 1:50 AM
Jdforrester-WMF added a project: CirrusSearch.Jan 12 2021, 1:51 AM
Jdforrester-WMF added a subtask: T263142: [EPIC] Upgrade Elasticsearch to version 7.10.
Reedy added parent tasks: T268861: CirrusSearch uses Elastica's Match class, T268863: Translate uses Elastica's Match class, T268866: WikibaseMediaInfo uses Elastica's Match class, T268864: WikibaseCirrusSearch uses Elastica's Match class, T268865: WikibaseLexemeCirrusSearch uses Elastica's Match class.Jan 12 2021, 8:44 AM
WMDE-leszek subscribed.Jan 21 2021, 6:04 AM
CBogen triaged this task as High priority.Jan 25 2021, 4:19 PM
CBogen moved this task from needs triage to elastic / cirrus on the Discovery-Search board.
Umherirrender mentioned this in T248925: Make MediaWiki release tarball compatible with PHP 8.0.Feb 7 2021, 12:14 AM
Reedy changed the task status from Open to Stalled.Feb 7 2021, 12:28 AM
Reedy subscribed.

Stalling. Upstream(s) haven't done the releases (and have gone a bit quiet)... And we're waiting on the upgrades of ES too..

Reedy updated the task description. (Show Details)Feb 11 2021, 10:28 PM
Reedy added a comment.Feb 11 2021, 10:32 PM

https://github.com/elastic/elasticsearch-php/releases/tag/v7.11.0 is out. 7.x of elasticsearch-php only supports ES >= 7.x...

7.1 of ruflin/Elastica looks like it should be out pretty soon - https://github.com/ruflin/Elastica/pull/1871

Reedy added a comment.Feb 11 2021, 10:46 PM

Just pushed https://github.com/ruflin/Elastica/pull/1898 up too...

So we should be able to use 6.1.2 (when released!) to swap Match out for MatchQuery to unblock the parent tasks above this

Jdforrester-WMF added a comment.Feb 12 2021, 12:10 AM

Is 7.11.0 etc. based on the new non-OSI licence?

Reedy added a comment.EditedFeb 12 2021, 12:27 AM
In T271777#6825206, @Jdforrester-WMF wrote:

Is 7.11.0 etc. based on the new non-OSI licence?

I'm not sure... Does that license change apply to the the php libraries too?

https://github.com/elastic/elasticsearch-php/tree/v7.11.0 still has Apache/LGPL

Copyright 2013-2020 Elasticsearch B.V.

elasticsearch-php is licensed under the Apache License, Version 2.0
(<LICENSE-APACHE> or <http://www.apache.org/licenses/LICENSE-2.0>) or the GNU
Lesser General Public License, Version 2.1 (<LICENSE-LGPL> or
<https://www.gnu.org/licenses/lgpl-2.1.html>) at your option.
Jdforrester-WMF added a comment.Feb 12 2021, 12:30 AM

Maybe not? T272111 mentions it as a change from Apache 2.0 to SSPL, but the ruflin/elastica code is MIT… Did you have to sign a CLA to contribute? If not, presumably it's still MIT forever…

Reedy added a comment.Feb 12 2021, 12:34 AM
In T271777#6825252, @Jdforrester-WMF wrote:

Maybe not? T272111 mentions it as a change from Apache 2.0 to SSPL, but the ruflin/elastica code is MIT… Did you have to sign a CLA to contribute? If not, presumably it's still MIT forever…

Noting ruflin/elastica != elasticsearch/elasticsearch

Jdforrester-WMF added a comment.Feb 12 2021, 12:38 AM

Yeah, and https://github.com/elastic/elasticsearch-php/blob/v7.11.0/LICENSE says it's still dual LGPL/Apache 2.0.

Reedy updated the task description. (Show Details)Feb 15 2021, 9:11 PM
Reedy updated the task description. (Show Details)Feb 15 2021, 9:14 PM
Reedy updated the task description. (Show Details)
Reedy added a comment.Feb 15 2021, 9:24 PM
In T271777#6824954, @Reedy wrote:

Just pushed https://github.com/ruflin/Elastica/pull/1898 up too...

So we should be able to use 6.1.2 (when released!) to swap Match out for MatchQuery to unblock the parent tasks above this

6.1.2 is a-comin'.

https://github.com/ruflin/Elastica/pull/1901

Reedy mentioned this in T274877: Update ruflin/elastica to 6.1.3.Feb 16 2021, 11:48 AM
Reedy added a comment.Feb 16 2021, 12:42 PM

6.1.2 is a dud xD

Reedy mentioned this in T275166: CirrusSearch ParseError: "syntax error, unexpected token "match", expecting ":"" (collision with new PHP8 match keyword?).Feb 18 2021, 10:49 PM
Reedy renamed this task from Bump rufin/elastica (and related libraries) to versions that support PHP 8.0, when they exist to Bump rufin/elastica (and related libraries) to versions that support PHP 8.0.Mar 23 2021, 1:42 AM
Jdforrester-WMF added a parent task: T283275: Make MW master tests pass on PHP 8.0.May 20 2021, 6:54 PM
Reedy updated the task description. (Show Details)Aug 23 2021, 11:24 AM
taavi mentioned this in T271736: Migrate WMF production from PHP 7.2 to PHP 7.4.Sep 2 2021, 12:42 PM
gerritbot added a comment.Oct 8 2021, 2:13 PM

Change 728475 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/Elastica@master] Upgrading ruflin/elastica (6.1.5 => 6.2.0)

https://gerrit.wikimedia.org/r/728475

gerritbot added a project: Patch-For-Review.Oct 8 2021, 2:13 PM

Change 728473 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/vendor@master] Upgrading ruflin/elastica (6.1.5 => 6.2.0)

https://gerrit.wikimedia.org/r/728473

Reedy removed a subtask: T263142: [EPIC] Upgrade Elasticsearch to version 7.10.Oct 8 2021, 2:25 PM
Daimona updated the task description. (Show Details)Oct 8 2021, 2:26 PM
Reedy changed the task status from Stalled to Open.Oct 8 2021, 2:36 PM

The question I guess does become whether there is going to be an Elasticsearch upgrade to a version before T263142: [EPIC] Upgrade Elasticsearch to version 7.10...

gerritbot added a comment.Oct 8 2021, 2:58 PM

Change 728521 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/Elastica@REL1_37] Upgrading ruflin/elastica (6.1.5 => 6.2.0)

https://gerrit.wikimedia.org/r/728521

gerritbot added a comment.Oct 8 2021, 2:59 PM

Change 728523 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/Elastica@REL1_36] Upgrading ruflin/elastica (6.1.5 => 6.2.0)

https://gerrit.wikimedia.org/r/728523

gerritbot added a comment.Oct 8 2021, 2:59 PM

Change 728525 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/Elastica@REL1_35] Upgrading ruflin/elastica (6.1.5 => 6.2.0)

https://gerrit.wikimedia.org/r/728525

Daimona subscribed.Oct 8 2021, 3:19 PM
In T271777#7412312, @Reedy wrote:

The question I guess does become whether there is going to be an Elasticsearch upgrade to a version before T263142: [EPIC] Upgrade Elasticsearch to version 7.10...

Eh, maybe we can get in touch with upstream and ask for a minor release with PHP 7.2 AND 8 support? That would be 6.7.3, I guess... I see this commit bumped the requirements to PHP >= 7.1 which is great for our needs, but didn't pass tests due to the usual PHPUnit incompabilities, apparently, and the requirement was then updated to PHP >= 7.3. Perhaps add polyfills like for Elastica?

Reedy added a comment.Oct 8 2021, 3:28 PM
In T271777#7412471, @Daimona wrote:
In T271777#7412312, @Reedy wrote:

The question I guess does become whether there is going to be an Elasticsearch upgrade to a version before T263142: [EPIC] Upgrade Elasticsearch to version 7.10...

Eh, maybe we can get in touch with upstream and ask for a minor release with PHP 7.2 AND 8 support? That would be 6.7.3, I guess... I see this commit bumped the requirements to PHP >= 7.1 which is great for our needs, but didn't pass tests due to the usual PHPUnit incompabilities, apparently, and the requirement was then updated to PHP >= 7.3. Perhaps add polyfills like for Elastica?

There's also Elasticsearch version compatability/support issues with the library, not just the PHP concerns. elasticsearch-php 6.7 may work, but it's entirely possible it won't either.

https://github.com/elastic/elasticsearch-php#version-matrix

If you are using Elasticsearch 6.6 to 6.7, use Elasticsearch-PHP 6.7.x branch.

If you are using Elasticsearch 6.0 to 6.5, use Elasticsearch-PHP 6.5.x branch.

And we're using 6.5.4 of Elasticsearch itself...

So we'd really need changes in https://github.com/elastic/elasticsearch-php/tree/6.5.x and then another 6.5.x release of elasticsearch-php

The branch is untouched since Feb 2020, so maybe not supported, or else generally EOL?

https://github.com/elastic/elasticsearch-php/tree/6.7.x is a little better, being Feb 2021...

https://www.elastic.co/support/eol says 6.5.x is EOL since 2020-05-14. As are the rest of the 6.x releases

https://github.com/elastic/elasticsearch-php/releases/tag/v6.8.2 would seemingly be the last 6.x release.

I don't know how receptive of patches to these branches upstream may or may not be :)

taavi subscribed.Oct 8 2021, 3:29 PM
gerritbot added a comment.Oct 8 2021, 3:34 PM

Change 728521 merged by jenkins-bot:

[mediawiki/extensions/Elastica@REL1_37] Upgrading ruflin/elastica (6.1.5 => 6.2.0)

https://gerrit.wikimedia.org/r/728521

gerritbot added a comment.Oct 8 2021, 3:35 PM

Change 728525 merged by jenkins-bot:

[mediawiki/extensions/Elastica@REL1_35] Upgrading ruflin/elastica (6.1.5 => 6.2.0)

https://gerrit.wikimedia.org/r/728525

gerritbot added a comment.Oct 8 2021, 3:36 PM

Change 728523 merged by jenkins-bot:

[mediawiki/extensions/Elastica@REL1_36] Upgrading ruflin/elastica (6.1.5 => 6.2.0)

https://gerrit.wikimedia.org/r/728523

gerritbot added a comment.Oct 8 2021, 5:02 PM

Change 728473 merged by jenkins-bot:

[mediawiki/vendor@master] Upgrading ruflin/elastica (6.1.5 => 6.2.0)

https://gerrit.wikimedia.org/r/728473

gerritbot added a comment.Oct 8 2021, 5:02 PM

Change 728475 merged by jenkins-bot:

[mediawiki/extensions/Elastica@master] Upgrading ruflin/elastica (6.1.5 => 6.2.0)

https://gerrit.wikimedia.org/r/728475

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.4; 2021-10-12).Oct 8 2021, 6:00 PM
Reedy removed a project: Patch-For-Review.Dec 17 2021, 4:46 PM
Reedy changed the task status from Open to Stalled.Dec 17 2021, 4:52 PM

Marking stalled.

ruflin/elastica is now good. elasticsearch/elasticsearch needs at least 6.8.x (and 6.8.2 at time of writing) to allow PHP 8

https://github.com/elastic/elasticsearch-php/releases/tag/v6.8.0
https://github.com/elastic/elasticsearch-php/releases/tag/v6.8.1
https://github.com/elastic/elasticsearch-php/releases/tag/v6.8.2

So basically blocked on the T263142: [EPIC] Upgrade Elasticsearch to version 7.10 tree again... Though, more specifically T295666: Upgrade Cirrus elasticsearch clusters to 6.8.23 - when that is done, we can bump elasticsearch/elasticsearch in the Elastica extension and then MediaWiki-Vendor

Reedy added a subtask: T295666: Upgrade Cirrus elasticsearch clusters to 6.8.23.Dec 17 2021, 4:53 PM
gerritbot added a comment.Dec 17 2021, 6:16 PM

Change 748168 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/Elastica@master] Allow elasticsearch/elasticsearch ~6.8.2

https://gerrit.wikimedia.org/r/748168

gerritbot added a project: Patch-For-Review.Dec 17 2021, 6:16 PM

Change 748187 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/Elastica@REL1_37] Allow elasticsearch/elasticsearch ~6.8.2

https://gerrit.wikimedia.org/r/748187

gerritbot added a comment.Dec 17 2021, 6:16 PM

Change 748188 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/Elastica@REL1_36] Allow elasticsearch/elasticsearch ~6.8.2

https://gerrit.wikimedia.org/r/748188

gerritbot added a comment.Dec 17 2021, 6:16 PM

Change 748189 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/Elastica@REL1_35] Allow elasticsearch/elasticsearch ~6.8.2

https://gerrit.wikimedia.org/r/748189

gerritbot added a comment.Dec 21 2021, 10:47 AM

Change 748168 merged by jenkins-bot:

[mediawiki/extensions/Elastica@master] Allow elasticsearch/elasticsearch ~6.8.2

https://gerrit.wikimedia.org/r/748168

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.16; 2022-01-03); removed MW-1.38-notes (1.38.0-wmf.4; 2021-10-12).Dec 21 2021, 11:00 AM
gerritbot added a comment.Dec 21 2021, 1:13 PM

Change 748187 merged by jenkins-bot:

[mediawiki/extensions/Elastica@REL1_37] Allow elasticsearch/elasticsearch ~6.8.2

https://gerrit.wikimedia.org/r/748187

gerritbot added a comment.Dec 21 2021, 1:14 PM

Change 748188 merged by jenkins-bot:

[mediawiki/extensions/Elastica@REL1_36] Allow elasticsearch/elasticsearch ~6.8.2

https://gerrit.wikimedia.org/r/748188

gerritbot added a comment.Dec 21 2021, 1:14 PM

Change 748189 merged by jenkins-bot:

[mediawiki/extensions/Elastica@REL1_35] Allow elasticsearch/elasticsearch ~6.8.2

https://gerrit.wikimedia.org/r/748189

Reedy removed a project: Patch-For-Review.Jan 29 2022, 5:05 PM
Reedy updated the task description. (Show Details)Mar 8 2022, 2:50 AM
Gehel edited subtasks, added: T263142: [EPIC] Upgrade Elasticsearch to version 7.10; removed: T295666: Upgrade Cirrus elasticsearch clusters to 6.8.23.Apr 6 2022, 7:46 PM
Gehel moved this task from elastic / cirrus to needs triage on the Discovery-Search board.
Gehel moved this task from needs triage to watching / waiting on the Discovery-Search board.Apr 11 2022, 3:34 PM
dcausse changed the task status from Stalled to Open.May 18 2022, 6:41 PM
dcausse subscribed.

Marking as Open as this should be resolved "soon" as we plan to ship ruflin/Elastica 7.1.5 & elasticsearch/elasticsearch 7.11.0 as part of https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/791634 in the coming weeks.

Reedy mentioned this in T312971: Enable PHP 8 CI for ProofreadPage.Jul 21 2022, 8:13 PM
Jdforrester-WMF closed this task as Resolved.Aug 30 2022, 9:20 PM
Jdforrester-WMF assigned this task to EBernhardson.

Done in {dEELA1245f59b30fdcbc1b6a41de17d2c2b4b433d16a9} indeed.

Back-porting all this code for ES7 support in REL1_38 seems infeasible; we'll just declare it unavailable until REL1_39.

Gehel closed subtask T263142: [EPIC] Upgrade Elasticsearch to version 7.10 as Resolved.Sep 29 2022, 7:24 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL