Page MenuHomePhabricator

Upgrade ElasticSearch to version >=2.2
Closed, DuplicatePublic

Description

Eventually, we should upgrade ES to version 2 (or whatever the newer point release is)

Chances are, this will have more breaking changes

Event Timeline

Reedy created this task.Dec 31 2015, 1:58 PM
Reedy raised the priority of this task from to Needs Triage.
Reedy updated the task description. (Show Details)
Reedy added a project: Elasticsearch.
Reedy added a subscriber: Reedy.
Restricted Application added a project: Discovery. · View Herald TranscriptDec 31 2015, 1:58 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
Reedy set Security to None.
Raftab added a subscriber: Raftab.Jan 20 2016, 6:35 PM

I'm hitting this issue:

../maintenance/updateSearchIndexConfig.php
content index...

        Fetching Elasticsearch version...2.1.1...ok
        Scanning available plugins...
                head
        Infering index identifier...PHP Fatal error:  Wrong parameters for Exception([string $exception [, long $code [, Exception $previous = NULL]]]) in /extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Exception/ResponseException.php on line 41

When trying to update the index, I think it's because of this issue: https://github.com/ruflin/Elastica/issues/974
It looks like the last ruflin/Elastica version you merged with was 2.2.1? https://github.com/wikimedia/mediawiki-extensions-Elastica/blob/master/composer.json

And the version out now is 3.0.1 which fixes this issue: https://github.com/ruflin/Elastica/issues/974

Am I reading it wrong or are you one version out with ruflin/Elastica?

Could I just merge the ruflin/Elastica stuff into /extensions/Elastica/vedor/ruflin/? Or is that a horrible idea because of conflicts?

Thanks!

Reedy added a comment.Jan 20 2016, 7:41 PM

We could use 2.3.1, as it supports PHP 5.3.3 (same as MW core), but it's tested against a newer version of ES than WMF is running, which may cause problems until we've upgraded to 1.7.4

We can't use 3.0.1 as it needs PHP 5.4 (till the last WMF servers are upgraded, and we then bump the min PHP version in MW too), and also is tested against ES 2.1.1

If you need to bump the version you're using, please modify the composer.json, and run composer to update it, don't just manually copy/paste stuff in, it probably won't work

See https://www.mediawiki.org/wiki/Composer for more information

Thanks Reedy, is there a roadmap item for when WMF is going to upgrade to 5.4?

WMF is actually using HHVM, bar wikitech that is using 5.5.9 and one deployment server that's still on 5.3

This last server is in the crosshairs to be upgraded/reinstalled in the very near future, and when that's done, we're waiting on T118932 being decided on (chances are, we'll be bumping MW core to 5.5.9 or above)

See T86081 and T87036 for the upgrade of "tin"

The PHP part is probably the easier part, comparatively

When we upgrade to ES 1.7.4, see T122697, we will bump elastica to 2.3.1... I added bugs for both, as the point upgrade won't be quick to do either, but it should be less upheaval than version 2 or 2.1 of ElasticSearch (which probably needs more testing etc)

Reedy renamed this task from Upgrade ElasticSearch to version 2 to Upgrade ElasticSearch to version >=2.1.1.Jan 21 2016, 10:18 AM
Reedy added a comment.Jan 22 2016, 2:44 PM

I'm hitting this issue:

../maintenance/updateSearchIndexConfig.php
content index...
        Fetching Elasticsearch version...2.1.1...ok
        Scanning available plugins...
                head
        Infering index identifier...PHP Fatal error:  Wrong parameters for Exception([string $exception [, long $code [, Exception $previous = NULL]]]) in /extensions/Elastica/vendor/ruflin/elastica/lib/Elastica/Exception/ResponseException.php on line 41

When trying to update the index, I think it's because of this issue: https://github.com/ruflin/Elastica/issues/974
It looks like the last ruflin/Elastica version you merged with was 2.2.1? https://github.com/wikimedia/mediawiki-extensions-Elastica/blob/master/composer.json
And the version out now is 3.0.1 which fixes this issue: https://github.com/ruflin/Elastica/issues/974
Am I reading it wrong or are you one version out with ruflin/Elastica?
Could I just merge the ruflin/Elastica stuff into /extensions/Elastica/vedor/ruflin/? Or is that a horrible idea because of conflicts?
Thanks!

https://gerrit.wikimedia.org/r/#/c/265510/1/includes/Maintenance/ConfigUtils.php,unified

Upgrade to elastic 2.x won't be easy, as far as I know we will have to rewrite the fuzzy like this query which has been removed in es 2.0 but still used by the translate extension.
Concerning cirrus I don't know yet but there are certainly compatibility issues.

EBernhardson moved this task from Needs triage to Search on the Discovery board.Feb 11 2016, 11:24 PM
Reedy renamed this task from Upgrade ElasticSearch to version >=2.1.1 to Upgrade ElasticSearch to version >=2.2.Feb 16 2016, 10:43 AM
Reedy triaged this task as Low priority.

Does this upgrade block something?

Restricted Application added a project: Discovery-Search. · View Herald TranscriptApr 22 2016, 5:20 AM

I wasn't aware this task already existed. Given all the work that @EBernhardson and I put into organising everything in T133120, I'm going to merge this task into that one.

Does this upgrade block something?

As noted in the above task, yes, this blocks us bringing real time updates to the completion suggester. Upgrading to Elasticsearch 2.3 is also, practically speaking, a pre-requisite to upgrading to Elasticsearch 5 in the future. Elasticsearch 5 will allow us to experiment with BM25 and see if it's better than tf-idf.