As part of the production upgrade to Elasticsearch 7, we need to ensure that Toolhub is compatible.
We will need to upgrade these python libraries to 7.x versions:
We will also want to update the Elasticsearch Docker image used in our local development and demo environments to a 7.1.x version, ideally the same version as the production hosts will be running.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T248925 Make MediaWiki release tarball compatible with PHP 8.0 | |||
| Resolved | Jdforrester-WMF | T300463 Make PHP 8.0 voting on MW master | |||
| Resolved | None | T283275 Make MW master tests pass on PHP 8.0 | |||
| Resolved | Reedy | T268861 CirrusSearch uses Elastica's Match class | |||
| Resolved | Reedy | T268863 Translate uses Elastica's Match class | |||
| Resolved | matthiasmullie | T268866 WikibaseMediaInfo uses Elastica's Match class | |||
| Invalid | None | T268864 WikibaseCirrusSearch uses Elastica's Match class | |||
| Resolved | Reedy | T268865 WikibaseLexemeCirrusSearch uses Elastica's Match class | |||
| Resolved | EBernhardson | T271777 Bump rufin/elastica (and related libraries) to versions that support PHP 8.0 | |||
| Resolved | Gehel | T263142 [EPIC] Upgrade Elasticsearch to version 7.10 | |||
| Resolved | bd808 | T303444 Ensure Toolhub is compatible with Elasticsearch 7.10 |
@Gehel is there a rough timeline for the production rollout? Exact dates are not needed, but I would like to have some idea of how to prioritize this in our roadmap.
Only very rough timeline at this point: probably at least a few weeks before we are there.
Change 785197 had a related patch set uploaded (by Raymond Ndibe; author: Raymond Ndibe):
[wikimedia/toolhub@main] dev DO NOT MERGE: upgrade to elasticsearch 7.10
While likely optimistic, we are anticipating the search platform side could be ready for elastic 7 deployment by mid may. Would that timeline work here?
That seems possible. We need to do a bit more testing, but it looks like we can change to elastic 7.x libraries pretty easily. I think we will also be able to make some small tweaks prior to that that would make Toolhub agnostic to 6.x and 7.x servers while still using the 6.x libraries so that we do not need lockstep deployments.
The main thing we need to be careful of is getting the proper version pins in place when upgrading as the upstream client libraries after 7.13.x refuse to work with OSS server releases. (Nice job Elastic! I was a huge fanboy until you decided that a fight between Shay and Amazon was worth throwing FOSS in the trash. I wish I could as easily revoke all of my past contributions to your ecosystem.)
Our real pain is likely to be in the (I assume) future migration to OpenSearch. That's going to mean switching out client library vendors and possibly forking upstreams that wrap these client libraries.
Change 797345 had a related patch set uploaded (by Raymond Ndibe; author: Raymond Ndibe):
[wikimedia/toolhub@main] dev DO NOT MERGE: update pyproject.toml and poetry.lock
Change 800311 had a related patch set uploaded (by BryanDavis; author: Raymond Ndibe):
[wikimedia/toolhub@main] dev: Upgrade elasticsearch container to 7.10.2
Change 785197 merged by jenkins-bot:
[wikimedia/toolhub@main] search: remove "standard" token filter
We are planning on migrating to Elasticsearch 7.x around the end of June (exact date to be defined). @bd808 / @Raymond_Ndibe: is that timeline reasonable on your side?
It should work out, yes. I am working on finding a set of libraries that work with both ES 6.x and ES 7.x servers at the same time which is our ideal short term solution (deploy changes while still on a 6.x backend which will keep everything working when the backend upgrades to 7.x). I will either find that solution this week (by my EOD 2022-06-03) or we will likely give up and prepare for a set of patches that we can merge and deploy to achieve 7.x backend compat immediately following the backend upgrade. I will be out on vacation June 4-20, but someone will be capable of doing a Toolhub deployment if necessary during that period.
Change 801413 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[wikimedia/toolhub@main] dev: Use patched elasticsearch library
Change 801413 merged by jenkins-bot:
[wikimedia/toolhub@main] dev: Use patched elasticsearch library
Change 810079 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[operations/deployment-charts@master] toolhub: Bump container version to 2022-06-30-170012-production
Change 810079 merged by jenkins-bot:
[operations/deployment-charts@master] toolhub: Bump container version to 2022-06-30-170012-production
The version of Toolhub running at https://toolhub.wikimedia.org is now using a forked version of elasticsearch-py which includes backwards compatibility flags for hits.total and index types changes in 7.x. With this in place Toolhub will work with both 6.x and 7.x versions of Elasticsearch. We will follow up with a full upgrade to 7.x client libraries following the Elasticsearch 7.10.2 upgrade.
There are pending patches, but I am going to resolve this task to provide a clear signal to watchers of T263142: [EPIC] Upgrade Elasticsearch to version 7.10 that we are no longer a blocker for that update work.
Change 800311 merged by jenkins-bot:
[wikimedia/toolhub@main] dev: Upgrade elasticsearch container to 7.10.2
Change 868183 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[operations/deployment-charts@master] toolhub: bump container to 2022-12-14-185830-production
Change 868183 merged by jenkins-bot:
[operations/deployment-charts@master] toolhub: bump container to 2022-12-14-185830-production