Elasticsearch, a CirrusSearch dependency, is switching to SSPL/Custom licence
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Gacelperfinian
	Jan 15 2021, 1:21 AM

Description

According to their blogpost (https://www.elastic.co/blog/licensing-change):

Starting with the upcoming Elastic 7.11 release, we will be moving the Apache 2.0-licensed code of Elasticsearch and Kibana to be dual licensed under SSPL and the Elastic License, giving users the choice of which license to apply.

Their FAQ on 2021 License Change : https://www.elastic.co/pricing/faq/licensing

Considering this is happening:

WIll CirrusSearch rely on ElasticSearch in the near future (say on version 7.x)?
Will CirrusSearch rely on ElasticSearch in long-term?
According to task T213996, the switch from MongoDB had happened because it was removed in Debian and therefore will be unsuitable for long-term use., and therefore it was unclear if a precedent was achieved. Are there any policy change or clarification if another dependency announced to switch to a license that may not be suitable for MediaWiki?

Context on proprietary relicensing:
https://sfconservancy.org/blog/2020/jan/06/copyleft-equality/

Existing alternative venues:
https://opendistro.github.io/for-elasticsearch/contribute.html ("distribution" or "fork" depending who you ask, no CLA)

Some announced forks:
https://aws.amazon.com/blogs/opensource/stepping-up-for-a-truly-open-source-elasticsearch/
https://logz.io/blog/open-source-elasticsearch-doubling-down/

Related Objects
Search...

Status	Assigned	Task
Resolved	Krinkle	T272111 Elasticsearch, a CirrusSearch dependency, is switching to SSPL/Custom licence
Resolved	• lmata	T272238 Elasticsearch and Kibana are switching to non-OSI-approved SSPL licence
Resolved	bking	T280482 Validate that OpenSearch is a viable replacement for Elasticsearch for CirrusSearch

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Restricted Application added a project: Discovery-Search. · View Herald TranscriptJan 15 2021, 1:21 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Gacelperfinian renamed this task from Elasticsearch will switch to SSPL to Elasticsearch, a CirrusSearch dependency, is switching to SSPL/Custom licence.Jan 15 2021, 1:22 AM

Gacelperfinian added a project: Software-Licensing.Jan 15 2021, 1:24 AM

Gacelperfinian updated the task description. (Show Details)

Gacelperfinian moved this task from Backlog to License switches on the Software-Licensing board.

WIll CirrusSearch rely on ElasticSearch in the near future (say on version 7.x)?

Yes. CirrusSearch is currently on 6.5.4. It will likely upgrade to 6.8.x soon. These versions are EOL and stopped receiving security updates from elastic in Nov 2020.

Will CirrusSearch rely on ElasticSearch in long-term?

Yes? No? Maybe? It will take months to figure this out.

According to task T213996, the switch from MongoDB had happened because it was removed in Debian and therefore will be unsuitable for long-term use., and therefore it was unclear if a precedent was achieved. Are there any policy change or clarification if another dependency announced to switch to a license that may not be suitable for MediaWiki?

The rule of thumb, it may or may not be a strict rule somewhere, is that WMF deploy's software with OSI approved licenses. There is at least some mention under development policys that new services must be licensed under an OSI-approved license.

Also for notetaking: Kibana will also be affected by this change, but I am not sure if WMF stack uses Kibana in some way (if there is any, this should be on another task anyways).

The Translate extension also uses ElasticSearch.

dcausse subscribed.Jan 15 2021, 8:46 AM

Nemo_bis subscribed.Jan 15 2021, 8:47 AM

fgiunchedi subscribed.Jan 15 2021, 10:48 AM

hashar updated the task description. (Show Details)Jan 15 2021, 10:48 AM

MoritzMuehlenhoff subscribed.Jan 15 2021, 10:50 AM

bd808 subscribed.Jan 15 2021, 5:01 PM

• dpifke subscribed.Jan 15 2021, 7:03 PM

Jdforrester-WMF subscribed.Jan 16 2021, 9:42 PM

taavi subscribed.Jan 17 2021, 11:47 AM

I filed T272238: Elasticsearch and Kibana are switching to non-OSI-approved SSPL licence for WMF's ELK setup

Tgr subscribed.Jan 18 2021, 5:26 AM

Gehel moved this task from needs triage to Current work on the Discovery-Search board.Jan 18 2021, 4:03 PM

Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Base subscribed.Jan 19 2021, 4:20 PM

• Mholloway subscribed.Jan 19 2021, 8:22 PM

revi subscribed.Jan 20 2021, 3:16 PM

colewhite subscribed.Jan 20 2021, 8:04 PM

AntiCompositeNumber subscribed.Jan 20 2021, 10:51 PM

WMDE-leszek subscribed.Jan 21 2021, 6:05 AM

akosiaris subscribed.Jan 21 2021, 9:44 AM

akosiaris added a subtask: T272238: Elasticsearch and Kibana are switching to non-OSI-approved SSPL licence.Jan 21 2021, 9:54 AM

jmnote subscribed.Jan 21 2021, 10:10 AM

Ladsgroup subscribed.Jan 22 2021, 4:49 AM

Kronoss subscribed.Jan 22 2021, 10:02 AM

Nemo_bis renamed this task from Elasticsearch, a CirrusSearch dependency, is switching to SSPL/Custom licence to Elasticsearch is going proprietary with a SSPL/custom licence; forks coming.Jan 24 2021, 4:47 PM

Nemo_bis added projects: Wikimedia-Logstash, MediaWiki-extensions-Translate, WMF-General-or-Unknown.

Nemo_bis updated the task description. (Show Details)Jan 24 2021, 4:52 PM

This isn't the task for the Logstash or Translate issues. Please see T272238: Elasticsearch and Kibana are switching to non-OSI-approved SSPL licence for the former and create a new task for the latter, as implied above.

Jdforrester-WMF renamed this task from Elasticsearch is going proprietary with a SSPL/custom licence; forks coming to Elasticsearch, a CirrusSearch dependency, is switching to SSPL/Custom licence.Jan 24 2021, 6:11 PM

CBogen moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.Jan 25 2021, 4:52 PM

Bugreporter mentioned this in T250227: Investigate and evaluate hCaptcha to replace Wikimedia's Fancy Captcha.Jan 25 2021, 7:29 PM

• MPhamWMF subscribed.Feb 1 2021, 4:31 PM

Jdforrester-WMF mentioned this in T271777: Bump rufin/elastica (and related libraries) to versions that support PHP 8.0.Feb 12 2021, 12:30 AM

Framawiki subscribed.Feb 13 2021, 8:23 PM

Addshore subscribed.Feb 22 2021, 10:14 AM

freephile subscribed.Feb 24 2021, 2:42 PM

RheingoldRiver subscribed.Feb 24 2021, 3:59 PM

Bryandamon subscribed.Feb 24 2021, 8:42 PM

Krenair subscribed.Apr 13 2021, 7:56 AM

Gehel added a subtask: T280482: Validate that OpenSearch is a viable replacement for Elasticsearch for CirrusSearch.Apr 19 2021, 8:21 AM

Lens0021 subscribed.Jun 18 2021, 4:29 PM

herron mentioned this in T225125: Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline.Aug 30 2021, 6:41 PM

• toan mentioned this in T297002: Elasticsearch 6.5.4 docker image contains critical security issues.Dec 9 2021, 11:09 AM

• lmata closed subtask T272238: Elasticsearch and Kibana are switching to non-OSI-approved SSPL licence as Resolved.Feb 14 2022, 4:16 AM

For the moment, we're staying on Elasticsearch 7.x (once we've migrated from 6.x to 7.x), but we're blocked on upgrading past 7.10.2 because of licensing. There is a larger question of what exactly are the needs in term of Search that needs to be resolved.

There isn't a clear acceptance criteria on this task, so closing it for now. Feel free to re-open as needed.

In T272111#7730031, @Gehel wrote:

For the moment, we're staying on Elasticsearch 7.x (once we've migrated from 6.x to 7.x), but we're blocked on upgrading past 7.10.2 because of licensing. There is a larger question of what exactly are the needs in term of Search that needs to be resolved.

There isn't a clear acceptance criteria on this task, so closing it for now. Feel free to re-open as needed.

I don't think staying on Elasticsearch 7.x is a long-term solution, is it? I think we can consider this ticket resolved once CirrusSearch has moved to using something that is properly open source under an OSI license.

Gehel edited projects, added Discovery-Search; removed Discovery-Search (Current work).Mar 4 2022, 7:45 AM

Gehel moved this task from needs triage to elastic / cirrus on the Discovery-Search board.

@Legoktm, see T272238 and T280482 - it seems OpenSearch is the relevant replacement in the future, and this task can be left closed.

RhinosF1 subscribed.Apr 10 2022, 4:45 PM

Aklapper mentioned this in T316990: Cirrussearch -> Elasticsearch licensing issue.Sep 4 2022, 10:16 AM

dcausse mentioned this in T317284: The search no longer works with the WMF28 (CirrusSearch).Sep 8 2022, 9:22 AM

TJones mentioned this in T317476: Filter and sort search results of Japanese kana search queries in accordance with how much of the query appears as a consecutive substring.Sep 12 2022, 6:38 PM

TheresNoTime removed a subscriber: RhinosF1.Dec 15 2022, 11:36 PM

Ciencia_Al_Poder subscribed.Dec 23 2022, 5:14 PM

TJones mentioned this in T330893: Map Romanian s&t with comma to cedilla internally.Mar 1 2023, 8:44 PM

Strainu subscribed.Mar 2 2023, 11:42 AM

bking subscribed.Feb 26 2024, 10:53 PM

Frostly subscribed.Mar 21 2024, 11:23 PM

Gehel removed Gehel as the assignee of this task.Apr 15 2024, 2:00 PM

bking closed subtask T280482: Validate that OpenSearch is a viable replacement for Elasticsearch for CirrusSearch as Invalid.May 1 2024, 1:52 PM

bking reopened subtask T280482: Validate that OpenSearch is a viable replacement for Elasticsearch for CirrusSearch as Open.

bking changed the status of subtask T280482: Validate that OpenSearch is a viable replacement for Elasticsearch for CirrusSearch from Open to In Progress.

Justin_C_Lloyd subscribed.May 1 2024, 1:59 PM

Gehel closed subtask T280482: Validate that OpenSearch is a viable replacement for Elasticsearch for CirrusSearch as Resolved.May 10 2024, 8:59 AM

Gehel mentioned this in T370147: [Epic] Migration of the Elasticsearch 7.10 search cluster to OpenSearch.Jul 16 2024, 12:50 PM

TehKittyCat subscribed.Jul 18 2024, 2:40 PM

For the record keeping: https://www.elastic.co/blog/elasticsearch-is-open-source-again

Jaideraf subscribed.Sep 5 2024, 11:17 AM

Regarding Elastic as used for Logstash:

In T272238#7706582, @lmata wrote on 14 Feb 2022:

[..] The Observability team identified two potential options: OpenSearch and Grafana Loki.

[..] In conclusion, the team recommended moving forward with OpenSearch [..].

After this evaluation, we implemented a testing environment in beta [..], followed by deployments to production [..]. Both codfw and eqiad have been migrated to OpenSearch, thus removing the observability stack of scope for SSPL concerns.

Regarding Elastic for MediaWiki search:

In T280482#9760984, @bking wrote on 1 May 2024:

Based on discussions with @dcausse and @EBernhardson last week, I believe we (as in Search Platform team) have decided that Opensearch is our only choice to replace Elasticsearch. [..]

In T370147#10125422, @Gehel wrote on 6 Sep 2024:

Given the recent changes to Elasticsearch license to AGPL, we are likely going to stay on Elasticsearch and upgrade to latest version. Final decision to come...

In T370661#10220958, @dr0ptp4kt wrote on 11 Oct 2024:

Update: Search Platform has reviewed further, and it intends to stick with its decision to migrate to OpenSearch. https://www.mediawiki.org/wiki/Wikimedia_Search_Platform/Decision_Records/Search_backend_replacement_technology has been updated.

[…]

In T370147#10459323, @bking wrote on 14 Jan 2025:

Following up on Gehel 's last comment. Search Platform team (along with stakeholders) has officially decided to migrate to Opensearch. See T370661 for more details about how any why we made that decision.