I've posted this at https://www.mediawiki.org/wiki/Draft:ABaso_(WMF)/Wikimedia_Search_Platform/Decision_Records/Search_backend_replacement_technology .

Full blown decision briefs with upper level sign off are usually more appropriate for decisions involving a significant amount of risk and controversy when the decision isn't easily made locally. Since we had discussion last week, the input from Trey I think shifts the context a bit from what could have been a difficult decision with controversy to what notionally seems somewhat simpler and uncontroversial. I've tried to reflect the decision rights piece in the draft wiki page decision record as much as possible. I have given our director a heads up about the forthcoming decision record / brief and will be looking for feedback from the director and VP as something of a right of first refusal, in case there's something we're seriously missing here.

Also, I have to allow that perhaps there are graver concerns with going to the Apache-licensed software than the SSPL-licensed software that I'm not understanding, and it's good to enumerate such concerns if they exist.

@bking @dcausse @EBernhardson @Gehel @pfischer @RKemper @TJones would you please review and, as appropriate, edit https://www.mediawiki.org/wiki/Draft:ABaso_(WMF)/Wikimedia_Search_Platform/Decision_Records/Search_backend_replacement_technology by close of business Thursday, 25-July-2024? If there are things where you're not sure about editing but would like me to figure out how to word or resolve a tension, please by all means feel free to use the Discussion page on this Draft:ABaso_(WMF) page.

Apologies in advance for things I've overlooked or misunderstood - your help is appreciated.

Following your review, or maybe even sooner if it's not a problem for you (please speak up), I'd like to get this under the decision records section of Search Platform documentation and ask for further input from the director and VP of our unit plus get it in front of some Site Reliability Engineering folks most affected in Mark's part of the org.

Yesterday's announcement that Elasticsearch is returning to an OSI-approved license makes this an open question once again. Reopening so we can have this discussion one (?) more time.

As third party user of mediawiki and planning on running the advanced search extensions that rely on ElasticSearch, wanted to bring up a few concerns

• Will the switch to OpenSearch result in any conflict with the remaining ELK stack such as Kibana, Beats, or Logstash? aware OpenSearch Dashboard is OSS fork of Kibana
• What is the impact of features lost from switching from ElasticSearch to OpenSearch?

Thanks @SpookyGhost8 for the questions!

In the Wikimedia-hosted environment, I don't really anticipate conflicts with the other parts of the observability stack. Maybe more conservatively, I can imagine how some payloads may need small adjustments for logging, but nothing dramatic. Now, this said, for environments that aren't Wikimedia-hosted: I think this probably depends upon the manner in which such hosting uses Elasticsearch tooling for its observability needs: do you happen to be utilizing that sort of functionality and if so, would you be okay sharing some details? We're very interested to learn about complications for third parties, so that we can plan ways to make things at least a little easier.

Right now we're not clear that there would be any loss of features so to speak with a switch from Elasticsearch to OpenSearch; it's expected of course that migration would likely be a bit more work to go to OpenSearch as contrasted with using Elasticsearch (at least this is the theory, as Elastic tries to make upgrades easy enough for its customers).

Now, I should also note that, due to the recent AGPL support Elastic has recently introduced (AGPL is OSI-compatible, but does require some analysis for certain use cases) we're re-examining the matters around staying on Elasticsearch and upgrading Elasticsearch versus doing a switch to OpenSearch. Although the glide path for OpenSearch looks navigable, at least at first glance the glide path for Elasticsearch looks even more direct- but, there are particulars, for example around what kinds of capabilities are or aren't supported easily with either option (e.g., vector embeddings options and Learning to Rank support within the ecosystems), in addition to considerations about licensing. There is also the matter of the degree of standardization within the Wikimedia hosting environment: more of the Wikimedia hosting environment has leaned toward OpenSearch; and although it's true that the needs of end users of Wikipedias, Commons, and other sibling projects are different than those of a good number of other parts of the hosting environments, there is something to be said for consistency and more uniform knowledge about the technology powering search experiences.

We do plan to share an update as we get further along in our re-examination of the alternatives.

We're very interested to learn about complications for third parties, so that we can plan ways to make things at least a little easier.

For those who don't know, Semantic MediaWiki has an Elasticsearch integration, and we decided to "Make Semantic MediaWiki compatible with the versions of Elasticsearch or OpenSearch (depending on the outcome) that are supported by CirrusSearch;".

As Semantic MediaWiki (SMW) is one of the most popular extension outside WMF projects (see the numbers), it is a good example of third party involved in the subject. We don't know how many SMW users do use Elasticsearch, but it is important that you have all that information.

@dr0ptp4kt At the moment, I rely on extension:TitleKey for enhanced wiki search across several third party wikis. I am planning to add on ElasticSearch functionality later this week for a wiki that I manage. I have done preliminary research as far as installation and discovered the minimum hardware requirement is 2 vCPU & 2 GB RAM just to start the services. While I was able to successfully install ElasticSearch 7.10.2 and OpenSearch, OpenSearch gave me a bit more trouble in terms of installing due to the lack of proper documentation. I did document what I needed so not as big of an impact since starting from scratch. Hopefully with the recent announcement of OpenSearch Foundation partnership with the Linux Foundation, documentation can be improved.

I am aware of a couple large third party independent wikis that are using ElasticSearch so that migration might be painful to OpenSearch.

If OpenSearch and OpenSearch Dashboard is the decided path, would that be compatible with logstash and beats?

In T370661#10154178, @SpookyGhost8 wrote:

If OpenSearch and OpenSearch Dashboard is the decided path, would that be compatible with logstash and beats?

The use of OpenSearch for log collection / management is not strongly linked to full text search. You could use different backends for search vs logs (yes, I understand that this would make things more complicated to operate). In WMF, those are treated as 2 separate problems (we are currently using OpenSearch for logs and Elasticsearch for search). Our team doesn't really have expertise in log management, and our decision is likely to be based on which solution is better for Search, not taking logging into much consideration.