This will probably be a couple parts:

• Network / Infrastructure / Mostly SRE
  • A proper production config would involve putting LVS (load balancer) in front of the relforge cluster. We can potentially make it work without LVS, but it's not a typical production deployment. This requires SRE to do most of the heavy lifting.
  • Relforge additionaly has no TLS. Usually this is done as part of setting up LVS and giving all the instances a shared name they can respond to. We can configure CirrusSearch to use http, but that is not a typical production deployment. This would land on SRE as well.
  • Typical production search traffic flows through envoy. If we setup LVS/TLS this is easy. If we skip LVS/TLS we need to look into what is possible there, if it can be used without the typical production setup. Gut feeling is we would have to skip envoy and put the host list directly in CirrusSearch configuration.
  • Relforge today only allows connections from analytics networks, a relatively simple puppet patch can relax that to the rest of production.
  • Open up appropriate network rules such that mediawiki can talk to relforge. Today neither bare metal hosts (such as deployment2002) or mw-on-k8s (such as mwscript-k8s) can talk to relforge, all requests get filtered by firewalls. Plausibly fixing the relforge-side firewall will open up to the bare metal hosts, but mw-on-k8s probably needs additional egress rules.
  • Our other clusters are named things like eqiad-chi, we might need to assign a shortname like relforge-alpha or some such to keep consistent with naming, but we should not name it eqiad-relforge as that would imply relforge is part of the unified "CirrusSearch cluster" when it's actually a test system with very limited data and no streaming updates.
  • Relforge does not seem to show up in our grafana dashboards. Look into why and ensure we have visibility into the system. While the dashboards all say elasticsearch, they were simply never renamed. Particularly interesting dashboards:
    • elasticsearch-percentiles is often the first stop when receiving latency alerts to understand what might be going on.
    • elasticsearch-per-node-percentiles would be convenient for understanding latency profiles of the semantic search shard requests.
  • Similarly, relforge has no alerting. As an internal test system it never needed any. Most alerting comes from the same data that shows up on the dashboards, hopefully this is mostly tweaking existing alerts to also include relforge.

• CirrusSearch
  • Need to define the new cluster in operations/mediawiki-config production services, and reference it from the cirrus cluster configuration.
  • The idea is to setup a ftqdprofile (full-text query-dependant profile) that selects for semantic search using the existing search profiles system.
    • We might ponder if we want to somehow limit access to this so someone doesn't see the new search profile in the api allowed-parameters and hammer the test service, but we don't have anything today that fulfills that requirement and can distinguish between mobile apps and arbitrary api traffic. Maybe a user-agent check and a warning response is sufficient. A determined user can always look at our source code/configuration unless we add some sort of secret key (unlikely, no precedent I'm aware of, and just doesn't feel very wiki).
  • Need to implement a semantic search querybuilder that assembles the appropriate query. This likely skips all the AST building that supports special syntax such as keywords. This can likely start out with relatively simple and un-abstracted query builder if we skip the AST and put the full query string directly into the k-nn query. This might be slightly awkward because users can still provide a ftqiprofile (full text query independant profile) to the api call, but we will ignore it.
    • This query builder should probably over-fetch results, as the update method allows for deleted pages to be returned. They will be filtered at the mediawiki rendering layer, but it may result in fewer results than desired.
  • Need to define a new PoolCounter (cluster-wide semaphore) limiting the number of parallel requests to this service and update CirrusSearch to use it when query_type is semantic.
    • If the whole cluster is falling over this is our primary lever to reduce traffic. Ideally we get enough information from capacity testing to set this at a value that doesn't need to be reduced once we go live.
    • If we setup LVS it should be able to gracefully handle one node failing over, but OpenSearch will also internally route requests to any node it knows about by-passing that limit. Since this cluster will be receiving uniform search queries it might make better use of ARS (adaptive replica selection) and avoid the issue (will also be interesting to learn from capacity testing).
  • May need to extend the profile system and/or query building and execution abstractions such that it can ignore ftqiprofile and skip the rescoring phase.
  • It seems likely the existing MediaWiki\Search\SearchResult can satisfy the requirements for this prototype, but we might find it needs changes.
  • The new profile will need to set the query_type to something indicating semantic search. That will flow into our cluster override system, which allows directing queries with a specific type or using specific features to a specified cluster (relforge).
  • Implementation is in-progress for transforming the enterprise structured content snapshots into passages and vectors. The intent is to import those to the relforge cluster once a week.
    • The import will use a production and fallback alias for the imported indices containing the last two completed imports. If the import "succeedes" but has problems an operator will need to use the OpenSearch alias API's to repoint the production alias at the fallback index. If the import fails the production alias will not be updated and we will continue serving the previous data.
    • The fallback index, if it exists, will be deleted prior to importing the next index. If there was a problem with the production index we assume it would have been alias-migrated by then. This helps keep down the disk and memory usage, otherwise we would have to have capacity for 3x index (and 3x replicas, and maybe 3x ceph replication as well).
    • The staleness of data has been determined to be acceptable. New articles will take some days to get into the dataset, but the rate of articles created (and not quickly deleted) is relatively low, articles gain importance and quality over time. Pages that have been deleted but which are still in the index will be filtered by the existing output layers.
    • We might want to have enough of this available, even just as a static file that can be used with gnu parallel / curl, to run include "QPS under indexing load" in capacity testing. We will likely setup timing on the import to run during the quietest part of the week (daytime over the pacific ocean on a weekend), but still.
• The existing backend CirrusSearchRequestSet logging shouldn't need any changes, we will have a full log of queries, results, etc. for later analysis. As always this is a very "operational" dataset and is not the easiest to build analysis from, but it should have all relevant backend data. Mobile will need to manage their own testing and metrics as usual.
  • These logs all have a unique id per web request. That id is returned in the headers as X-Search-ID. If mobile wants to join their data collection against the backend logging they will need to record that value.

• Open Questions
  • What do we do if relforge is getting hammered and rejecting requests? Can we lean on the existing tooling used for dealing with abusive traffic? That tooling is generally human-resource intensive in determining what to block.
  • Can relforge handle 5-10qps? Capacity evaluation to be done in T414623.
  • If relforge does fall over, whats the plan? Do we need a kill-switch, perhaps add a "banned profiles" configuration parameter to CirrusSearch?
    • Wikimedia\FeatureManager is new and can do this without a deploy, but is not stable and says "Please don't bind to it unless you absolutely need to."
    • Based on the use case, I don't think falling back to standard search is appropriate, we likely should return an error. We should double check with mobile/product.