Page MenuHomePhabricator
Create Task
Maniphest T413969

Make semantic search accessible through Action API
Open, Needs TriagePublic8 Estimated Story Points
Actions

Description

For the semantic search MVP on Android, we need an HTTP endpoint that they can consume to retrieve semantic search results. The Android app uses the Action API for both, prefix and full text search. For matters of convenience, we would reuse the Action API. Apparently, there are multiple discriminators already:

  • *gps* gps(search|limit|offset) = prefix search + generator=prefix
  • *gsr* gsr(search|limit|offset) = full text search + generator=fulltext

TBD: Do we need another param family or can we expand the possible values, for example, generator=semantic?

Alternatively might be solved via search profile, see fulltext query dependent profile.

Corresponding Android app task: T412986

AC:

  • w/api.php? can be called to fetch semantic results

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
semantic: Provide instructions from profile to queryEbernhardsonmediawiki/extensions/CirrusSearchmaster+38 -1
cirrus: Align semanticsearch cluster group name with routingEbernhardsonoperations/mediawiki-configmaster+1 -1
cirrus: Correct semantic builder configEbernhardsonoperations/mediawiki-configmaster+9 -7
cirrus: Add semantic search test clusterEbernhardsonoperations/mediawiki-configmaster+40 -5
Wire up semantic query buildingEbernhardsonmediawiki/extensions/CirrusSearchwmf/1.46.0-wmf.18+645 -13
Introduce a Semantic Search query route and builderEbernhardsonmediawiki/extensions/CirrusSearchwmf/1.46.0-wmf.18+647 -9
Introduce a Semantic Search query route and builderEbernhardsonmediawiki/extensions/CirrusSearchmaster+647 -9
Wire up semantic query buildingEbernhardsonmediawiki/extensions/CirrusSearchmaster+645 -13
Initial implementation of Semantic query builderEbernhardsonmediawiki/extensions/CirrusSearchmaster+344 -2
opensearch-semantic-search: Permit pods up to 32GB RAMBkingoperations/deployment-chartsmaster+9 -9
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenEBernhardsonT413969 Make semantic search accessible through Action API
 OpenNoneT414683 Support the deployment of opensearch 3.x clusters
 DeclinedNoneT414685 Deploy opensearch-k8s-operator 2.8.0 to the dse-k8s Kubernetes clusters
 ResolvedbrouberolT414686 Build and release the opensearch-k8s-operator 2.8.0 docker image
 DeclinedNoneT414689 Deploy the opensearch operator 2.8.0 to dse-k8s-eqiad
 ResolvedbkingT414693 Build and deploy the opensearch 3.x docker image
 ResolvedbkingT414697 Build the required plugins for opensearch 3
 ResolvedbkingT415699 Create a trixie-based OpenSearch 3 docker image
 ResolvedbkingT414695 Create openjdk-21 docker images based on Bookworm
 ResolvedGehelT414703 Deploy the opensearch-semantic-search opensearch 3.x clusters
 ResolvedRKemperT414691 Deploy the opensearch semantic-search cluster on dse-k8s-eqiad
 ResolvedRKemperT414699 Define the opensearch-semantic-search kubeconfig
 ResolvedRKemperT414702 Provision opensearch-semantic-search namespaces
 ResolvedRKemperT414692 Deploy the opensearch semantic-search cluster on dse-k8s-codfw
 ResolvedRKemperT414700 Define the opensearch-semantic-search kubeconfig
 DuplicateRKemperT414701 Define the opensearch-semantic-search namespace
 DeclinedEBernhardsonT417087 Deploy Qwen3-Reranker-0.6B inference service for semantic search reranking
 ResolvedbkingT418405 Plan hardware expansion for Semantic Search
 ResolvedBTullisT418776 Consider options for automating the setting of read_ahead_kb for OpenSearch pod volumes
 ResolvedbkingT419041 Enable custom readahead settings for Ceph block devices serving workload on the dse-k8s clusters
 ResolvedEBernhardsonT419174 qwen3-embedding:predict returning 503 to all requests
 OpenNoneT424090 Undeploy opensearch-semantic-search environments from the DSE k8s clusters - All data to be removed 21 May 2026 @ 2100 UTC

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 7 2026, 12:11 PM
pfischer updated the task description. (Show Details)Jan 9 2026, 11:23 AM
Dbrant subscribed.Jan 9 2026, 3:49 PM

This is great - any kind of integration with the existing Action API would be equally simple for us to consume on the client side, whether it's adding something like generator=semantic, or something like generator=search with gsrwhat=semantic, whatever makes more sense architecturally on the backend.
And presumably this would also reuse the snippet field (returned by other types of searches) to provide the paragraph that's relevant to the semantic result.

pfischer updated the task description. (Show Details)Jan 12 2026, 4:41 PM
pfischer edited projects, added: Discovery-Search (2026.01.05 - 2026.01.30); removed: Discovery-Search.
pfischer set the point value for this task to 8.Jan 12 2026, 4:58 PM
• dcausse added a project: Semantic Search.Jan 13 2026, 3:36 PM
dr0ptp4kt subscribed.Jan 13 2026, 3:37 PM
EBernhardson moved this task from Incoming to In Progress on the Discovery-Search (2026.01.05 - 2026.01.30) board.Jan 14 2026, 7:37 PM
bking subscribed.Jan 14 2026, 9:35 PM
EBernhardson claimed this task.EditedJan 14 2026, 11:00 PM
EBernhardson subscribed.

This will probably be a couple parts:

  • Network / Infrastructure / Mostly SRE
    • A proper production config would involve putting LVS (load balancer) in front of the relforge cluster. We can potentially make it work without LVS, but it's not a typical production deployment. This requires SRE to do most of the heavy lifting.
    • Relforge additionaly has no TLS. Usually this is done as part of setting up LVS and giving all the instances a shared name they can respond to. We can configure CirrusSearch to use http, but that is not a typical production deployment. This would land on SRE as well.
    • Typical production search traffic flows through envoy. If we setup LVS/TLS this is easy. If we skip LVS/TLS we need to look into what is possible there, if it can be used without the typical production setup. Gut feeling is we would have to skip envoy and put the host list directly in CirrusSearch configuration.
    • Relforge today only allows connections from analytics networks, a relatively simple puppet patch can relax that to the rest of production.
    • Open up appropriate network rules such that mediawiki can talk to relforge. Today neither bare metal hosts (such as deployment2002) or mw-on-k8s (such as mwscript-k8s) can talk to relforge, all requests get filtered by firewalls. Plausibly fixing the relforge-side firewall will open up to the bare metal hosts, but mw-on-k8s probably needs additional egress rules.
    • Our other clusters are named things like eqiad-chi, we might need to assign a shortname like relforge-alpha or some such to keep consistent with naming, but we should not name it eqiad-relforge as that would imply relforge is part of the unified "CirrusSearch cluster" when it's actually a test system with very limited data and no streaming updates.
    • Relforge does not seem to show up in our grafana dashboards. Look into why and ensure we have visibility into the system. While the dashboards all say elasticsearch, they were simply never renamed. Particularly interesting dashboards:
      • elasticsearch-percentiles is often the first stop when receiving latency alerts to understand what might be going on.
      • elasticsearch-per-node-percentiles would be convenient for understanding latency profiles of the semantic search shard requests.
    • Similarly, relforge has no alerting. As an internal test system it never needed any. Most alerting comes from the same data that shows up on the dashboards, hopefully this is mostly tweaking existing alerts to also include relforge.
  • CirrusSearch
    • Need to define the new cluster in operations/mediawiki-config production services, and reference it from the cirrus cluster configuration.
    • The idea is to setup a ftqdprofile (full-text query-dependant profile) that selects for semantic search using the existing search profiles system.
      • We might ponder if we want to somehow limit access to this so someone doesn't see the new search profile in the api allowed-parameters and hammer the test service, but we don't have anything today that fulfills that requirement and can distinguish between mobile apps and arbitrary api traffic. Maybe a user-agent check and a warning response is sufficient. A determined user can always look at our source code/configuration unless we add some sort of secret key (unlikely, no precedent I'm aware of, and just doesn't feel very wiki).
    • Need to implement a semantic search querybuilder that assembles the appropriate query. This likely skips all the AST building that supports special syntax such as keywords. This can likely start out with relatively simple and un-abstracted query builder if we skip the AST and put the full query string directly into the k-nn query. This might be slightly awkward because users can still provide a ftqiprofile (full text query independant profile) to the api call, but we will ignore it.
      • This query builder should probably over-fetch results, as the update method allows for deleted pages to be returned. They will be filtered at the mediawiki rendering layer, but it may result in fewer results than desired.
    • Need to define a new PoolCounter (cluster-wide semaphore) limiting the number of parallel requests to this service and update CirrusSearch to use it when query_type is semantic.
      • If the whole cluster is falling over this is our primary lever to reduce traffic. Ideally we get enough information from capacity testing to set this at a value that doesn't need to be reduced once we go live.
      • If we setup LVS it should be able to gracefully handle one node failing over, but OpenSearch will also internally route requests to any node it knows about by-passing that limit. Since this cluster will be receiving uniform search queries it might make better use of ARS (adaptive replica selection) and avoid the issue (will also be interesting to learn from capacity testing).
    • May need to extend the profile system and/or query building and execution abstractions such that it can ignore ftqiprofile and skip the rescoring phase.
    • It seems likely the existing MediaWiki\Search\SearchResult can satisfy the requirements for this prototype, but we might find it needs changes.
    • The new profile will need to set the query_type to something indicating semantic search. That will flow into our cluster override system, which allows directing queries with a specific type or using specific features to a specified cluster (relforge).
    • Implementation is in-progress for transforming the enterprise structured content snapshots into passages and vectors. The intent is to import those to the relforge cluster once a week.
      • The import will use a production and fallback alias for the imported indices containing the last two completed imports. If the import "succeedes" but has problems an operator will need to use the OpenSearch alias API's to repoint the production alias at the fallback index. If the import fails the production alias will not be updated and we will continue serving the previous data.
      • The fallback index, if it exists, will be deleted prior to importing the next index. If there was a problem with the production index we assume it would have been alias-migrated by then. This helps keep down the disk and memory usage, otherwise we would have to have capacity for 3x index (and 3x replicas, and maybe 3x ceph replication as well).
      • The staleness of data has been determined to be acceptable. New articles will take some days to get into the dataset, but the rate of articles created (and not quickly deleted) is relatively low, articles gain importance and quality over time. Pages that have been deleted but which are still in the index will be filtered by the existing output layers.
      • We might want to have enough of this available, even just as a static file that can be used with gnu parallel / curl, to run include "QPS under indexing load" in capacity testing. We will likely setup timing on the import to run during the quietest part of the week (daytime over the pacific ocean on a weekend), but still.
  • The existing backend CirrusSearchRequestSet logging shouldn't need any changes, we will have a full log of queries, results, etc. for later analysis. As always this is a very "operational" dataset and is not the easiest to build analysis from, but it should have all relevant backend data. Mobile will need to manage their own testing and metrics as usual.
    • These logs all have a unique id per web request. That id is returned in the headers as X-Search-ID. If mobile wants to join their data collection against the backend logging they will need to record that value.
  • Open Questions
    • What do we do if relforge is getting hammered and rejecting requests? Can we lean on the existing tooling used for dealing with abusive traffic? That tooling is generally human-resource intensive in determining what to block.
    • Can relforge handle 5-10qps? Capacity evaluation to be done in T414623.
    • If relforge does fall over, whats the plan? Do we need a kill-switch, perhaps add a "banned profiles" configuration parameter to CirrusSearch?
      • Wikimedia\FeatureManager is new and can do this without a deploy, but is not stable and says "Please don't bind to it unless you absolutely need to."
      • Based on the use case, I don't think falling back to standard search is appropriate, we likely should return an error. We should double check with mobile/product.
EBernhardson added subscribers: RKemper, CDanis, • dcausse.Jan 14 2026, 11:02 PM

@dcausse @CDanis @bking @RKemper The above plan is my first draft of how we go from where we are today, to having semantic search available for testing in production. Please review.

I wasn't sure where exactly to put this, ideally anyone could edit it. I tried using a phabricator paste but it renders terribly. We can maybe move it into the ticket description?

• dcausse added a comment.Jan 15 2026, 10:50 AM
In T413969#11523393, @EBernhardson wrote:

This will probably be a couple parts:

  • CirrusSearch
    • Need to define the new cluster in operations/mediawiki-config production services, and reference it from the cirrus cluster configuration.
    • The idea is to setup a ftqdprofile (full-text query-dependant profile) that selects for semantic search using the existing search profiles system.
      • We might ponder if we want to somehow limit access to this so someone doesn't see the new search profile in the api allowed-parameters and hammer the test service, but we don't have anything today that fulfills that requirement and can distinguish between mobile apps and arbitrary api traffic. Maybe a user-agent check and a warning response is sufficient. A determined user can always look at our source code/configuration unless we add some sort of secret key (unlikely, no precedent I'm aware of, and just doesn't feel very wiki).

We also discussed using an undocumented API param for now (cirrusEnableXYZ) but this is just buying a bit more time.
Cirrus has some facility to do the routing with DefaultSearchQueryDispatchService which could be helpful.

  • Need to implement a semantic search querybuilder that assembles the appropriate query. This likely skips all the AST building that supports special syntax such as keywords. This can likely start out with relatively simple and un-abstracted query builder if we skip the AST and put the full query string directly into the k-nn query. This might be slightly awkward because users can still provide a ftqiprofile (full text query independence profile) to the api call, but we will ignore it.

Agreed, I doubt there's a need to interpret the AST, in the long run we might perhaps honor double quote with an additional filter.

  • May need to extend the profile system and/or query building and execution abstractions such that it can ignore ftqiprofile and skip the rescoring phase.

If using the DefaultSearchQueryDispatchService a new "semantic_search" profile context will have to be created and this one will dictate what rescore profile to use and could be forced to "empty".
If a user explicitly selects a rescore profile (not using engine_autoselect for ftqiprofile) I think we could re-route to classic fulltext (the SearchQuery class keeps the list of forced profile and this can be inspected to make this decision).
Haven't looked closely but this could be an indication that using ftqbprofile to route to the knn query builder might not play well as it gives the false impression that you can assemble query bits freely. So perhaps using another criteria (custom cirrus param for now?) to trigger the knn search could be easier, the query router could cancel the knn query if some profiles have been selected explicitly (ftqiprofile/ftqbprofile != engine_autoselect).

  • Open Questions
    • What do we do if relforge is getting hammered and rejecting requests? Can we lean on the existing tooling used for dealing with abusive traffic? That tooling is generally human-resource intensive in determining what to block.

I think the android app will have to allow for graceful failures at this point.

bking added a subscriber: brouberol.Jan 15 2026, 2:42 PM

@pfischer , @brouberol and myself have had a couple of discussions about this today. We believe the most efficient way forward is to run this on Kubernetes. We will create subtasks from this ticket and start dividing up the work.

We'll be doing many of the same tasks we already outlined in T414217 (alpha-testing the latest Kubernetes operator), but instead of deploying alpha software, we'll deploy the latest stable 2.8 tag of the operator , along with Balthazar's backported watchNamespace patch.

brouberol added a subtask: T414683: Support the deployment of opensearch 3.x clusters.Jan 15 2026, 2:52 PM
bking added a comment.EditedJan 15 2026, 3:26 PM

Per Erik's comment above:

elasticsearch-percentiles is often the first stop when receiving latency alerts to understand what might be going on.
elasticsearch-per-node-percentiles would be convenient for understanding latency profiles of the semantic search shard requests.

These important metrics are sourced from our custom exporter, which is not currently available in OpenSearch on K8s. (See T414345 for a further discussion). @EBernhardson / @pfischer , would you consider these metrics a hard requirement for the project? If so, we might have to:

A. (SRE) Modify the helm chart to run our custom exporter as a sidecar
B. (Search Platform) rewrite our custom exporter as an OpenSearch plugin.

Let us know if this is a hard requirement and if so, level of effort for implementing the exporter in Java as an OpenSearch plugin. SRE will also look into making a docker image for the exporter and how to integrate it into the chart if these metrics are indeed required.

EBernhardson added a comment.Jan 15 2026, 3:38 PM
In T413969#11525371, @bking wrote:

Per Erik's comment above:

elasticsearch-percentiles is often the first stop when receiving latency alerts to understand what might be going on.
elasticsearch-per-node-percentiles would be convenient for understanding latency profiles of the semantic search shard requests.

This important metrics are sourced from our custom exporter, which is not currently available in OpenSearch on K8s. (See T414345 for a further discussion). @EBernhardson / @pfischer , would you consider these metrics a hard requirement for the project? If so, we might have to:

A. (SRE) Modify the helm chart to run our custom exporter as a sidecar
B. (Search Platform) rewrite our custom exporter as an OpenSearch plugin.

Let us know if this is a hard requirement and if so, level of effort for implementing the exporter in Java as an OpenSearch plugin. SRE will also look into making a docker image for the exporter and how to integrate it into the chart if these metrics are indeed required.

I did a quick look through the metrics exposed by prometheus-elasticsearch-exporter and prometheus-wmf-elasticsearch-exporter in the current prod clusters, i think the upstream exporter, already available in k8s iiuc, is a hard requirement. The wmf exporter feeds the per-node-percentiles graph and would be nice to have, but is not a requirement for deployment.

bking created subtask T414695: Create openjdk-21 docker images based on Bookworm.Jan 15 2026, 3:54 PM
brouberol added a subtask: T414703: Deploy the opensearch-semantic-search opensearch 3.x clusters.Jan 15 2026, 4:10 PM
bking changed the status of subtask T414695: Create openjdk-21 docker images based on Bookworm from Open to In Progress.Jan 15 2026, 4:54 PM
EBernhardson added a comment.Jan 15 2026, 6:42 PM
In T413969#11524363, @dcausse wrote:

If using the DefaultSearchQueryDispatchService a new "semantic_search" profile context will have to be created and this one will dictate what rescore profile to use and could be forced to "empty".
If a user explicitly selects a rescore profile (not using engine_autoselect for ftqiprofile) I think we could re-route to classic fulltext (the SearchQuery class keeps the list of forced profile and this can be inspected to make this decision).
Haven't looked closely but this could be an indication that using ftqbprofile to route to the knn query builder might not play well as it gives the false impression that you can assemble query bits freely. So perhaps using another criteria (custom cirrus param for now?) to trigger the knn search could be easier, the query router could cancel the knn query if some profiles have been selected explicitly (ftqiprofile/ftqbprofile != engine_autoselect).

That all makes sense. Given the current constraints and timelines, a custom query param seems a reasonable way forward.

bking closed subtask T414695: Create openjdk-21 docker images based on Bookworm as Resolved.Jan 20 2026, 11:04 PM
gerritbot added a comment.Jan 21 2026, 6:50 PM

Change #1229643 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] [WIP] Introduce a Semantic Search query route

https://gerrit.wikimedia.org/r/1229643

gerritbot added a project: Patch-For-Review.Jan 21 2026, 6:50 PM

Change #1229644 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] Initial implementation of Semantic query builder

https://gerrit.wikimedia.org/r/1229644

gerritbot added a comment.Jan 21 2026, 6:50 PM

Change #1229645 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] [WIP] Wire up semantic query building

https://gerrit.wikimedia.org/r/1229645

pfischer edited projects, added: Discovery-Search (2026.02.02 - 2026.02.27); removed: Discovery-Search (2026.01.05 - 2026.01.30).Feb 9 2026, 12:47 PM
pfischer moved this task from Incoming to In Progress on the Discovery-Search (2026.02.02 - 2026.02.27) board.
EBernhardson created subtask T417087: Deploy Qwen3-Reranker-0.6B inference service for semantic search reranking.Feb 10 2026, 10:03 PM
Seddon mentioned this in T412986: [L] Adopt Hybrid Search API Supporting Lexical and Semantic Retrieval for Android MVP.Feb 13 2026, 5:15 AM
Gehel closed subtask T414703: Deploy the opensearch-semantic-search opensearch 3.x clusters as Resolved.Feb 17 2026, 2:45 PM
bking added a subtask: T418405: Plan hardware expansion for Semantic Search.Feb 25 2026, 6:33 PM
gerritbot added a comment.Feb 26 2026, 4:53 PM

Change #1244713 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Add semantic search test cluster

https://gerrit.wikimedia.org/r/1244713

bking closed subtask T418405: Plan hardware expansion for Semantic Search as Resolved.Feb 26 2026, 8:36 PM
gerritbot added a comment.Feb 27 2026, 3:09 PM

Change #1245384 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] opensearch-semantic-search: Permit pods up to 32GB RAM

https://gerrit.wikimedia.org/r/1245384

gerritbot added a comment.Feb 27 2026, 3:29 PM

Change #1245384 merged by jenkins-bot:

[operations/deployment-charts@master] opensearch-semantic-search: Permit pods up to 32GB RAM

https://gerrit.wikimedia.org/r/1245384

EBernhardson added a comment.Mar 2 2026, 3:24 PM

Reviewed the prior work list to see where we are at:

In T413969#11523393, @EBernhardson wrote:

This will probably be a couple parts:

  • Network / Infrastructure / Mostly SRE

This portion has mostly been resolved, although some setup is still in progress. An opensearch cluster has been deployed to the dpe-sre k8s cluster and we have approval for enough resources to run the initial tests. We are still working on optimizing the latency / resource usage of this setup.

  • CirrusSearch

Various patches have been written but are pending review/merge:

  • Build and run semantic search queries from CirrusSearch (and the patches it depends on).
    • This works by attaching &cirrusSemanticSearch to any full-text search query.
    • Example query (once deployed).
    • Example response: P89521. This is the same as our standard response format, but the snippet is populated with the paragraph content and section identifies where that paragraph came from. Section may be missing if it's from the lead section of an article.
    • Logged queries (in hadoop at event.mediawiki_cirrussearch_request) are identified by having semantic in the query_type field.
    • Logged queries can be joined against frontend logging by recording the X-Search-Id http response header.
  • Configure cluster for prod access.
  • Data transformations necessary for converting enterprise dumps into opensearch indexes is mostly complete, some iteration will likely continue.
  • Open Questions

Capacity evaluation is being done this week in T414623

EBernhardson moved this task from In Progress to Needs Review on the Discovery-Search (2026.02.02 - 2026.02.27) board.Mar 2 2026, 4:04 PM
bking added a subtask: T418776: Consider options for automating the setting of read_ahead_kb for OpenSearch pod volumes.Mar 2 2026, 5:46 PM
gerritbot added a comment.Mar 3 2026, 4:21 PM

Change #1229644 abandoned by Ebernhardson:

[mediawiki/extensions/CirrusSearch@master] Initial implementation of Semantic query builder

Reason:

Squashed into I1a7f5ba87b7e9

https://gerrit.wikimedia.org/r/1229644

gerritbot added a comment.Mar 4 2026, 3:45 PM

Change #1229643 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Introduce a Semantic Search query route and builder

https://gerrit.wikimedia.org/r/1229643

gerritbot added a comment.Mar 4 2026, 3:45 PM

Change #1229645 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Wire up semantic query building

https://gerrit.wikimedia.org/r/1229645

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.19; 2026-03-10).Mar 4 2026, 4:00 PM
BTullis closed subtask T418776: Consider options for automating the setting of read_ahead_kb for OpenSearch pod volumes as Resolved.Mar 4 2026, 5:38 PM
gerritbot added a comment.Mar 4 2026, 7:36 PM

Change #1248084 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@wmf/1.46.0-wmf.18] Introduce a Semantic Search query route and builder

https://gerrit.wikimedia.org/r/1248084

gerritbot added a comment.Mar 4 2026, 7:36 PM

Change #1248085 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@wmf/1.46.0-wmf.18] Wire up semantic query building

https://gerrit.wikimedia.org/r/1248085

gerritbot added a comment.Mar 4 2026, 9:51 PM

Change #1248084 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@wmf/1.46.0-wmf.18] Introduce a Semantic Search query route and builder

https://gerrit.wikimedia.org/r/1248084

gerritbot added a comment.Mar 4 2026, 9:54 PM

Change #1248085 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@wmf/1.46.0-wmf.18] Wire up semantic query building

https://gerrit.wikimedia.org/r/1248085

Stashbot added a comment.Mar 4 2026, 9:54 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-04T21:54:39Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1248084|Introduce a Semantic Search query route and builder (T413969)]], [[gerrit:1248085|Wire up semantic query building (T413969)]]

ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.18; 2026-03-03); removed: MW-1.46-notes (1.46.0-wmf.19; 2026-03-10).Mar 4 2026, 10:00 PM
Stashbot added a comment.Mar 4 2026, 10:14 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-04T22:14:24Z] <tgr@deploy2002> tgr, ebernhardson: Backport for [[gerrit:1248084|Introduce a Semantic Search query route and builder (T413969)]], [[gerrit:1248085|Wire up semantic query building (T413969)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Mar 4 2026, 10:33 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-04T22:33:07Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1248084|Introduce a Semantic Search query route and builder (T413969)]], [[gerrit:1248085|Wire up semantic query building (T413969)]] (duration: 38m 28s)

gerritbot added a comment.Mar 5 2026, 2:59 PM

Change #1244713 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Add semantic search test cluster

https://gerrit.wikimedia.org/r/1244713

Stashbot added a comment.Mar 5 2026, 3:00 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-05T15:00:46Z] <ebernhardson@deploy2002> Started scap sync-world: Backport for [[gerrit:1244713|cirrus: Add semantic search test cluster (T413969)]]

Stashbot added a comment.Mar 5 2026, 3:02 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-05T15:02:56Z] <ebernhardson@deploy2002> ebernhardson: Backport for [[gerrit:1244713|cirrus: Add semantic search test cluster (T413969)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Mar 5 2026, 3:10 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-05T15:10:04Z] <ebernhardson@deploy2002> Finished scap sync-world: Backport for [[gerrit:1244713|cirrus: Add semantic search test cluster (T413969)]] (duration: 09m 18s)

gerritbot added a comment.Mar 5 2026, 3:12 PM

Change #1248506 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Correct semantic builder config

https://gerrit.wikimedia.org/r/1248506

gerritbot added a comment.Mar 5 2026, 3:15 PM

Change #1248506 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Correct semantic builder config

https://gerrit.wikimedia.org/r/1248506

Stashbot added a comment.Mar 5 2026, 3:16 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-05T15:15:59Z] <ebernhardson@deploy2002> Started scap sync-world: Backport for [[gerrit:1248506|cirrus: Correct semantic builder config (T413969)]]

Stashbot added a comment.Mar 5 2026, 3:18 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-05T15:18:05Z] <ebernhardson@deploy2002> ebernhardson: Backport for [[gerrit:1248506|cirrus: Correct semantic builder config (T413969)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Mar 5 2026, 3:23 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-05T15:23:39Z] <ebernhardson@deploy2002> Finished scap sync-world: Backport for [[gerrit:1248506|cirrus: Correct semantic builder config (T413969)]] (duration: 07m 39s)

gerritbot added a comment.Mar 5 2026, 3:25 PM

Change #1248508 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Align semanticsearch cluster group name with routing

https://gerrit.wikimedia.org/r/1248508

EBernhardson moved this task from Needs Review to To be Deployed on the Discovery-Search (2026.02.02 - 2026.02.27) board.Mar 5 2026, 5:26 PM
gerritbot added a comment.Mar 5 2026, 9:35 PM

Change #1248508 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Align semanticsearch cluster group name with routing

https://gerrit.wikimedia.org/r/1248508

Stashbot added a comment.Mar 5 2026, 9:36 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-05T21:36:05Z] <ebernhardson@deploy2002> Started scap sync-world: Backport for [[gerrit:1248508|cirrus: Align semanticsearch cluster group name with routing (T413969)]]

Stashbot added a comment.Mar 5 2026, 9:37 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-05T21:37:59Z] <ebernhardson@deploy2002> ebernhardson: Backport for [[gerrit:1248508|cirrus: Align semanticsearch cluster group name with routing (T413969)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Mar 5 2026, 9:43 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-05T21:43:25Z] <ebernhardson@deploy2002> Finished scap sync-world: Backport for [[gerrit:1248508|cirrus: Align semanticsearch cluster group name with routing (T413969)]] (duration: 07m 20s)

EBernhardson created subtask T419174: qwen3-embedding:predict returning 503 to all requests.Mar 5 2026, 9:59 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 5 2026, 10:31 PM
EBernhardson closed subtask T419174: qwen3-embedding:predict returning 503 to all requests as Resolved.Mar 6 2026, 7:26 PM
EBernhardson added a comment.Mar 6 2026, 8:59 PM

There are probably still a number of rough edges, but this is generally working now: frwiki example

pfischer edited projects, added: Discovery-Search (2026.03.03 - 2026.04.03); removed: Discovery-Search (2026.02.02 - 2026.02.27).Mar 9 2026, 11:14 AM
pfischer moved this task from Incoming to To be Deployed on the Discovery-Search (2026.03.03 - 2026.04.03) board.
bking mentioned this in T396501: Decide on/run a benchmark for DPE SRE-owned OpenSearch clusters.Mar 9 2026, 1:27 PM
EBernhardson moved this task from To be Deployed to Done on the Discovery-Search (2026.03.03 - 2026.04.03) board.Mar 9 2026, 4:03 PM
gerritbot added a comment.Mar 9 2026, 8:18 PM

Change #1249389 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] semantic: Provide instructions from profile to query

https://gerrit.wikimedia.org/r/1249389

gerritbot added a project: Patch-For-Review.Mar 9 2026, 8:18 PM
gerritbot added a comment.Mar 10 2026, 8:48 AM

Change #1249389 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] semantic: Provide instructions from profile to query

https://gerrit.wikimedia.org/r/1249389

ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.20; 2026-03-17); removed: MW-1.46-notes (1.46.0-wmf.18; 2026-03-03).Mar 10 2026, 9:01 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 10 2026, 9:32 AM
bking mentioned this in T421763: Migrate beta cluster to OpenSearch 2.x.Mar 30 2026, 6:27 PM
bking mentioned this in T422860: Migrate Cloudelastic to OpenSearch 2.x.Apr 9 2026, 4:41 PM
pfischer moved this task from Done to Reported on the Discovery-Search (2026.03.03 - 2026.04.03) board.Apr 10 2026, 8:41 PM
bking created subtask T424090: Undeploy opensearch-semantic-search environments from the DSE k8s clusters - All data to be removed 21 May 2026 @ 2100 UTC.Apr 21 2026, 8:27 PM
EBernhardson closed subtask T417087: Deploy Qwen3-Reranker-0.6B inference service for semantic search reranking as Declined.May 4 2026, 3:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits