First tests with the full frwiki semantic search dataset showed high latency and significant ceph IO at ~4GB/sec. This appears to be a problem with readahead on the ceph-backed storage system. It defaults to 8MB which is far too much for the random-access nature of knn search.

We've evaluated significantly reduced readahead of 64kb and found reasonable performance, but we've only tested that so far on single-shard queries. Our existing k8s deployments don't have the ability for us to set readaheads from the deployment side, we've so far relied on @bking manually applying the settings from the host machine to support this evaluation. In T418776 dpe-sre is looking into how to best support this change.

Approximate results from single-shard testing. Note that 1:1 means the entire index fits in memory and skips ceph, deploying that for the full test would require over 1TB of memory from dpe-sre-k8s and should be avoided if possible.

Change #1249382 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/deployment-charts@master] semanticsearch: Increase heap by 1gb

https://gerrit.wikimedia.org/r/1249382

index/memory ratio has been a bit vague, to be more concrete:

Further evaluation. This tests disabling the readaheads completely, along with evaluating less memory for the same indexes.

This demonstrates that completely removing readahead is a bad idea in this context. Not captured in the table is that warmup takes significantly longer (~10 min to stabilize, vs ~2 minutes), and that cross-node index transfers are also perhaps 2x-4x slower than with a minimal readahead (not measured, but observed). This test went up to 14 concurrents as i was trying to see if it could get the full 35qps available from the cpus, but with 3:1 memory and no readaheads it wasn't getting there.

We then updated to opensearch 3.5.0. This was not expected to, but seems to have brought with it some significant perf improvements. An alternate idea might be that something with the embeddings predictions changes, we don't have the visibility to be sure. This led to hitting circuit breakers for heap memory, rejecting requests instead of queueing them.

Instances were updated with +25% heap memory (5g vs 4g) and that seems to have gone away:

This is looking to be in a pretty good space now, the reduced memory usage is promissing for the full rollout.

As for resource requirements, from david's initial review we have an expected size, with 3 copies of each shard, of 1073 gb. From this testing I think we can manage with around 450G of disk cache. It seems reasonable to round that up to 512G to leave space for growth. The exact amount of heap is uncertain, but the current 5G will most likely work, and 6G should be more than sufficient. We are expecting to use 15 shards for enwiki, it seems like 16 nodes is then a viable minimum number we should aim for. We could potentially go with 24 or even 32 nodes, but i think i would lean towards fewer nodes if possible.

For each set of nodes we would need varying disk per-instance. We estimate the total size is 1073g, and we need another 700g to have space for reindexing. Opensearch requires us to keep the disk < 70% full. Rounding up to 1900gb to leave a bit of room, we get to 2715gb required.

Change #1249382 abandoned by Ebernhardson:

[operations/deployment-charts@master] semanticsearch: Increase heap by 1gb

Reason:

pod sizing has been separately addressed

https://gerrit.wikimedia.org/r/1249382