Page MenuHomePhabricator

Kubernetes setup for similarity search
Open, Needs TriagePublic

Description

Tasks

  • Create namespace vector-search for the similarity search service for research workloads
  • Create namespace dding for realtime workloads for DDoS detection
  • Create a deployment chart for the similarity search service
  • Create a deployment chart for the streaming pipeline
  • Create secrets for S3 credentials

Stretch tasks

  • Configure internal egress so that other services can reach the vector-search namespace
  • Integrate similarity search with CAS/IdP auth and expose it via a fully qualified route at vector-search.wikimedia.org

Event Timeline

Change #1255948 had a related patch set uploaded (by Fabian Kaelin; author: Fabian Kaelin):

[operations/deployment-charts@master] vector-search: add initial deployment chart

https://gerrit.wikimedia.org/r/1255948

Gehel subscribed.

@fkaelin : could we have a few more details about what this service will be doing? Is there persistance involved? Do you have any idea of the sizing?

For the vector-search service, which is the main priority (with CR)

  • the service is the UI component for the traffic pattern pipline
    • this readme documents the data pipeline that produces the data queried by this service
    • see the user guide for the type of functionality the service provides
  • the service reads data from Ceph S3 generated by the pipeline linked to above, no data is persisted from the service itself
  • regarding sizing: this is an internal tool so the number of requests are limited and latency not the main concern. I deployed it with 4cores and 4GB and that seems to work fine.

For the streaming pipeline (no CR yet, but working chart):

  • processes webrequest kafka streams. There are different variants (e.g. for pageviews, ddos detection, etc), the variant I want to deploy is the prototype for streaming trend detection for pages.
  • uses Ceph S3 for persistence (read & write from service), no persistence otherwise
  • requires the maxmind dbs (T405509), currently uses a temporary pvc for that
  • regarding sizing: greatly depends on the configuration ~8cpu/20GB of ram