[Epic] Re-architect the Search Update Pipeline
Open, HighPublic
Actions

Assigned To

None

Authored By

	Gehel
	Sep 5 2022, 2:54 PM

Description

The Search Update Pipeline was architected ~8 years ago. It has served its purpose well, but now is time to review its architecture and address a few of the long lived limitations it has. Design document here.

High level plan:

Test the updater job on the dse-k8s cluster
- create a namespace for the cirrus-streaming-updater on the dse-k8s cluster: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/951960
- T328675 create a helmfile service using the FlinkDeployment resource via the flink-app helm chart
- T341792 Provision Zookeeper Cluster for storing Flink HA data
- T344614 Add Zookeeper config to 'cirrus-streaming-updater' test service on DSE cluster
- test various maintenance operations: taking savepoint, job upgrade, H/A recoveries (kill pods manually), k8s upgrade (wipe out the namespace, T293063), ... (see also T328561)
Enable the k8s-operator on the staging wikikube cluster for the cirrus-streaming-updater namespace (might need a dedicated task)
- test various maintenance operations on staging wk: taking savepoint, job upgrade, H/A recoveries (kill pods manually), k8s upgrade (wipe out the namespace, T293063), ... (see also T328561)
Enable the k8s-operator on the production wikikube cluster for the cirrus-streaming-updater namespace (might need a dedicated task)

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T317045 [Epic] Re-architect the Search Update Pipeline
Resolved		EBernhardson	T314063 Create a skeleton project for the Flink part of the Search Update Pipeline
Resolved		Gehel	T317046 Coordinate with Platform Engineering / Data Value Stream Team about a rework of the Search Update Pipeline
Resolved		EBernhardson	T317023 Investigate moving incoming_links computation to a batch job
Resolved		EBernhardson	T265056 Make Cirrus Search dump script more resilient to failures (elasticsearch restarts)
Resolved		dcausse	T316016 Limit the size of the documents indexed by CirrusSearch
Resolved		pfischer	T317202 Model the update document used by the CirrusSearch Update Pipeline
Resolved		EBernhardson	T322327 CirrusSearch should generate a document consistent to a given schema
Resolved		Gehel	T317283 Coordinate with ServiceOps Team about a rework of the Search Update Pipeline
Resolved		dcausse	T317309 Create an API that renders the CirrusSearch indexable document based on a page_id and a revision id
Resolved		dcausse	T317609 Create a schema for fetch failures
Resolved		Gehel	T317611 Implement the enrichment function
Declined		None	T318388 Configure spotless to better match WMF java coding style
Resolved		pfischer	T318396 Use assertj for unit tests
Declined		None	T318649 The enrichment function should call a API endpoint based on the event domain
Resolved		pfischer	T322186 Consume revision based changes from the mediawiki.page-state stream
Resolved		pfischer	T323506 Implement the ingestion job
Resolved		dcausse	T323508 The preparation job should discover what index to write to
Resolved		Gehel	T323509 Deploy the cirrus-update pipeline to YARN for testing
Resolved		EBernhardson	T323687 Enable the wmf_capped doc size limiter in the mediawiki-config for CirrusSearch
Resolved		pfischer	T323688 The kafka consumer group should be configurable
Resolved		pfischer	T323690 Add support page deletion
Resolved		pfischer	T325315 Add support for redirects in CirrusSearch
Resolved		bking	T344366 Rollout Elasticsearch extra plugins package and restart cluster to apply
Resolved		dcausse	T325273 EventRowTypeInfo should support schema evolution of rows seriliazed in flink application state
Open		pfischer	T325565 Add support for page re-renders
Resolved		EBernhardson	T325672 Re-order and optimize change events
Open		pfischer	T326318 Create docker images for the cirrus-streaming-updater flink jobs
Open		None	T326328 Create helm charts for the cirrus-streaming-updater flink jobs
Open		dcausse	T328330 Create SLI / SLO on Search update lag and error rate
Resolved		dcausse	T320408 Monitor CirrusSearch update lag
Resolved	BUG REPORT	dcausse	T331127 phantom redirects lingering in incategory searches after page moves
Resolved		pfischer	T332763 The search update pipeline should support events compatible with the /mediawiki/page/change/1.0.0 schema
Open		None	T340548 [EPIC] Deployment of the Search Update Pipeline on Flink / k8s
Open		lbowmaker	T328561 Flink Operations
Resolved		gmodena	T328563 [Flink Operations] How to handle restarting a Flink application
Open		None	T328565 [Flink Operations] Automate Replay of Failed Events
Resolved		gmodena	T328569 [Flink Operation] How to handle app upgrades
Open		bking	T326409 Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model
Resolved		dcausse	T289836 Upgrade the WDQS streaming updater to latest flink (1.16)
Resolved		Ottomata	T324576 Flink on Kubernetes Helm charts
Resolved		Ottomata	T326252 k8s deployment-charts mesh module should allow use of mesh without public_port Service
Resolved		dcausse	T328675 Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart
Resolved		dcausse	T334244 Upgrade the wdqs updater jobs to flink 1.16.1
Resolved		bking	T341792 Provision Zookeeper Cluster for storing Flink HA data
Resolved		bking	T341705 eqiad: 3 VMs requested for Zookeeper
Resolved		None	T343715 codfw: 3 VMs requested for Zookeeper
Open		bking	T342149 Test common operations in the flink operator/k8s/Flink ZK environment
Resolved		bking	T344614 Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster
Resolved		bking	T345957 Restore dse-k8s' rdf-streaming-updater from savepoint/improve bootstrapping process
Resolved		bking	T346048 Troubleshoot rdf-streaming-updater/dse-k8s cluster
Resolved		MatthewVernon	T342620 Storage request: swift s3 bucket for flink search-update-pipeline checkpointing
Open		None	T341332 [EPIC] The CirrusSearch streaming updater should support private wikis
Open		None	T345185 Provide a method for internal services to run api requests in a private context
Open		None	T346046 [Search Update Pipeline] Source streams for private wikis
Open		EBernhardson	T344357 [Search Update Pipeline] avoid duplicate updates (multi DC)
Resolved		Gehel	T341625 Requesting permission to use kafka-main cluster to transport CirrusSearch updates
Resolved		EBernhardson	T345634 [Search Update Pipeline] Add a way to filter input events per wiki
Duplicate		None	T345638 [Search Update Pipeline] Add a way to configure a default http route
Resolved		dcausse	T346015 [Search Update Pipeline] Consider dropping support for java8
Open		None	T346717 [Search Update Pipeline] Name and identify operators that have a state
Open		None	T346718 [Search Update Pipeline] Set max parallelism explicitely on operators with a state
Open		None	T347075 Deploy test instance of cirrus updater in k8s
Open		pfischer	T347184 [Search Update Pipeline] Rename InputEvent to UpdateEvent
Open		None	T346895 [Search Update Pipeline] Reference latest streams/schemas
Open		pfischer	T346719 [Search Update Pipeline] Upgrade to flink 1.17.1
Open		None	T347543 [Search Update Pipeline] Fetch: Handle Timeout of AsyncAwaitOperator