Adapt Flink-related rdf-streaming-updater alerts for Cirrus Streaming Updater
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bking
	Mar 5 2024, 8:30 PM

Description

Per today's pairing session, @EBernhardson pointed out that we don't have any flink-related alerts for the new Search Update Pipeline in the alerts repo yet. Creating this ticket to:

Adapt existing alerts from rdf-streaming-updater
Verify operation
~~Add new alerts as needed~~

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	cirrus-streaming-updater: link alerts to DPE SRE	operations/alerts	master	+8 -0
	Add Flink alerts for Cirrus Streaming Updater	operations/alerts	master	+136 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Gehel	T317045 [Epic] Re-architect the Search Update Pipeline
Resolved	Gehel	T340548 [EPIC] Deployment of the Search Update Pipeline on Flink / k8s
Resolved	Ottomata	T328561 [Event Platform] Flink Operations
Resolved	gmodena	T328563 [Flink Operations] How to handle restarting a Flink application
Open	None	T328565 [Flink Operations] Automate Replay of Failed Events
Resolved	gmodena	T328569 [Flink Operation] How to handle app upgrades
Resolved	bking	T344614 Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster
Resolved	bking	T345957 Restore dse-k8s' rdf-streaming-updater from savepoint/improve bootstrapping process
Resolved	bking	T346048 Troubleshoot rdf-streaming-updater/dse-k8s cluster
Resolved	Gehel	T341625 Requesting permission to use kafka-main cluster to transport CirrusSearch updates
Resolved	MatthewVernon	T342620 Storage request: swift s3 bucket for flink search-update-pipeline checkpointing
Resolved	bking	T347075 Deploy test instance of cirrus updater in k8s
Resolved	bking	T348350 Set requests (not limits) for cirrus-streaming-updater in k8s
Resolved	EBernhardson	T350186 Cirrus-streaming-updater test: validate relforge indices are correctly updated
Resolved	EBernhardson	T350299 EventBus change events involving redirect changes are sometimes incorrect
Resolved	dcausse	T326914 Migrate the WDQS streaming updater from FlinkKafkaConsumer/Producer to KafkaSource/Sink
Invalid	None	T349848 Determine and control cirrus streaming updater's usage of MWAPI resources
Invalid	None	T350185 Estimate cirrus streaming updater's usage of MWAPI
Resolved	dcausse	T350826 Test backfilling for cirrus-streaming-updater
Resolved	EBernhardson	T358518 Deploy streaming updater for 100% of writes to cloudelastic
Duplicate	None	T345698 [Epic] define a strategy around alerting for Data Platform SRE and implement it
Open	None	T346438 [Epic] Review alerting strategy for Data Platform SRE
Resolved	bking	T349772 Create dashboards/alerts for new Cirrus Streaming Updater
Resolved	bking	T359213 Adapt Flink-related rdf-streaming-updater alerts for Cirrus Streaming Updater

Event Timeline

bking created this task.Mar 5 2024, 8:30 PM

Change rMW100935969185 had a related patch set uploaded (by Bking; author: Bking):

[operations/alerts@master] WIP: Add Flink alerts for Cirrus Streaming Updater

https://gerrit.wikimedia.org/r/1009359

gerritbot added a project: Patch-For-Review.Mar 6 2024, 10:25 PM

Gehel triaged this task as High priority.Mar 20 2024, 9:07 AM

Gehel edited projects, added Data-Platform-SRE (2024.03.04 - 2024.03.24); removed Data-Platform-SRE.

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.03.04 - 2024.03.24) board.

Gehel edited projects, added Data-Platform-SRE (2024.03.25 - 2024.04.14); removed Data-Platform-SRE (2024.03.04 - 2024.03.24).Mar 22 2024, 8:44 AM

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.03.25 - 2024.04.14) board.Mar 22 2024, 8:45 AM

Gehel edited projects, added Data-Platform-SRE; removed Data-Platform-SRE (2024.03.25 - 2024.04.14), Patch-For-Review.Apr 2 2024, 3:30 PM

Gehel moved this task from Incoming to Quarterly Goals on the Data-Platform-SRE board.

bking changed the task status from Open to In Progress.Apr 4 2024, 1:33 PM

bking claimed this task.

bking lowered the priority of this task from High to Medium.

bking edited projects, added Data-Platform-SRE (2024.03.25 - 2024.04.14); removed Data-Platform-SRE.

Gehel edited projects, added Data-Platform-SRE (2024.04.15 - 2024.05.05); removed Data-Platform-SRE (2024.03.25 - 2024.04.14).Apr 15 2024, 12:39 PM

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.04.15 - 2024.05.05) board.

Change #1009359 merged by Bking:

[operations/alerts@master] Add Flink alerts for Cirrus Streaming Updater

https://gerrit.wikimedia.org/r/1009359

Mentioned in SAL (#wikimedia-operations) [2024-04-24T19:08:20Z] <inflatador> bking@deploy1002 stop consumer-cloudelastic release to test alerting T359213

Change #1023942 had a related patch set uploaded (by Bking; author: Bking):

[operations/alerts@master] cirrus-streaming-updater: link alerts to DPE SRE

https://gerrit.wikimedia.org/r/1023942

gerritbot added a project: Patch-For-Review.Apr 24 2024, 7:56 PM

bking updated the task description. (Show Details)Apr 30 2024, 8:40 PM

The above patch is more about cloning all current search platform alerts to data platform SRE, so I'll create a separate ticket for that. The rest of this ticket's AC has been satisfied, so I'm closing it out.

bking mentioned this in T356302: setup production Cirrus Streaming Updater alerts .May 13 2024, 3:11 PM

Change #1023942 abandoned by Bking:

[operations/alerts@master] cirrus-streaming-updater: link alerts to DPE SRE

Reason:

superseded by I566bc7df4ce05dc8650aed042ae66c65e4c70258