Maniphest T340793

Implement depool (source only) and keep-downtime options on data-transfer cookbook
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	bking
	Jun 29 2023, 10:13 PM

Tags

Referenced Files

None

Subscribers

Description

As we bring more hosts online in T332314 , we're inadvertently triggering a lot of alerts. We're also getting some unnecessary cookbook failures.

Creating this ticket to:

~~Add a "keep downtime" option to prevent alerts from triggering on hosts that aren't yet ready for production.~~ Don't remove downtimes, just rely on the duration.
Add a "depool source only" option to allow repooling of the source host only. The current options force us to either not depool (so we have to remember to do that manually), or depool and get a failure because the "pool" option is not yet available on the new hosts.

Details

	Subject	Repo	Branch	Lines +/-
	wdqs.data-transfer: Keep downtime	operations/cookbooks	master	+38 -37
	wdqs.data-transfer: Add more pool options	operations/cookbooks	master	+33 -10
	wdqs.data-transfer: reformat using black	operations/cookbooks	master	+120 -70

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T291916 Tracking task for Bullseye migrations in production
		Resolved		Gehel	T323921 [Epic] Migrate all Search Platform servers to Debian Bullseye
		Resolved		bking	T332314 Service implementation for wdqs20[13-22]
		Resolved		bking	T331300 Ensure WDQS stack works on Bullseye
		Resolved		bking	T340793 Implement depool (source only) and keep-downtime options on data-transfer cookbook

Event Timeline

bking created this task.Jun 29 2023, 10:13 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 29 2023, 10:13 PM

bking claimed this task.Jun 30 2023, 2:04 PM

bking triaged this task as High priority.

bking added a project: Discovery-Search (Current work).

Change 934595 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] wdqs.data-transfer: reformat using black

https://gerrit.wikimedia.org/r/934595

gerritbot added a project: Patch-For-Review.Jun 30 2023, 5:46 PM

Change 934602 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] wdqs.data-transfer: Add more pool options

https://gerrit.wikimedia.org/r/934602

Change 934595 abandoned by Bking:

[operations/cookbooks@master] wdqs.data-transfer: reformat using black

Reason:

We still need to do this, but will hold off until we finish making functional changes described in T340793

https://gerrit.wikimedia.org/r/934595

bking set the point value for this task to 3.Jul 10 2023, 3:32 PM

bking moved this task from Incoming to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.

bking added a parent task: T331300: Ensure WDQS stack works on Bullseye.Jul 10 2023, 9:23 PM

Change 934602 merged by jenkins-bot:

[operations/cookbooks@master] wdqs.data-transfer: Add more pool options

https://gerrit.wikimedia.org/r/934602

Maintenance_bot removed a project: Patch-For-Review.Jul 12 2023, 9:30 PM

Gehel moved this task from Incoming to In Progress on the Data-Platform-SRE board.Aug 10 2023, 6:36 PM

Gehel removed a project: Discovery-Search (Current work).

Gehel removed a project: KaiOS-Wikipedia-app (Discovery).

Based on recent cookbook runs, it appears that the "lvs_strategy=both" option is not working. Leaving this as a note to myself to look at it Monday.

Change 937535 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] wdqs.data-transfer: Keep downtime

https://gerrit.wikimedia.org/r/937535

gerritbot added a project: Patch-For-Review.Aug 14 2023, 3:40 PM

RKemper updated the task description. (Show Details)Sep 6 2023, 8:55 PM

RKemper moved this task from In Progress to Done on the Data-Platform-SRE board.

Change 937535 merged by jenkins-bot:

[operations/cookbooks@master] wdqs.data-transfer: Keep downtime

https://gerrit.wikimedia.org/r/937535

Maintenance_bot removed a project: Patch-For-Review.Sep 6 2023, 9:10 PM

bking added a subtask: T342162: "scap deploy"'s config-deploy should check for broken symlinks.Sep 12 2023, 4:53 PM

Removed subtask because I think the scap ticket is not directly related to this one.

Gehel closed this task as Resolved.Oct 17 2023, 3:33 PM