Maniphest T303134

Should wdqs LVS checks page
Open, HighPublic
Actions

Assigned To

None

Authored By

	jbond
	Mar 6 2022, 6:03 PM

Tags

Referenced Files

None

Subscribers

Description

During a recent incident, We didn't get paged until the entire service was unavailable, even though there where early warning signs of the issue in icinga. this task is to evaluate if the early alerts should have also paged. Or alternatively if we should stop paging all together for this service

Alerts which started the incident

<+icinga-wm> PROBLEM - LVS wdqs eqiad port 80/tcp - Wikidata Query Service IPv4 #page on wdqs.svc.eqiad.wmnet 
          is CRITICAL: CRITICAL - Socket timeout after 10 seconds 
          https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems

Early warning alert

PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL -
                   wdqs-heavy-queries_8888: Servers wdqs1013.eqiad.wmnet, wdqs1004.eqiad.wmnet are marked down but
                   pooled: wdqs-ssl_443: Servers wdqs1004.eqiad.wmnet, wdqs1013.eqiad.wmnet are marked down but
                   pooled: wdqs_80: Servers wdqs1004.eqiad.wmnet, wdqs1013.eqiad.wmnet are marked down but pooled
                   https://wikitech.wikimedia.org/wiki/PyBal

Acceptance Criteria

Evaluate whether we can/should disable paging on pybal host marked down but pooled as well

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		RKemper	T313751 Create WDQS uptime SLO
		Resolved		RKemper	T325324 Evaluate options to soften wdqs paging
		Open		None	T303134 Should wdqs LVS checks page

Event Timeline

jbond created this task.Mar 6 2022, 6:03 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2022, 6:03 PM

Maintenance_bot added a project: Wikidata.Mar 6 2022, 6:45 PM

Restricted Application added a project: [DEPRECATED] wdwb-tech. · View Herald TranscriptMar 6 2022, 6:45 PM

• MPhamWMF moved this task from Incoming to Current work on the Wikidata-Query-Service board.Mar 7 2022, 4:42 PM

• MPhamWMF added a project: Discovery-Search (Current work).

jbond updated the task description. (Show Details)Mar 7 2022, 5:03 PM

• MPhamWMF moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.Mar 14 2022, 3:38 PM

bking claimed this task.Mar 14 2022, 3:39 PM

We'll talk about this in the Weds meeting this week

bking removed bking as the assignee of this task.Oct 12 2022, 1:46 PM

bking moved this task from In Progress to Ready for Dev -- SWE on the Discovery-Search (Current work) board.

bking subscribed.

EBernhardson moved this task from Ready for Dev -- SWE to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.Oct 24 2022, 3:30 PM

Gehel moved this task from Ready for Dev -- SRE/Ops to Incoming on the Discovery-Search (Current work) board.Nov 14 2022, 4:52 PM

Gehel triaged this task as High priority.Nov 21 2022, 4:39 PM

Gehel edited projects, added Discovery-Search; removed Discovery-Search (Current work).

Gehel moved this task from needs triage to Ops / SRE on the Discovery-Search board.

RKemper added a parent task: T325324: Evaluate options to soften wdqs paging.Feb 28 2023, 8:10 PM

RKemper mentioned this in T325324: Evaluate options to soften wdqs paging.

Gehel removed a project: Discovery-Search.Mar 16 2023, 7:31 PM

Gehel moved this task from Current work to Operations/SRE on the Wikidata-Query-Service board.

RKemper updated the task description. (Show Details)Apr 17 2023, 6:40 PM

RKemper updated the task description. (Show Details)May 9 2023, 8:03 PM