Maniphest T119479

Create a script that'll allow easier swatting of queries that're overloading ToolsDB
Closed, InvalidPublic
Actions

Assigned To

None

Authored By

	yuvipanda
	Nov 24 2015, 2:11 AM

Description

So right now spotting queries that are being too resource intensive for no reason (such as: running 9 concurrent and exactly-the-same queries!) and swatting them is a bit hard. It involves:

Finding which user is running these
Killing the query
Commenting out crontabs / suspending / killing jobs
Emailing the user

We need to automate a fair bit more of this.

Related Objects

Mentioned In: T248843: Upgrade wmf-pt-kill package for Debian Buster
Mentioned Here: T183983: Re-institute query killer for the analytics WikiReplica

Event Timeline

yuvipanda created this task.Nov 24 2015, 2:11 AM

yuvipanda raised the priority of this task from to Needs Triage.

yuvipanda updated the task description. (Show Details)

yuvipanda added projects: Cloud-Services, DBA.

yuvipanda added subscribers: yuvipanda, jcrespo, Legoktm.

Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptNov 24 2015, 2:11 AM

Krenair subscribed.Nov 24 2015, 2:28 AM

• chasemp triaged this task as High priority.Nov 30 2015, 4:51 PM

• chasemp subscribed.

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:46 PM

With the deployment of T183983: Re-institute query killer for the analytics WikiReplica there is now a log placed at labsdb10XX:/var/log/wmf-pt-kill/wmf-pt-kill.log which can be parsed if someone from cloud-services-team still think this is a valid ticket.
I doubt the priority is high, as the query killer has been doing its job since we set up the new labsdb hosts, but not touching the priority myself here.

bd808 renamed this task from Create a script that'll allow easier swatting of queries that're overloading labsdb to Create a script that'll allow easier swatting of queries that're overloading ToolsDB.Jan 4 2020, 9:32 PM

bd808 edited projects, added Data-Services; removed Cloud-Services.

bd808 moved this task from Backlog to ToolsDB on the Data-Services board.

jcrespo mentioned this in T248843: Upgrade wmf-pt-kill package for Debian Buster.Mar 30 2020, 11:17 AM

• Bstorm added a project: cloud-services-team (Kanban).Oct 15 2020, 9:13 PM

• nskaggs closed this task as Invalid.Oct 20 2020, 4:41 PM

Create a script that'll allow easier swatting of queries that're overloading ToolsDBClosed, InvalidPublicActions

Description

Related Objects

Event Timeline

Create a script that'll allow easier swatting of queries that're overloading ToolsDB
Closed, InvalidPublic
Actions