Maniphest T181634

Investigate overload condition, seems that we lose nodes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	awight
	Nov 29 2017, 3:11 PM

Tags

Referenced Files

None

Subscribers

Description

I was noticing this while stress testing on the new cluster. If overload lasts long enough, celery services start dying off.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T181538 ORES overload incident, 2017-11-28
		Resolved		awight	T181634 Investigate overload condition, seems that we lose nodes

Event Timeline

awight created this task.Nov 29 2017, 3:11 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptNov 29 2017, 3:11 PM

greg moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Dec 5 2017, 8:13 PM

awight moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.Jun 21 2018, 11:38 AM

Fixed on the new cluster. This was caused by out-of-memory.

Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM