Maniphest T168390

Alarm on HDFS related script failures
Closed, DuplicatePublic
Actions

Assigned To

None

Authored By

	elukey
	Jun 20 2017, 12:14 PM

Project Tags

Referenced Files

None

Subscribers

Description

Today we started seeing some HDFS datanode partitions on analytics brokers to fill up due to HDFS free space reduced to a tiny percentage. Over the past three months there was a steady increase that we didn't notice, mostly due to the refinery-drop-webrequest-partitions script (an1003) failing due to a file system permission issue.

We should alarm on refinery-drop-webrequest-partitions errors, do we need to do anything more than MAILTO in crontab? (we will need to review all scripts to see whether they put errors only on std out)

Related Objects

Mentioned In: T168497: Data Lake queries abort with HDFS write fail

Event Timeline

elukey created this task.Jun 20 2017, 12:14 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 20 2017, 12:14 PM

Should we put this on kanban? seems important enough.

• Tbayer mentioned this in T168497: Data Lake queries abort with HDFS write fail.Jun 21 2017, 4:06 PM

• Tbayer subscribed.

• Nuria moved this task from Incoming to Dashiki on the Analytics board.Jun 26 2017, 4:01 PM

• Nuria renamed this task from Alarm on HDFS space used and related script failures to Alarm on HDFS related script failures.Jul 6 2017, 4:38 PM

• Nuria updated the task description. (Show Details)

elukey added a project: User-Elukey.Aug 4 2017, 11:57 AM

elukey closed this task as a duplicate of T172532: Refactor analytics cronjobs to alarm on failure reliably.Aug 4 2017, 2:51 PM